0130 terms
Answer engines and retrieval
How a generated answer gets made, and what your page is competing for inside it.
AEOGEOAI visibility scoreshare of voiceprompt setbrand mentionRAG+23 more
Reference · glossary
Every definition says what the term can support and what it cannot, because most of the confusion in this field is a word doing more work than the evidence behind it.
In one sentence
The CrawlCheck glossary defines 792 AI-visibility, GEO, AEO and AI-crawler terms across 19 areas — from AI visibility score, llms.txt and entity graph to verified crawler, RAG and grounding — and every definition states what the term can support and what it cannot, because most confusion in this field is a word doing more work than the evidence behind it.
Read it as an agent
Written against what CrawlCheck measures. Where a term is commonly used to mean more than it can carry — a citation, a forged crawler, a visibility score, a passing file — the definition says so rather than repeating the looser sense. Terms that name a finding code show the live share of scans carrying it.
Four stages
A term in a later stage is worth nothing if an earlier one never happened: a page that was never fetched cannot be misread, and a fact that was never parsed cannot be quoted.
Reach
can a client get the bytes
Crawlers and access · Delivery and rendering · HTTP, edge and rendering · Crawling and indexing
Read
can it parse what it got
Machine files · Entities and schema · Linked data and entities · Indexing and discovery · Content architecture and SEO
Quote
can an engine safely repeat it
Answer engines and retrieval · Retrieval engineering · Models and generation · Measurement and evidence · Measurement and evaluation · Proof and provenance
Operate
can an agent act on it, safely
Agents and protocols · Security and privacy · Governance and rights · Accessibility and provenance
Added the week of November 16, 2026
The glossary grows every week, each term written against something CrawlCheck serves or measures. Every week’s additions as JSON.
Cookie prefixPartitioned cookieSameSitePublic suffixSchemeful same-siteSubresource IntegrityIntegrity metadataCSP report-onlyCSP hashstrict-dynamicform-actionupgrade-insecure-requestsbase-uriobject-srcReferrer leakageDangling CNAMESubdomain takeoverSPF lookup limitTLS downgradeCertificate Transparency
19 areas
0130 terms
How a generated answer gets made, and what your page is competing for inside it.
AEOGEOAI visibility scoreshare of voiceprompt setbrand mentionRAG+23 more
0246 terms
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolvepublished crawler rangesoperator originsigned agentagent arrivalagent conduct recordagent traffic index+39 more
0324 terms
The files addressed to machines rather than readers, and what publishing one does and does not buy.
llms.txtllms-full.txtagents.mdentity mapentity map paritymachine layerwell-known URI+17 more
0445 terms
Declaring who you are in a form a resolver can use without guessing.
invented valuesigned business recordlisting auditlisting supplierfix receiptstructured dataJSON-LD+38 more
0536 terms
Whether a page can be found and kept, separately from whether it can be fetched.
nosnippetmax-snippetmax-image-previewunavailable_afternoimageindexindex coveragecanonical+29 more
0617 terms
What arrives on the wire, and what only exists after something else runs.
broken embedpayload ratiorender dependencerender dependence gapedge cache pinningTTFBCore Web Vitals+10 more
0752 terms
The vocabulary for saying how strongly something is known.
technographic dataread equivalencecontent-change clockself-claim ledgeragent task canaryrevocation deltasigned verdict+45 more
0826 terms
Showing a record existed at a date without asking anyone to trust us.
BibTeXRIS citation formattransparency logsigned tree headinclusion prooflog witnesssplit-view attack+19 more
0935 terms
How a model produces text, what its training and decoding controls change, and what none of them can prove about a source.
Autoregressive modelTransformerAttentionSelf-attentionParameterInferencePretraining+28 more
1045 terms
How pages become chunks, vectors and candidates inside an answer engine, and why being retrieved is not being cited.
CorpusDocument ingestionData connectorParsingOCRLayout-aware parsingSemantic chunking+38 more
1146 terms
How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.
Sitemap directiveCrawler identityProduct tokenIdentification stringCrawl frontierCrawl queueCrawl scheduler+39 more
1272 terms
The delivery layer between an origin and a client: proxies, caches, redirects, status codes and the headers that decide what a crawler actually received.
Content-Language headerHTTP/3Alt-SvcHTTPS DNS recordLink headercontent negotiationMarkdown negotiation+65 more
1353 terms
The RDF and JSON-LD machinery behind structured data, and the schema.org types a business page actually needs.
SearchActionDataset (schema.org)DataDownloadDataCatalogvariableMeasuredTechArticleCollectionPage+46 more
1445 terms
How pages, links and intent are organised so an entity is findable, and what a link or a structure can and cannot transfer.
lang attributex-defaultSearch intentQuery classNavigational queryInformational queryTransactional query+38 more
1533 terms
How a measurement earns trust: evaluation sets, error rates, calibration, reproducibility, and the biases that make a number look better than it is.
AI answer readingname confusionhedged answerBenchmarkEvaluation setGolden datasetGround truth+26 more
1665 terms
How autonomous agents plan, remember, call tools and talk to each other, including MCP and A2A, and where each step can go wrong.
OpenSearch descriptiontraceparentJSON Schemaprotected-resource metadatatoken audiencePKCErefresh token+58 more
1768 terms
Where an agent, a tool or a page can be turned against its operator, and the controls that limit the damage without proving safety.
Cookie prefixPartitioned cookieSameSitePublic suffixSchemeful same-siteSubresource IntegrityIntegrity metadata+61 more
1832 terms
Who may use content for what: rights reservations, opt-outs, licences, model documentation and the accountability vocabulary around them.
per-use rightsrights feedText and data miningRights reservationMachine-readable opt-outUsage preferenceContent-Usage header+25 more
1922 terms
What a page owes a keyboard, a screen reader and a fact-checker: the accessibility basics and the C2PA vocabulary for proving where a file came from.
Keyboard accessibilityFocus orderFocus indicatorARIAAccessible descriptionColor contrastReduced motion+15 more
How to read these
New · measured weekly
659 labels a reasoning model writes while it picks a local business, from hidden_intent_possible to trust_risk, defined in the order it uses them, with what a business can do about each and a weekly count of which ones it actually uses. Open the atlas.
Every term
#
202 interstitial301 redirect302 redirect304 response307 redirect308 redirect404 and 410 status codes404 response410 response429 response500 response502 response503 response504 response@contextA
A2AA2A clientA2A protocolA2A serverAbstentionaccessibility structureAccessible descriptionaccessible nameAccountabilityAccuracyaction assuranceAEOAge headeragent arrivalAgent artifactAgent authorizationagent cardagent conduct recordAgent discoveryAgent handoffAgent identityAgent loopagent manifestAgent memoryAgent orchestrationAgent sandboxAgent skillAgent taskagent task canaryagent traffic indexagentic knowledge graphagentic trafficagentic webagents.mdAggregateRatingAI agentAI answer readingAI ModeAI opt-outAI OverviewAI referral attributionAI training licenseAI visibility scoreai.txtAlgorithmic transparencyalias domainAllow headerAllowlistalt attributealt textAlt-SvcAnchor textanchoringanswer blockanswer engineanswer stabilityAnswer synthesisapex domainAPI keyApproval gateApproximate nearest neighborARIAArticleas-of answerASN blockingAttentionAudio descriptionAudit logAuthor relationshipAuthority weightingAutoregressive modelB
base-uriBearer tokenBenchmarkBibTeXBlank nodeblocked render resourceBlogPostingBM25boilerplatebot management platformBot scoreBot spoofingbrand mentionBreadcrumbListbroken embedBroken linkBrotlibylineBytespiderC
C2PAC2PA assertionC2PA claimCAA recordCache hitCache keyCache misscache splitcache-busterCache-ControlCalibrationcanaryCandidate generationcanonicalCanonical clustercanonical originCanonical RDFcanonical URLCapability negotiationCaptionscase consistencyCAWG training and data mining assertionCCBotCDNCertificate TransparencyChain of thoughtchallenge page at 200change receiptchangefreq and priorityChunk boundaryChunk overlapchunkingcitationcitation anchorCitation mappingcitation ratecite-as linkClaim signatureClaudeBot vs Claude-UsercloakingCLScohort asymmetryCollectionPageColor contrastCommercial investigationCompressionConditional requestConfidence intervalConfidence scoreConnection timeoutConsent recordConstrained decodingContactPointContent bindingContent CredentialContent decayContent encodingContent hashContent hubcontent negotiationContent refreshContent SignalsContent typecontent-addressed storagecontent-change clockContent-DigestContent-Language headerContent-Security-PolicyContent-SignalContent-Usage headerContext cachingContext compactionContext poisoningcontext windowcontrast ratioCookie prefixCopyright provenanceCore Web VitalsCorpusCosine similaritycosmetic rowCoverageCrawl anomalycrawl budgetCrawl demandCrawl frequencyCrawl frontierCrawl queueCrawl rateCrawl schedulerCrawl-delaycrawler classificationCrawler identityCrawler trapCredential scopecross-encoderCrUXCSP hashCSP nonceCSP report-onlyD
daily sealdangling @idDangling CNAMEdata aggregatorData AxleData connectorData controllerData exfiltrationData leakageData lineageData minimizationData processorData residencyData sheetdata snapshotDataCatalogdatacenter IPDataDownloadDataset (schema.org)Dataset licenseDatatypedated seriesdatelinedead anchordecision receiptDeduplicationDeep linkDefault graphdefault scope requestDefinedTermDefinedTermSetDeindexingDelegationdelivery comparisondenominator notedense retrievalDense vectorDenylistDeterministic decodingDeveloper messageDirect prompt injectionDKIMDMARCDMARC alignmentDNS resolutionDNS timeoutDNSSECDocument ingestionDofollowDOM depthdomainIncludesDot-product similaritydouble-counted defectDuplicate clusterduplicate contentduplicate idDynamic renderingE
E-E-A-TE-E-A-T proxiesedge cacheedge cache pinningEdge nodeEdge redirectEdge transformationembeddingEncrypted Client HelloEnd-anchor ruleentity collisionentity disambiguationentity graphentity mapentity map parityentity resolutionEntity-first contentEpisodic memoryETagEvaluation setevidence levelExact-match retrievalExcessive agencyExecutorEXIFExplainabilityExternal linkF
F1 scorefaceted navigationFair useFalse negativefalse positiveFalse positive rateFAQPageFAQPage schemaFCrDNSfetcher disagreementfield datafinding agefinding fingerprintfinding stateFine-tuningfix listfix receiptFixed-size chunkingFocus indicatorFocus orderforged crawler identityform control nameform-actionFoundation modelfounder edgefoundingDateframe-ancestorsFreshness boostingfreshness windowFunction callingG
gated rowGeneral-purpose AI modelGEOgeocodingGeoCoordinatesGET requestGolden datasetGoogle Business ProfileGoogle-AgentGoogle-ExtendedGPTBot vs OAI-SearchBotgraded through a wallGraph mergeGraph normalizationGround truthgroundingH
hallucinated attributionhallucinationHard bindinghard bouncehash-prefix lookupHEAD requestHeader parityHeading hierarchyheadroomhedged answerhigh-risk fieldHistorical optimizationHNSWhomepage refusedhomoglyphhoneypothost duplicateHost politenesshreflangHSTSHSTS preload listHTTP Message SignaturesHTTP/3HTTPS DNS recordHuman evaluationHuman-in-the-loopHuman-on-the-loopHybrid searchHyDEHydration mismatchI
Identification stringIf-None-Matchimage placeholderImage sitemapImpact assessmentInbound linkincident switchinclusion proofindex coverageIndex freshnessIndex lagIndex selectionIndexing pipelineIndexNowIndirect prompt injectionInferenceInformation architectureinformation gainInformational queryIngredientINPInput validationInsecure output handlingInstruction tuningIntegrity metadataintent coverageInter-rater agreementInternal linkinvented valueInverse propertyIRIItemListL
lab datalandmarklandmark regionlang attributeLanguage tagLast-ModifiedLate interactionLawful basisLayout-aware parsinglazy loadingLCPLeast privilegeLexical retrievalLighthouseLink contextLink equityLink extractionLink headerLink neighborhoodlisting auditlisting supplierLiteralLLM-as-judgellms-full.txtllms.txtlocal citationlocal packLocalBusinesslog file analysislog witnessLong-term memoryLongest-match rulelookalike domainM
machine layerMachine-readable opt-outmain landmarkmainEntityManifest storeManifest tamperingMarkdown negotiationmax-image-previewmax-snippetMCPMCP clientMCP hostMCP preflightMCP promptMCP resourceMCP rug pullMCP serverMCP toolMCP transportMean reciprocal rankMerkle treeMetadata filteringmicroformats (h-card)MIME sniffingMinHashMixture of expertsModel cardModel driftModel routingModel versionMTA-STSmTLSMulti-agent systemMulti-query retrievalMultimodal modelN
name confusionname driftnamed entity recognitionNamed graphNAPNAP consistencyNAP schemaNavigational querynear-duplicateNear-duplicate detectionNews sitemapNewsArticleNofollownoimageindexnoindexnonexistent-path controlNormalized discounted cumulative gainnosnippetO
OAuth scopeObjectobject-srcOCRoff-host redirectOfferOntologyOpen licenseOpenAPI specificationOpeningHoursSpecificationOpenSearch descriptionOpenTimestampsoperator feedoperator originOrganization schemaOrigin bypassOrigin serverorphan pageOrphaned entityOutput validationown-host loopP
PageRankPairwise evaluationParameterParent-child retrievalparked domainParsingPartitioned cookiePassage rankingpayload ratiopayload text shareper-use rightspercentile rankPermissions-PolicyPerplexityBot vs Perplexity-UserPerson schemaPIIPillar pagePKCEPlaceplaceholder contentplaceholder in recordPlannerPointwise evaluationpopulation baselinePostalAddressPredicatePreference optimizationpreflightpresence ratePretrainingProductProduct tokenprompt injectionprompt setprotected-resource metadataPublic domainPublic suffixpublication floorpublished crawler rangesPublisher relationshipPurpose limitationR
RAGrangeIncludesrate limitrate limitingRate-limit policyRCERDFRDF datasetRDFa and MicrodataRe-rankingread equivalenceRead timeoutReasoning tracereceipt headerReciprocal rank fusionreciprocated sameAsRecrawl intervalRed teamingRedirect chainRedirect loopRedirected internal linkReduced motionReferrer leakageReferrer-PolicyReferring domainrefresh tokenrefusalregressionRelevance scoreRemote contextRemoval requestrender dependencerender dependence gaprender gaprender-blocking resourceRendered HTMLRendering queueRepeatabilityReproducibilityresidential proxyresolve answerResponse parityRetention periodretrievalretrieval crawlerRetrieval cutoffRetrieval latencyRetrieval precisionRetrieval recallRetry-AfterReverse proxyReviewrevocation deltarights feedRights reservationRIS citation formatRisk assessmentrival rootsRLHFRobots cacheRobots group mergingRobots parse errorRobots redirectRobots unavailablerobots.txtrow reviewRubricRUMS
sameAsSameSiteSample sizeSampling biasSchema propertySchema typeSchemeful same-sitescore versionSearch intentSearchActionSecret leakagesecurity.txtSeed URLSelection biasSelf-attentionself-auditself-audit invariantself-claim ledgerSemantic cacheSemantic chunkingSemantic HTMLSemantic memorySemantic searchSensitive information disclosureServer-TimingServiceservice areaSHA-256share of voiceShared IP rangeshinglesigned absencesigned agentsigned business recordsigned tree headsigned verdictSitemapSitemap directiveSitemap hreflangsitemap indexSitemap lastmodsitemap origin errorSitemap URL setSKILL.mdskip linkSmall language modelSoft 404soft 404 on a machine fileSoft bindingSource diversitySparse vectorspeakableSPFSPF lookup limitsplit-view attackSponsored linkSSRFstable @idstaging leakStale responseStale-while-revalidatestapled verdictStatus messagestrict-dynamicstructured datastructured data validationStructured outputSubdomain takeoverSubjectSubprocessorSubresource IntegritySupervised fine-tuningSurvivorship biasSystem promptSystemic riskT
TaxonomyTDM exceptiontdmrep.jsonTechArticletechnographic dataTemperatureTenant isolationTest contaminationText and data miningthin contentThreat modelthreshold verdictTLS downgradeTLS handshakeTLS-RPTtokentoken audienceTokenizerTool callTool definitionTool descriptiontool lockfileTool permissiontool poisoningTool resultTool schemaTool selectiontool shadowingTop-kTop-p samplingTopic clustertopical authoritytopicalitytraceparenttraining dataTraining-data summaryTransactional queryTransformative useTransformertransparency logtripleTrue negativeTrue positiveTrust boundarytrust centerTrust listTTFBU
unavailable_afteruniform refusalunreciprocated profileunscored measurementunverifiableupgrade-insecure-requestsURIURLURL discoveryURL inspectionUsage preferenceuser agentuser-agent groupUser-generated linkuser-triggered fetchuser-triggered fetch exemptionNo engine publishes its ranking function, so every definition here describing influence is a description of a mechanism, not a lever. Terms marked as proposals — llms.txt, agents.md, entity maps — are bets with a low cost, not standards anyone has committed to reading. Where adoption figures exist they are on the dataset page, with the population they were counted against.