CrawlCheck

Reference · glossary

The words of the agentic web, defined with their limits.

Every definition says what the term can support and what it cannot, because most of the confusion in this field is a word doing more work than the evidence behind it.

In one sentence

The CrawlCheck glossary defines 688 AI-visibility, GEO, AEO and AI-crawler terms across 19 areas — from AI visibility score, llms.txt and entity graph to verified crawler, RAG and grounding — and every definition states what the term can support and what it cannot, because most confusion in this field is a word doing more work than the evidence behind it.

Read it as an agent

Every term as JSON-LDGET /glossary.jsonAny page as MarkdownAccept: text/markdownThe site map for modelsGET /llms.txt

Written against what this scanner measures. Where a term is commonly used to mean more than it can carry — a citation, a forged crawler, a visibility score, a passing file — the definition says so rather than repeating the looser sense. Terms that name a finding code show the live share of scans carrying it.

Browse by area A to Z

688terms
19areas
58terms with a live finding rate
58guides the terms link into

Four stages

Where a term belongs

A term in a later stage is worth nothing if an earlier one never happened: a page that was never fetched cannot be misread, and a fact that was never parsed cannot be quoted.

  1. 1
  2. 2
  3. 3
  4. 4

19 areas

Browse by area

How to read these

Three questions to ask of any term

What is the denominator?A rate without one is decoration.
Was absence measured or assumed?Unverifiable is not forged, and not found is not the same as not there.
What would it look like if it were wrong?A term that cannot fail a test is not measuring anything.

Every term

A to Z

#

202 interstitial301 redirect302 redirect304 response307 redirect308 redirect404 and 410 status codes404 response410 response429 response500 response502 response503 response504 response@context

A

A2AA2A clientA2A protocolA2A serverAbstentionaccessibility structureAccessible descriptionaccessible nameAccountabilityAccuracyaction assuranceAEOAge headeragent arrivalAgent artifactAgent authorizationagent cardagent conduct recordAgent discoveryAgent handoffAgent identityAgent loopAgent memoryAgent orchestrationAgent sandboxAgent skillAgent taskagent traffic indexagentic knowledge graphagentic trafficagentic webagents.mdAggregateRatingAI agentAI ModeAI opt-outAI OverviewAI referral attributionAI training licenseAI visibility scoreai.txtAlgorithmic transparencyalias domainAllowlistalt attributealt textAnchor textanchoringanswer blockanswer engineanswer stabilityAnswer synthesisapex domainAPI keyApproval gateApproximate nearest neighborARIAArticleASN blockingAttentionAudio descriptionAudit logAuthor relationshipAuthority weightingAutoregressive model

B

Bearer tokenBenchmarkBlank nodeblocked render resourceBlogPostingBM25boilerplatebot management platformBot scoreBot spoofingbrand mentionBreadcrumbListBroken linkBrotlibylineBytespider

C

C2PAC2PA assertionC2PA claimCAA recordCache hitCache keyCache misscache splitcache-busterCache-ControlCalibrationcanaryCandidate generationcanonicalCanonical clusterCanonical RDFcanonical URLCapability negotiationCaptionscase consistencyCAWG training and data mining assertionCCBotCDNChain of thoughtchallenge page at 200change receiptchangefreq and priorityChunk boundaryChunk overlapchunkingcitationCitation mappingcitation rateClaim signatureClaudeBot vs Claude-UsercloakingCLScohort asymmetryColor contrastCommercial investigationCompressionConditional requestConfidence intervalConfidence scoreConnection timeoutConsent recordConstrained decodingContactPointContent bindingContent CredentialContent decayContent encodingContent hashContent hubContent refreshContent SignalsContent typecontent-addressed storageContent-Security-PolicyContent-SignalContent-Usage headerContext cachingContext compactionContext poisoningcontext windowcontrast ratioCopyright provenanceCore Web VitalsCorpusCosine similaritycosmetic rowCoverageCrawl anomalycrawl budgetCrawl demandCrawl frequencyCrawl frontierCrawl queueCrawl rateCrawl schedulerCrawl-delaycrawler classificationCrawler identityCrawler trapCredential scopecross-encoderCrUXCSP nonce

D

daily sealdangling @iddata aggregatorData AxleData connectorData controllerData exfiltrationData leakageData lineageData minimizationData processorData residencyData sheetdata snapshotdatacenter IPDataset licenseDatatypedated seriesdatelinedead anchordecision receiptDeduplicationDeep linkDefault graphDefinedTermDeindexingDelegationdelivery comparisondenominator notedense retrievalDense vectorDenylistDeterministic decodingDeveloper messageDirect prompt injectionDKIMDMARCDMARC alignmentDNS resolutionDNS timeoutDNSSECDocument ingestionDofollowDOM depthdomainIncludesDot-product similaritydouble-counted defectDuplicate clusterduplicate contentduplicate idDynamic rendering

E

E-E-A-TE-E-A-T proxiesedge cacheedge cache pinningEdge nodeEdge redirectEdge transformationembeddingEnd-anchor ruleentity collisionentity disambiguationentity graphentity mapentity map parityentity resolutionEntity-first contentEpisodic memoryETagEvaluation setevidence levelExact-match retrievalExcessive agencyExecutorEXIFExplainabilityExternal link

F

F1 scorefaceted navigationFair useFalse negativefalse positiveFalse positive rateFAQPageFAQPage schemaFCrDNSfetcher disagreementfield datafinding agefinding fingerprintfinding stateFine-tuningfix listFixed-size chunkingFocus indicatorFocus orderforged crawler identityform control nameFoundation modelfounder edgefoundingDateFreshness boostingfreshness windowFunction calling

G

gated rowGeneral-purpose AI modelGEOgeocodingGeoCoordinatesGET requestGolden datasetGoogle Business ProfileGoogle-AgentGoogle-ExtendedGPTBot vs OAI-SearchBotgraded through a wallGraph mergeGraph normalizationGround truthgrounding

H

hallucinated attributionhallucinationHard bindinghard bouncehash-prefix lookupHEAD requestHeader parityHeading hierarchyheadroomHistorical optimizationHNSWhomepage refusedhoneypothost duplicateHost politenesshreflangHSTSHSTS preload listHTTP Message SignaturesHuman evaluationHuman-in-the-loopHuman-on-the-loopHybrid searchHyDEHydration mismatch

I

Identification stringIf-None-Matchimage placeholderImage sitemapImpact assessmentInbound linkindex coverageIndex freshnessIndex lagIndex selectionIndexing pipelineIndexNowIndirect prompt injectionInferenceInformation architectureinformation gainInformational queryIngredientINPInput validationInsecure output handlingInstruction tuningintent coverageInter-rater agreementInternal linkInverse propertyIRIItemList

J

JailbreakJSON-LDJSON-LD compactionJSON-LD expansionJSON-LD flatteningJSON-RPC

K

k-anonymityKeyboard accessibilityKnowledge distillationknowledge graphknowsAbout

L

lab datalandmarklandmark regionLanguage tagLast-ModifiedLate interactionLawful basisLayout-aware parsinglazy loadingLCPLeast privilegeLexical retrievalLighthouseLink contextLink equityLink extractionLink neighborhoodLiteralLLM-as-judgellms-full.txtllms.txtlocal citationlocal packLocalBusinesslog file analysisLong-term memoryLongest-match rule

M

machine layerMachine-readable opt-outmain landmarkmainEntityManifest storeManifest tamperingMCPMCP clientMCP hostMCP promptMCP resourceMCP serverMCP toolMCP transportMean reciprocal rankMerkle treeMetadata filteringmicroformats (h-card)MIME sniffingMinHashMixture of expertsModel cardModel driftModel routingModel versionMTA-STSmTLSMulti-agent systemMulti-query retrievalMultimodal model

N

name driftnamed entity recognitionNamed graphNAPNAP consistencyNavigational querynear-duplicateNear-duplicate detectionNews sitemapNewsArticleNofollownoindexnonexistent-path controlNormalized discounted cumulative gain

O

OAuth scopeObjectOCRoff-host redirectOfferOntologyOpen licenseOpenAPI specificationOpeningHoursSpecificationOpenTimestampsoperator feedoperator originOrganization schemaOrigin bypassOrigin serverorphan pageOrphaned entityOutput validationown-host loop

P

PageRankPairwise evaluationParameterParent-child retrievalparked domainParsingPassage rankingpayload ratiopayload text sharepercentile rankPerplexityBot vs Perplexity-UserPerson schemaPIIPillar pagePlaceplaceholder contentplaceholder in recordPlannerPointwise evaluationpopulation baselinePostalAddressPredicatePreference optimizationpreflightpresence ratePretrainingProductProduct tokenprompt injectionprompt setPublic domainpublication floorpublished crawler rangesPublisher relationshipPurpose limitation

Q

QuantizationQuery classQuery expansionQuery fan-outQuery rewriting

R

RAGrangeIncludesrate limitrate limitingRate-limit policyRCERDFRDF datasetRDFa and MicrodataRe-rankingRead timeoutReasoning tracereceipt headerReciprocal rank fusionreciprocated sameAsRecrawl intervalRed teamingRedirect chainRedirect loopRedirected internal linkReduced motionReferring domainrefusalregressionRelevance scoreRemote contextRemoval requestrender dependencerender dependence gaprender gaprender-blocking resourceRendered HTMLRendering queueRepeatabilityReproducibilityresidential proxyresolve answerResponse parityRetention periodretrievalretrieval crawlerRetrieval cutoffRetrieval latencyRetrieval precisionRetrieval recallReverse proxyReviewRights reservationRisk assessmentrival rootsRLHFRobots cacheRobots group mergingRobots parse errorRobots redirectRobots unavailablerobots.txtrow reviewRubricRUM

S

sameAsSample sizeSampling biasSchema propertySchema typescore versionSearch intentSecret leakagesecurity.txtSeed URLSelection biasSelf-attentionself-auditself-audit invariantSemantic cacheSemantic chunkingSemantic HTMLSemantic memorySemantic searchSensitive information disclosureServiceservice areaSHA-256share of voiceShared IP rangeshinglesigned agentsigned verdictSitemapSitemap hreflangsitemap indexSitemap lastmodsitemap origin errorSitemap URL setSKILL.mdskip linkSmall language modelSoft 404soft 404 on a machine fileSoft bindingSource diversitySparse vectorspeakableSPFSponsored linkSSRFstable @idstaging leakStale responseStale-while-revalidateStatus messagestructured datastructured data validationStructured outputSubjectSubprocessorSupervised fine-tuningSurvivorship biasSystem promptSystemic risk

T

TaxonomyTDM exceptiontdmrep.jsonTemperatureTenant isolationTest contaminationText and data miningthin contentThreat modelTLS handshakeTLS-RPTtokenTokenizerTool callTool definitionTool descriptionTool permissionTool resultTool schemaTool selectionTop-kTop-p samplingTopic clustertopical authoritytopicalitytraining dataTraining-data summaryTransactional queryTransformative useTransformertripleTrue negativeTrue positiveTrust boundarytrust centerTrust listTTFB

U

uniform refusalunreciprocated profileunscored measurementunverifiableURIURLURL discoveryURL inspectionUsage preferenceuser agentuser-agent groupUser-generated linkuser-triggered fetchuser-triggered fetch exemption

V

vantage pointVarianceVary headerVector databaseVector indexverified crawlerVideo sitemapVision-language modelvisitor resolveVocabulary term

W

WAFWeb Bot AuthWebMCPWebPagewell-known URIWikidataWildcard ruleWorking memory

X

X-Robots-Tag

What a glossary cannot settle

No engine publishes its ranking function, so every definition here describing influence is a description of a mechanism, not a lever. Terms marked as proposals — llms.txt, agents.md, entity maps — are bets with a low cost, not standards anyone has committed to reading. Where adoption figures exist they are on the dataset page, with the population they were counted against.

Questions about this page

QWhat is an AI visibility score?
A 0–100 measure of whether an answer engine can reach a site, read it and quote a fact about it, computed from scored sections in that order. It measures what a site controls; it does not measure whether an engine actually cites the site.
QWhat is llms.txt?
A plain-text or Markdown file at /llms.txt that names a site, what it does and the pages worth reading, for language-model agents. Adoption is low; the scanner reports it as absent when the path returns HTML or a challenge page with a 200.
QWhat is a verified crawler?
A request whose user-agent names a crawler and whose source IP falls inside the ranges that operator publishes. A user-agent alone is a claim; only the address can confirm it.
QWhy do the definitions state limits?
Because terms like citation, visibility and forged crawler are routinely used to mean more than any measurement supports. Each entry says what evidence the term can rest on.