Reference

Machine-layer glossary

34 terms this scanner uses, each defined with the limit of what it can claim.

Definitions here are written against what CrawlCheck measures. Where a term is commonly used to mean more than it can support — a citation, a forged crawler, a passing file — the definition says so rather than repeating the looser sense.

llms.txt

A plain-text file at the root of a site that tells language models what the site is and which pages matter. It is a proposal, not a standard: no engine has committed to reading it, and publishing one is a low-cost bet rather than a ranking action.

agents.md

A markdown instruction file addressed to autonomous agents rather than crawlers, stating what a site does, what an agent may collect, and what it must never assert. Most files served at this path are platform defaults rather than anything the site owner wrote.

entity map

A pair of files - JSON for machines, an HTML twin for people - listing the entities a site claims to be about. The measurement that matters is not whether the pair exists but whether the two halves agree.

entity map parity

The share of entities named in a site's entity-map JSON that also appear in its rendered HTML twin. Anything present in one and missing from the other is a fact the machine record asserts and no reader can ever see.

reciprocated sameAs

A sameAs link that the destination profile returns. Counting sameAs links measures how many claims a site makes about its own identity; fetching each destination and looking for the domain measures how many of those claims anyone else confirms.

unverifiable

The third outcome, and the one most tools omit. A profile behind a login wall, a 429, or a shell page under 512 bytes can be neither confirmed nor accused, and folding it into either bucket produces a number that overstates or understates by construction.

forged crawler identity

A request whose user-agent names a known crawler while its source address falls outside the ranges that operator publishes. Verified against a published feed it is a fact, not an accusation; without a feed it is unverifiable.

verified crawler

A request whose claimed crawler identity is confirmed by checking the source address against the operator's own published IP ranges. Reverse DNS and user-agent matching are weaker: only the operator's feed is authoritative.

Content-Signal

A robots.txt directive expressing separate permissions for search indexing, AI input and AI training. It communicates intent to operators who choose to honour it and carries no enforcement of its own.

AI opt-out

Any published instruction asking that a site's content not be used for model training. It is a request rather than a control, and whether it is honoured varies by operator.

machine layer

Everything a site serves to non-human clients: robots.txt, sitemaps, llms.txt, agents.md, entity maps, structured data and response headers. A site can look finished to a visitor and be nearly empty at this layer.

structured data

Machine-readable statements about a page, usually JSON-LD, describing what the page is and what it is about. Validity is cheap to check; agreement with the visible page is the part that is usually wrong.

JSON-LD

The JSON serialisation of linked data, embedded in a script tag. It is the format search and answer engines read most reliably, which also means an error in it is an error stated confidently to machines.

stable @id

A durable identifier that lets separate statements be understood as being about the same thing. Two records sharing an @id are the same record twice; without one, every page's claims start again from nothing.

entity collision

Two distinct entities in a site's structured data sharing an identifier, or one entity split across identifiers. Either way a machine reading the graph cannot tell how many things are being described.

payload ratio

The share of a delivered page that is text a machine can quote, against markup, scripts and styling that it cannot. A page can be large and still carry almost nothing an answer engine can use.

render dependence

How much of a page's meaning exists only after JavaScript runs. Crawlers that do not execute scripts see whatever the server sent, which on some sites is a shell.

cloaking

Serving materially different content to a crawler than to a browser. A server refusing a datacentre address while serving a residential one is not cloaking - it is IP-based access control, and calling it cloaking is a false accusation.

challenge page at 200

A bot-protection interstitial returned with a success status. Any validator that trusts the status code records the file as present and readable when nothing readable was served.

edge cache pinning

A cached response continuing to be served after the origin has changed. A purge API returning 200 is a statement about the API call, not evidence that anything was evicted.

TTFB

Time to first byte: how long a server takes to begin answering. It is the speed measurement that matters most to a crawler, because a crawler that times out records nothing at all.

Core Web Vitals

Google's field measurements of loading, interactivity and layout stability, drawn from real Chrome users. Two of the three describe rendering and interaction, which no AI crawler performs.

CrUX

The Chrome User Experience Report, which publishes real-user performance for origins with enough Chrome traffic. Absence of data is a statement about traffic volume, never about speed.

field data

Performance measured from real visits, as opposed to a laboratory run. It is the more honest number and it does not exist for most small sites.

answer engine

A system that returns a composed answer with citations rather than a list of links. Its retrieval step is what a site can influence; its generation step is not.

citation

A source an answer engine names when composing an answer. A citation proves a document was retrieved and used once; it does not describe the model's ranking and it decays as weighting changes.

retrieval

The step in which an answer engine selects documents to compose from. Everything a site controls acts here, which is why machine-readable clarity outperforms persuasion.

hallucinated attribution

An answer that credits a claim to a source which does not contain it. The defence is not more content but content that states its own scope, dates and limits.

dated series

The same measurement repeated on a schedule and kept. It is the one class of evidence that cannot be produced retroactively at any budget, because it exists only if something was already watching.

change receipt

An artifact proving what moved between two dated measurements, and that neither was edited afterwards. It proves existence, integrity and difference - never that either measurement was correct or that the change caused anything.

cohort asymmetry

The gap between how many sites look clean today and how many were broken at least once during a window. A single crawl at any budget can only produce the first number.

population baseline

What share of a measured population does the thing being scored. Without it, a finding says a file is missing and cannot say whether missing is normal or unusual.

self-audit invariant

A check comparing two of a scanner's own outputs to catch contradictions inside its own record. Every one worth encoding is a wrong answer that was actually shipped.

unscored measurement

A finding reported without moving the grade, used where the population is not yet measured or where the applicability of the check depends on the site. Scoring a site for lacking something it has no reason to have is the most common defect in automated audits.