CrawlCheck
Learn · guides, findings, glossary

How do you check a site for the agentic web?

In one sentence

The CrawlCheck library is the guides, findings and glossary behind the scanner: each guide checks one thing on any site and reports how often it fails across every scan, each finding is a dated case from a real measurement, and each glossary term is defined once and linked from the report that measures it.

64 guides, 93 measured findings and a glossary of 751 terms — each written from a measurement, with the rate at which the thing occurs across every scan.

Nothing here is advice from a survey. A guide tells you how to check one thing on any site and what the check usually finds; a finding is a case from a real scan with the bytes behind it; a term is defined once and linked from the report that measures it.

Paying never changes a reading. The rulebook · The crawlers · How a scan works

Signed answer · a public site, measured daily · observed 12 days agodenverpost.comreadwarncitewarnconnectnone declaredtransactneeds a human8 of 9 AI-crawler identities were served the page; robots.txt allows 73 of 114 tested agentsCrawl policydeclared · 73 of 114 agents allowedDeliveryobserved · 14 identities, 1 refusedMachine filesobserved · 5 read, bodies hashedFindings8 open · 4 seriousEd25519 · verifies offlineOpen the signed answer →
Three paths down

Guides. Findings. Glossary.

Guides

How to check one thing.

Each guide is one check: what to fetch, what to look for, what it means, and how often it fails across every scan. Start with the three that explain how to read a report.

All guides · Start here

Findings

What real scans found.

Dated case studies from the public dataset: a robots.txt that could not be read, a crawler name Google says it never sends, a page that served two businesses one identity.

All findings · The data behind them

Glossary

Every term, defined once.

751 terms across the machine layer, each with the page that measures it. The definitions are the ones the report and the API use.

The glossary · Agentic web · Rules

The library, counted

Every number here is read from the manifest that publishes it.

64guidesHow to check one thing, with the rate at which it fails across every scan.Guides →93measured findingsCase studies from real scans: what was served, what was declared, what differed.Findings →751glossary termsEvery term the product uses, defined once and linked from where it is measured.Glossary →86published rulesWhat each finding tests, its severity, and how to fix it. Computed from the live service.The rulebook →15crawler identitiesWho is fetching the web for agents and answer engines, and how each one is tested.The crawlers →
Start here

Three guides that explain the report.

By topic

What the guides cover.

Latest findings

Measured this month.

All findings →

Questions about this page

QWhere do I start?
With the three guides that explain how to read a report: how to read a CrawlCheck report, how to check whether AI crawlers can read your site, and how to measure AI visibility. Each is a check you can run on any domain today.
QWhat is a finding, as opposed to a guide?
A finding is a dated case from a real scan — what was served, what was declared, what differed — with the bytes behind it. A guide is the reusable check that case taught. Findings never name a site before contact has been made.
QAre the rates in the guides real?
Yes. The rate printed in a guide is the share of scans in the public dataset that carried that finding, read live from the same counts the data pages publish.
QIs the glossary the same vocabulary the product uses?
Yes. A term is defined once; the report, the API and the specifications use that definition and link back to it.