What a site declares to agents, verified by a fetch.
Before an agent trusts a site’s llms.txt, its agents.md, its press facts, its entity graph, its crawler policy or its callable surface, this registry says whether the thing is really served. Every one of the 695 sites here is listed because a fetch produced it; nothing is submitted, rows re-derive from each new reading, and no payment moves a shelf.
In one sentence
The CrawlCheck registry records what each measured website declares to AI agents and whether the declaration is really served, filed on six shelves a site can only earn by a fetch: a resolving llms.txt, a served agents.md, a machine-readable media kit at /.well-known/media-kit.json, an entity graph whose sameAs links point back, a robots.txt that names AI crawlers, and an agent-callable surface such as an API catalog, MCP server card or agent-skills index; the capability registry lists the operations those surfaces declare, by policy class. An agent reads it for one domain inside the signed Resolve answer, for every domain as JSON at /api/registry and in the daily snapshot, with no key; nothing is submitted, rows re-derive from each new reading, and no payment moves a shelf.
Three ways to read it
Every call is public: no key, no account, nothing counted against a plan. Answers are signed and verify offline.
One site
Ask about a domain
The signed answer carries every shelf the site earned, alongside what it serves to crawlers and a decision for read, cite, connect and transact.
GET /api/v1/resolve?domain=example.com
What it can do
The capability registry
What sites declare an agent may do on them — the operations in their API catalogs, MCP server cards and agent-skills indexes — read from declarations, with the policy class each one falls in.
GET /api/registry/capabilities
All of it
Rows, shelves and the daily snapshot
The whole registry as JSON with shelf counts, each shelf as its own page, and every public signed answer once a day in files a mirror can serve and verify.
GET /api/registry
What each shelf means
Each shelf has one test, run on every scan. A site is filed when the test passes and leaves when it stops passing. Reciprocity is tested, never declared.
llms.txt
266 of 695 · 38.3%The file resolves and its links point at pages that exist on the same host.Open the shelf →agents.md
38 of 695 · 5.5%An /agents.md is served and is not an HTML 404 dressed as one.Open the shelf →Media kit
13 of 695 · 1.9%A /.well-known/media-kit.json is served — machine-readable press facts, assets and usage terms.Open the shelf →Entity graph
300 of 695 · 43.2%At least one sameAs was fetched and found pointing back. Reciprocity tested, not declared.Open the shelf →AI access policy
235 of 695 · 33.8%robots.txt names AI crawlers by name rather than leaving them to the wildcard.Open the shelf →Agent-callable
62 of 695 · 8.9%An API catalog, MCP server card or agent-skills index is served.Open the shelf →
Recently fetched
Machine-readable: /api/registry. Rows re-derive from each new scan; a row older than 30 days is marked.
| Site | Grade | Shelves | Last fetched |
|---|---|---|---|
| producthunt.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| scrapingbee.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| scaleway.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| scalenut.com | B | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| treeservicedenverllc.com | A | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| scale.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| sambanova.ai | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| sama.com | B | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| rytr.me | D | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| runwayml.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| runpod.io | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| robinai.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| rev.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| retool.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| resemble.ai | B | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| requestmetrics.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| replicate.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| remove.bg | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| relixir.ai | D | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| reducto.ai | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| recursion.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| readme.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| read.ai | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| qure.ai | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| questdb.io | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| qodo.ai | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| qdrant.tech | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| protectai.com | D | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| example.net | D | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| example.org | D | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| example.com | D | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-08 |
| pagespeed.web.dev | D | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| pactsafe.com | D | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| oxylabs.io | B | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| owkin.com | B | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| outranking.io | D | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| otter.ai | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| opus.pro | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| opensearch.org | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| yerainabreu.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| observe.ai | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| kentcdodds.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| sindresorhus.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| rauchg.com | D | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| stephango.com | B | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| newrelic.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| nullprogram.com | D | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| danluu.com | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| n8n.io | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
| mostly.ai | C | llms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable | 2026-10-07 |
To be removed, put User-agent: CrawlCheck and Disallow: / in your robots.txt — it is honoured on the next fetch and the row goes with it, or write to the opt-out address.