CrawlCheck
The registry · what sites declare to agents

What a site declares to agents, verified by a fetch.

Before an agent trusts a site’s llms.txt, its agents.md, its press facts, its entity graph, its crawler policy or its callable surface, this registry says whether the thing is really served. Every one of the 913 sites here is listed because a full scan produced it: a graded report on a domain that can be published and has not opted out. The larger “sites measured” figure on the homepage also counts the census’s lighter readings, which get no registry row. Nothing is submitted, rows re-derive from each new reading, and no payment moves a shelf. Owners can prove a site is theirs, free, so every agent is told who stands behind it.

In one sentence

The CrawlCheck registry records what each measured website declares to AI agents and whether the declaration is really served, filed on six shelves a site can only earn by a fetch: a resolving llms.txt, a served agents.md, a machine-readable media kit at /.well-known/media-kit.json, an entity graph whose sameAs links point back, a robots.txt that names AI crawlers, and an agent-callable surface such as an API catalog, MCP server card or agent-skills index; the capability registry lists the operations those surfaces declare, by policy class. An agent reads it for one domain inside the signed Resolve answer, for every domain as JSON at /api/registry and in the daily snapshot, with no key; nothing is submitted, rows re-derive from each new reading, and no payment moves a shelf.

Register your site free →

913sites listed
6shelves, each earned by a fetch
368on the fullest shelf: Entity graph
0shelves a payment can move
For site owners

Make sure AI agents read your site, and know it’s yours

Free, no account, no card. Prove you run the site once, and every AI agent that asks about it is told in the signed answer it reads before it cites you, connects or buys.

1 · Prove it

One DNS record or one meta tag

Enter your domain and you get two strings. Put either one on the site; anyone can check the proof the same way we do. No account, no card.

2 · Agents are told

Verified owner in every signed answer

The Resolve answer agents read before they cite, connect or transact says control is proven, by whom and since when — and so does the CrawlCheck extension’s Agent view.

3 · Your record

A signed business record

Name, phone, address and service area, signed, versioned and logged, servable from your own site at /.well-known/crawlcheck-business.json. Check your record first →

One DNS record or one meta tag proves the site is yours. No account, no card. Not sure AI crawlers can read it? Check it free first →

For agents

Three ways to read it

Every call is public: no key, no account, nothing counted against a plan. Answers are signed and verify offline.

One site

Ask about a domain

The signed answer carries every shelf the site earned, alongside what it serves to crawlers and a decision for read, cite, connect and transact.

GET /api/v1/resolve?domain=example.com

What to call before acting →

What it can do

The capability registry

What sites declare an agent may do on them — the operations in their API catalogs, MCP server cards and agent-skills indexes — read from declarations, with the policy class each one falls in.

GET /api/registry/capabilities

The capability registry →

All of it

Rows, shelves and the daily snapshot

The whole registry as JSON with shelf counts, each shelf as its own page, and every public signed answer once a day in files a mirror can serve and verify.

GET /api/registry

The snapshot → · Open data →

Who earned each shelf
llms.txt37%338agents.md3.4%31Media kit0.8%7Entity graph40.3%368AI access policy34%310Agent-callable7.9%72Share of 913 measured sites on each shelf. A shelf is earned by a fetch, never by a submission.
Six shelves

What each shelf means

Each shelf has one test, run on every scan. A site is filed when the test passes and leaves when it stops passing. Reciprocity is tested, never declared.

Latest fetches

Recently fetched

Machine-readable: /api/registry. Rows re-derive from each new scan; a row older than 30 days is marked.

SiteGradeShelvesLast fetched
dremio.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
domino.aiCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
docusign.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
descript.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
cityofgolden.netCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
emmen3d.nlBllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
modalpoint.comAllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
ewrdigital.comAllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
deepset.aiCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
deepscribe.aiCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
deepmind.googleCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
deepgram.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
deepcrawl.comBllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
bestseopodcast.comAllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
matthewbertram.comAllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
decagon.aiCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
debugbear.comAllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
datarobot.comBllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
dataiku.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
datadoghq.comBllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
databricks.comBllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
darrow.aiCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
d-id.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
cursor.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
crusoe.aiCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
crewai.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
denverpost.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
cresta.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
credo.aiBllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
cradle.bioCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-27
coveo.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
corti.aiCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
coreweave.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
copyscape.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
copy.aiCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
convex.devCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
contentatscale.aiBllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
comet.comBllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
colossyan.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
cohere.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
codium.aiCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
logitech.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
itch.ioCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
pueblo.usCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
audible.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
cockroachlabs.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
clipdrop.coCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
example.orgDllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
who.intCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26
clickhouse.comCllms.txtagents.mdMedia kitEntity graphAI access policyAgent-callable2026-10-26

To be removed, put User-agent: CrawlCheck and Disallow: / in your robots.txt — it is honoured on the next fetch and the row goes with it, or write to the opt-out address.

Questions about this page

QHow does a site get on the registry?
By being scanned. Every domain CrawlCheck measures is listed, and it is filed on a shelf only when the fetch that defines that shelf succeeds — a resolving llms.txt, a served agents.md, a media kit, a reciprocated sameAs, a robots.txt that names AI crawlers, or an agent-callable surface. There is no submission form and nothing to pay.
QCan a site be removed?
Yes. Put User-agent: CrawlCheck and Disallow: / in robots.txt and the row goes on the next fetch, or write to the opt-out address on the data policy page.
QWhy does a row say unverified since?
Rows re-derive from each new scan. When a domain has not been fetched for 30 days the row is marked rather than silently kept, because an old reading is not evidence of the present.
QIs the registry available as data?
Yes: GET /api/registry returns the rows and shelf counts with no key, CORS open, and each shelf has its own page under /registry/.
Certificates

Certified sites (71)

Sites that met every CrawlCheck certificate criterion on their latest reading. Each certificate is signed and re-checked.

acquiredigital.com · activeloop.ai · aeotool.ai · ahrefs.com · anthropic.com · anyscale.com · app.alkhemyst-ai.cloud · arize.com · assemblyai.com · athenahq.ai · augmentcode.com · azpest.com · bandypuls.se · baseten.co · bbc.co.uk · beget.ru · bestseopodcast.com · blackberry.com · bolt.new · braintrust.dev · brand24.com · brandlight.ai · browserless.io · builder.io · byword.ai · calibreapp.com · cleanlab.ai · clickhouse.com · clipdrop.co · cloudflare.com · cockroachlabs.com · cohere.com · colossyan.com · comet.com · comviq.se · contentatscale.ai · convex.dev · copyscape.com · coveo.com · cradle.bio · crawlcheck.io · crewai.com · cursor.com · datadoghq.com · dataiku.com · debugbear.com · deepgram.com · deepmind.google · diffbot.com · dreamztech.com · emmanuelorta.com · emmen3d.nl · ewrdigital.com · firebolt.io · firecrawl.dev · genmo.ai · gmail.com · gong.io · grain.com · grammarly.com · groq.com · h2o.ai · harvard.edu · harvey.ai · hoppscotch.io · humanloop.com · huntclubauto.ca · inflection.ai · infralyte.in · ipify.org · terrariumstation.com

Check it yourself

Every number here comes from the signed JSON at /api/v1/instruments/registry/now.json; each copy carries its own sha256 and an Ed25519 signature by key 1hHmE3fo1rg5…. Verify it with no call to us:

node crawlcheck-verify.mjs --online registry.json

The verifier: /crawlcheck-verify.mjs, zero dependencies.

Editions

One sealed edition per ISO week, never edited. Feed: Atom · all instruments: /api/v1/instruments.