Datasets · AI consent

What every site tells AI crawlers, as signed data

For each site we read: what its robots.txt allows each of 16 named AI agents to do, its Content-Signal line word for word, whether it serves llms.txt, and the time we read it. Every release is signed, so you can prove later what a site said on the day you trained or answered.

Ask for a licence Free sample (1,000 sites)Signed sample manifest · verify it yourself

102,567 sites in the current release
4,097 close the homepage to at least one panel agent
406 carry a Content-Signal line
14,563 serve llms.txt

The 16 agents in every row

Share of sites with a readable answer that close the homepage to each agent, in the current release (reads to 2026-10-17).

Agent	Company	Use	Sites closed	Share
`GPTBot`	OpenAI	training	2,278	3%
`OAI-SearchBot`	OpenAI	search	799	1.1%
`ChatGPT-User`	OpenAI	user fetch	1,186	1.6%
`ClaudeBot`	Anthropic	training	1,694	2.2%
`Claude-SearchBot`	Anthropic	search	456	0.6%
`Claude-User`	Anthropic	user fetch	419	0.6%
`Google-Extended`	Google	training	1,367	1.8%
`PerplexityBot`	Perplexity	search	1,107	1.5%
`Perplexity-User`	Perplexity	user fetch	419	0.6%
`Applebot-Extended`	Apple	training	1,264	1.7%
`CCBot`	Common Crawl	training	2,135	2.8%
`meta-externalagent`	Meta	training	1,343	1.8%
`Bytespider`	ByteDance	training	2,126	2.8%
`cohere-ai`	Cohere	training	790	1%
`MistralAI-User`	Mistral	user fetch	366	0.5%
`DeepSeekBot`	DeepSeek	training	464	0.6%

Who it is for

AI companies: a per-site record of what each site allowed your crawlers when you read it, to train, cite and answer within it.
Licensing and legal teams: signed evidence of a site’s stated policy on a date, checkable without asking us.
Researchers: how consent to AI use spreads across software, government, local businesses and the wider web, week by week.

Fields

One row per site: its newest read in the last 7 days. NDJSON, one file per lane.

Field	What it holds
`domain`	Registrable domain
`read_at`	When robots.txt and the homepage were read (UTC, ISO 8601)
`lane`	saas, gov, local or filler
`kind`	Site kind from the page
`robots`	read = robots.txt parsed; unread = absent or unreadable when read (the three agent lists are then empty: an absent file allows every agent, an unreadable one cannot be judged)
`blocked`	Panel agents robots.txt closes the homepage to
`allowed`	Panel agents robots.txt lets read the homepage
`unknown`	Panel agents not determinable from this read (older rows kept only the first 20 closed agents)
`ai_closed_all`	Every AI agent closed, of the ~60 AI agents tested (rows read from 16 Oct 2026)
`content_signal`	The Content-Signal line, verbatim, when one was read
`signal_ai_train`	yes, no, or empty
`signal_ai_input`	yes, no, or empty
`signal_search`	yes, no, or empty
`llms_txt`	Serves /llms.txt

Verify a release

Each release has a manifest listing every file with its SHA-256. The manifest is signed with Ed25519 by the documents key published at /.well-known/http-message-signatures-directory. This script checks the signature and the files with no call to us beyond fetching the public key:

// verify.mjs: node verify.mjs sample.manifest.json sample.ndjson
import fs from "node:fs"; import crypto from "node:crypto";
const canon = (v) => v === null || typeof v !== "object" ? JSON.stringify(v) : Array.isArray(v) ? "[" + v.map(canon).join(",") + "]" : "{" + Object.keys(v).sort().map((k) => JSON.stringify(k) + ":" + canon(v[k])).join(",") + "}";
const m = JSON.parse(fs.readFileSync(process.argv[2], "utf8")), s = m.signature; delete m.signature;
const h = crypto.createHash("sha256").update(canon(m)).digest("hex");
const dir = await (await fetch("https://crawlcheck.io/.well-known/http-message-signatures-directory")).json();
const jwk = dir.keys.find((k) => k.kid === s.kid);
const ok = !!jwk && h === s.manifest_sha256 && crypto.verify(null, Buffer.from(s.message), crypto.createPublicKey({ key: jwk, format: "jwk" }), Buffer.from(s.sig, "base64url"));
console.log("manifest signature:", ok ? "valid" : "INVALID");
for (const f of process.argv.slice(3)) { const p = m.parts.find((x) => f.endsWith(x.file)); const d = crypto.createHash("sha256").update(fs.readFileSync(f)).digest("hex"); console.log(f + ":", p && p.sha256 === d ? "matches the manifest" : "DOES NOT MATCH"); }

Licence

From $25,000 to $100,000 a year, by scope: the lanes covered, how often you take a release, and whether you redistribute. A contract and an invoice; the key opens the full files and every daily release for the term.

Sites that opt out of CrawlCheck leave every file at read time. Local businesses are never in a public file. Data policy · terms.