What the top sites tell AI crawlers, and what they actually serve them.
A robots.txt rule is a statement. Serving the page is the act. CrawlCheck checks both for the same 9,776 sites, the Tranco top 10,000 frozen as one list. In the run that finished 2026-10-24, robots.txt closed the door to GPTBot on 644 sites, but 1,108 refused a client declaring GPTBot when it asked for the homepage.
20.0%
refuse GPTBot at the door
14.3%
close robots.txt to GPTBot
23.3%
refuse at least one AI identity
16.2%
serve /llms.txt
robots.txt percentages are of the 4,501 sites that serve a robots.txt; refusal and file percentages are of the 5,553 sites whose homepage answered a browser. The two columns count sites, not the same sites: a site can say one thing and do another in either direction.
Declared and served, crawler by crawler
| Crawler | robots.txt closes / | Refused at the door |
|---|---|---|
| GPTBot | 644 (14.3%) | 1,108 (20.0%) |
| ClaudeBot | 591 (13.1%) | 1,124 (20.2%) |
| OAI-SearchBot | 280 (6.2%) | 702 (12.6%) |
| PerplexityBot | 409 (9.1%) | 733 (13.2%) |
| Google-Extended | 516 (11.5%) | not requested as this identity |
| Applebot-Extended | 498 (11.1%) | not requested as this identity |
| CCBot | 679 (15.1%) | not requested as this identity |
| Bytespider | 659 (14.6%) | not requested as this identity |
| Meta-ExternalAgent | 527 (11.7%) | not requested as this identity |
| Amazonbot | 501 (11.1%) | not requested as this identity |
| ChatGPT-User | 365 (8.1%) | not requested as this identity |
| Claude-SearchBot | 260 (5.8%) | not requested as this identity |
This week, top 1,000 (re-read daily)
| Measure, top 1,000 | Change | Sites |
|---|---|---|
| At least one AI identity is refused (non-2xx) where the browser got 2xx | +1.4 pts | +8 / −1 |
| ClaudeBot refused where the browser got 2xx | +1.2 pts | +6 / −0 |
| Homepage answers a browser (2xx after redirects) | -0.7 pts | +3 / −10 |
| PerplexityBot refused where the browser got 2xx | +0.4 pts | +4 / −2 |
| OAI-SearchBot refused where the browser got 2xx | +0.4 pts | +3 / −1 |
How it is measured
- One frozen list: the top 10,000 registrable domains of one Tranco list, so every run measures the same sites. Hosts the publish filter refuses and opted-out domains are counted, never named.
- Each homepage is requested as a browser and as each crawler’s user-agent string, from CrawlCheck’s own addresses. A refusal is a non-2xx answer to the crawler identity where the browser got 2xx. A firewall that checks the real crawler’s IP ranges may treat the real crawler differently: these figures describe what a client declaring that name received. Why it is done this way.
- robots.txt is read and resolved per RFC 9309 for each agent; “closes /” means the agent’s own group, or the wildcard group when it has none, disallows the homepage.
- No site is named on this page. Per-site records are public and signed, one site at a time.
The data
| What | Where |
|---|---|
| This summary, as JSON | /api/census?frame=top10k |
| Weekly editions, signed and sealed | /data/census/weekly |
| Feed of new editions | Atom feed |
| Embeddings of every catalogue | /data/embeddings |
| Crawler IP-range history | /data/ip-ranges |
| One site’s signed record | /api/census/records?frame=top10k&domain= |
Aggregate figures on this page are CC BY 4.0; cite “CrawlCheck census” with the run date. Read by CrawlCheck Research and the CrawlCheck census.