Findings · 2026-10-26 · By VSNARY | Emmanuel Orta

Which AI crawlers the top 10,000 sites block most, October 2026

CCBot, Bytespider and GPTBot are closed off most often in robots.txt; AI search crawlers far less. And at the door, a fifth of top sites refuse GPTBot and ClaudeBot whatever robots.txt says.

The robots.txt of a top site is a public statement about which AI crawlers it lets in. CrawlCheck reads that statement for the Tranco top 10,000 every week, and also asks for each homepage as a browser and as named AI crawlers, so the statement can be checked against what actually happens. These are the figures from the run that finished on 24 October 2026.

The ranking #

Of 4,501 sites serving a robots.txt, the share whose rules close the homepage to each crawler:

892 sites, 19.8%, close the homepage to at least one of the 12.

Training is closed off twice as often as search #

OpenAI and Anthropic each run a training crawler and a separate search crawler. Top sites treat them differently: GPTBot is closed off on 644 sites and OAI-SearchBot on 280; ClaudeBot on 591 and Claude-SearchBot on 260. Most of the sites that block training leave the door open to being found and cited in AI answers.

The door is stricter than the file #

robots.txt is not where most refusals happen. Of the 5,553 sites whose homepage answered a browser, 1,108 (20.0%) answered a client declaring GPTBot with a non-2xx status, against 644 that close the homepage to it in robots.txt. For ClaudeBot it was 1,124 (20.2%), for PerplexityBot 733 (13.2%) and for OAI-SearchBot 702 (12.6%). In all, 1,295 sites (23.3%) refused at least one AI identity the browser got through. A firewall or bot-management rule answers before any crawler gets as far as reading robots.txt.

These requests come from CrawlCheck’s own addresses with the crawler’s user-agent string, never from the operator’s published ranges, so a rule that checks addresses may treat the real crawler differently. What the figures show is what a client declaring that name received.

Few sites say anything finer-grained #

A Content-Signal line in robots.txt can separate search, AI input and AI training without naming any crawler. Only 192 sites (4.3% of those with a robots.txt) carry one; 133 of them say ai-train=no. 898 sites (16.2% of those that answered) serve an /llms.txt.

Method #

One frozen list: the top 10,000 registrable domains of one Tranco list, measured weekly, the top 1,000 daily. robots.txt is resolved per crawler per RFC 9309: “closes the homepage” means the crawler’s own group, or the * group when it has none, disallows /. No site is named. The figures, every week, and per-crawler comparisons: the radar and crawlers compared. Data: /api/census?frame=top10k, CC BY 4.0.

Questions this post answers

Which AI crawler is blocked most often?

In CrawlCheck's October 2026 census of the top 10,000 sites, CCBot: 15.1% of sites serving a robots.txt close the homepage to it, ahead of Bytespider (14.6%) and GPTBot (14.3%).

Do top sites block AI search crawlers?

Less often than training crawlers: 6.2% close the homepage to OAI-SearchBot and 5.8% to Claude-SearchBot, against 14.3% for GPTBot and 13.1% for ClaudeBot.

Why do more sites refuse GPTBot than block it in robots.txt?

Firewalls and bot-management rules answer before robots.txt is read. 20.0% of top sites that answered a browser refused a client declaring GPTBot, against 14.3% of sites with a robots.txt that close the homepage to it there.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works