Findings · 2026-11-03 · By VSNARY | Emmanuel Orta

One Cloudflare setting is behind most of the new AI training blocks on top sites

Between 3 October and 1 November 2026, about 120 of the top 10,000 sites started blocking AI training crawlers in robots.txt. Three in four did it the same way: by switching on Cloudflare’s managed robots.txt. AI search crawlers were left alone.

CrawlCheck reads the robots.txt of every site in the Tranco top 10,000 and records, for 12 named AI crawlers, whether the homepage is open to it. Comparing the run that finished on 3 October 2026 with the run that started on 1 November, among the 5,175 sites that served a robots.txt both times, one change stands out: blocks on AI training crawlers rose by roughly the same number of sites for every crawler, while blocks on AI search crawlers stayed flat.

What changed in a month #

For each training crawler, the number of sites whose robots.txt was open to it on 3 October and closed on 1 November, and how many of those sites added Cloudflare’s managed block in between:

The same eight crawlers, in the same order of magnitude, is the fingerprint of one source. Cloudflare’s managed robots.txt adds a block marked BEGIN Cloudflare Managed content that disallows exactly these eight: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and Meta-ExternalAgent. 20 of the 5,175 sites carried that block on 3 October; 140 carried it on 1 November. 121 added it and 1 removed it.

Search was left open #

Cloudflare’s managed block does not list AI search crawlers, and the sites did not block them separately either:

Of the 121 sites that switched the managed block on, 7 also blocked OAI-SearchBot. The other 114 now ask AI companies not to train on their pages while still allowing the crawlers that put them in AI search answers. Every one of the 121 also carries Cloudflare’s Content-Signal line, Content-Signal: search=yes,ai-train=no,use=reference, which says the same thing in words.

What did not move: the door #

The census also fetches every homepage as a browser and as named AI crawlers. Over the same month, the share of sites that turn a crawler away at the server, while serving a browser normally, stayed where it was (within one point for each crawler). So the new robots.txt rules are requests, added on top of server behaviour that was already there. On 1 November the two layers looked like this:

For every crawler the server refuses more sites than robots.txt asks to stay out. The gap is widest for OAI-SearchBot, the crawler that decides whether a site can appear in ChatGPT search: 6.2% of robots.txt files close the door to it, but 12.7% of servers turn it away. A firewall or CDN rule can answer before robots.txt is ever read. Figures are from CrawlCheck’s own addresses, so a refusal shows what a site does to that crawler name, not necessarily to the operator’s own network. The chart is free to reuse under CC BY 4.0: Wikimedia Commons.

What this means for a site owner #

• If you turned on Cloudflare’s managed robots.txt, it is doing what it says: asking eight training crawlers to stay out and leaving search crawlers alone. Check that your own rules below it still say what you meant, because the managed block is added in front of them.
• If you want to appear in AI search answers, robots.txt is only half of it. Check that your server or firewall does not refuse OAI-SearchBot, Claude-SearchBot and PerplexityBot even when robots.txt allows them.
• Robots.txt is a request. Cloudflare’s own documentation says the managed block expresses a preference and does not technically stop a crawler; enforcement is a separate setting.

How this was measured #

Two complete census runs of the Tranco top 10,000: run 1 (3 October 2026) and run 6 (1 to 2 November 2026). robots.txt was read once per run and each crawler’s access to the homepage decided under RFC 9309. A site counts as “newly blocked” for a crawler when its robots.txt was open to it in run 1 and closed in run 6. “Through Cloudflare’s managed block” means the run 6 file carries Cloudflare’s managed-content marker and the run 1 file did not. We checked a sample by hand against the live files on 3 November: the marked sites still carried the block, except one that had removed it again since 1 November. The census page reports month-on-month change on a slightly narrower base (sites whose homepage also answered both times), so its counts are a little lower; the direction and size are the same. We cannot see from outside why so many sites switched the setting on in October; Cloudflare’s documentation describes it as off unless a site owner turns it on.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

Why did so many top sites block AI crawlers in October 2026?

Most of the new blocks came from one setting: between 3 October and 1 November 2026, 121 of the 5,175 top-10,000 sites that served a robots.txt switched on Cloudflare’s managed robots.txt, which disallows eight AI training crawlers. It accounts for 90% of the new GPTBot blocks in CrawlCheck’s census.

Does Cloudflare’s managed robots.txt block AI search crawlers?

No. Its block lists eight training crawlers (Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and Meta-ExternalAgent) and leaves AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot open. Its Content-Signal line reads search=yes,ai-train=no,use=reference.

Is robots.txt enough to stay out of AI training?

No. robots.txt asks; it does not stop a crawler that ignores it. Cloudflare’s documentation says the same of its managed robots.txt and offers enforcement as a separate setting.

Related findings

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works