Agent view · Findings · 2026-11-03

One Cloudflare setting is behind most of the new AI training blocks on top sites

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/one-cloudflare-setting-behind-most-new-ai-training-blocks-november-2026: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

Findings · 2026-11-03 · By [VSNARY | Emmanuel Orta](/author)

# One Cloudflare setting is behind most of the new AI training blocks on top sites

Between 3 October and 1 November 2026, about 120 of the top 10,000 sites started blocking AI training crawlers in robots.txt. Three in four did it the same way: by switching on Cloudflare’s managed robots.txt. AI search crawlers were left alone.

Comparing CrawlCheck’s census runs of 3 October and 1 November 2026 across the 5,175 top-10,000 sites that served a robots.txt both times, 118 sites newly blocked GPTBot and 130 newly blocked ClaudeBot. 121 sites added Cloudflare’s managed robots.txt block in the same month (20 had it on 3 October, 140 on 1 November), and it accounts for 90% of the new GPTBot blocks and 74% to 84% of the new blocks on the other seven training crawlers it lists. AI search crawlers barely moved: 9 sites newly blocked OAI-SearchBot.

On this page

- What changed in a month
- Search was left open
- What did not move: the door
- What this means for a site owner
- How this was measured

CrawlCheck reads the [robots.txt](/glossary/robots-txt) of every site in the Tranco top 10,000 and records, for 12 named AI crawlers, whether the homepage is open to it. Comparing the run that finished on 3 October 2026 with the run that started on 1 November, among the 5,175 sites that served a robots.txt both times, one change stands out: blocks on AI **training** crawlers rose by roughly the same number of sites for every crawler, while blocks on AI **search** crawlers stayed flat.

## What changed in a month #

For each training crawler, the number of sites whose robots.txt was open to it on 3 October and closed on 1 November, and how many of those sites added Cloudflare’s managed block in between:

| Crawler | Operator | Sites that newly blocked it | Of those, through Cloudflare’s managed block |
| --- | --- | --- | --- |
| GPTBot | OpenAI | 118 | 106 (90%) |
| ClaudeBot | Anthropic | 130 | 109 (84%) |
| [CCBot](/glossary/ccbot) | Common Crawl | 142 | 109 (77%) |
| [Google-Extended](/glossary/google-extended) | Google | 147 | 111 (76%) |
| Applebot-Extended | Apple | 137 | 111 (81%) |
| Amazonbot | Amazon | 153 | 113 (74%) |
| [Bytespider](/glossary/bytespider) | ByteDance | 139 | 110 (79%) |
| Meta-ExternalAgent | Meta | 146 | 110 (75%) |

The same eight crawlers, in the same order of magnitude, is the fingerprint of one source. Cloudflare’s [managed robots.txt](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt) adds a block marked `BEGIN Cloudflare Managed content` that disallows exactly these eight: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and Meta-ExternalAgent. 20 of the 5,175 sites carried that block on 3 October; 140 carried it on 1 November. 121 added it and 1 removed it.

## Search was left open #

Cloudflare’s managed block does not list AI search crawlers, and the sites did not block them separately either:

| Crawler | Operator | Sites that newly blocked it |
| --- | --- | --- |
| OAI-SearchBot | OpenAI | 9 |
| Claude-SearchBot | Anthropic | 12 |
| PerplexityBot | Perplexity | 28 |
| ChatGPT-User | OpenAI | 15 |

Of the 121 sites that switched the managed block on, 7 also blocked OAI-SearchBot. The other 114 now ask AI companies not to train on their pages while still allowing the crawlers that put them in AI search answers. Every one of the 121 also carries Cloudflare’s [Content-Signal](/glossary/content-signal) line, `Content-Signal: search=yes,ai-train=no,use=reference`, which says the same thing in words.

## What did not move: the door #

The census also fetches every homepage as a browser and as named AI crawlers. Over the same month, the share of sites that turn a crawler away at the server, while serving a browser normally, stayed where it was (within one point for each crawler). So the new robots.txt rules are requests, added on top of server behaviour that was already there. On 1 November the two layers looked like this:

| Crawler | robots.txt asks it to stay out | Server turns it away |
| --- | --- | --- |
| GPTBot | 16.2% | 19.5% |
| ClaudeBot | 15.1% | 19.8% |
| PerplexityBot | 9.3% | 13.0% |
| OAI-SearchBot | 6.2% | 12.7% |

For every crawler the server refuses more sites than robots.txt asks to stay out. The gap is widest for OAI-SearchBot, the crawler that decides whether a site can appear in ChatGPT search: 6.2% of robots.txt files close the door to it, but 12.7% of servers turn it away. A firewall or [CDN](/glossary/cdn) rule can answer before robots.txt is ever read. Figures are from CrawlCheck’s own addresses, so a [refusal](/glossary/refusal) shows what a site does to that crawler name, not necessarily to the operator’s own network. The chart is free to reuse under CC BY 4.0: [Wikimedia Commons](https://commons.wikimedia.org/wiki/File:AI_crawlers,_robots.txt_disallow_vs_server_refusal,_top_10,000_sites,_November_2026.svg).

## What this means for a site owner #

- **If you turned on Cloudflare’s managed robots.txt,** it is doing what it says: asking eight training crawlers to stay out and leaving search crawlers alone. Check that your own rules below it still say what you meant, because the managed block is added in front of them.
- **If you want to appear in AI search answers,** robots.txt is only half of it. Check that your server or firewall does not refuse OAI-SearchBot, Claude-SearchBot and PerplexityBot even when robots.txt allows them.
- **Robots.txt is a request.** Cloudflare’s own documentation says the managed block expresses a preference and does not technically stop a crawler; enforcement is a separate setting.

## How this was measured #

Two complete census runs of the Tranco top 10,000: run 1 (3 October 2026) and run 6 (1 to 2 November 2026). robots.txt was read once per run and each crawler’s access to the homepage decided under RFC 9309. A site counts as “newly blocked” for a crawler when its robots.txt was open to it in run 1 and closed in run 6. “Through Cloudflare’s managed block” means the run 6 file carries Cloudflare’s managed-content marker and the run 1 file did not. We checked a sample by hand against the live files on 3 November: the marked sites still carried the block, except one that had removed it again since 1 November. The census page reports month-on-month change on a slightly narrower base (sites whose homepage also answered both times), so its counts are a little lower; the direction and size are the same. We cannot see from outside why so many sites switched the setting on in October; Cloudflare’s documentation describes it as off unless a site owner turns it on.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fone-cloudflare-setting-behind-most-new-ai-training-blocks-november-2026%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fone-cloudflare-setting-behind-most-new-ai-training-blocks-november-2026%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=One%20Cloudflare%20setting%20is%20behind%20most%20of%20the%20new%20AI%20training%20blocks%20on%20top%20sites)[Email](mailto:?subject=One%20Cloudflare%20setting%20is%20behind%20most%20of%20the%20new%20AI%20training%20blocks%20on%20top%20sites&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fone-cloudflare-setting-behind-most-new-ai-training-blocks-november-2026)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### Why did so many top sites block AI crawlers in October 2026?

Most of the new blocks came from one setting: between 3 October and 1 November 2026, 121 of the 5,175 top-10,000 sites that served a robots.txt switched on Cloudflare’s managed robots.txt, which disallows eight AI training crawlers. It accounts for 90% of the new GPTBot blocks in CrawlCheck’s census.

### Does Cloudflare’s managed robots.txt block AI search crawlers?

No. Its block lists eight training crawlers (Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and Meta-ExternalAgent) and leaves AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot open. Its Content-Signal line reads search=yes,ai-train=no,use=reference.

### Is robots.txt enough to stay out of AI training?

No. robots.txt asks; it does not stop a crawler that ignores it. Cloudflare’s documentation says the same of its managed robots.txt and offers enforcement as a separate setting.

## Related findings

[2026-10-26 Which AI crawlers the top 10,000 sites block most, October 2026 Top 10,000 sites, October 2026: robots.txt block rates for 12 AI crawlers, how many refuse them at the door, and why training is b](/blog/which-ai-crawlers-top-sites-block-most-october-2026)[2026-10-02 Content-Signal in robots.txt: search, ai-input, ai-train Content-Signal states preferences for search, ai-input and ai-train. It blocks nothing. Refusing ai-train costs no reach; refusing](/guides/content-signals-in-robots-txt-search-ai-input-ai-train)[2026-08-28 Is Cloudflare blocking AI crawlers on your site? Since 15 September 2026 Cloudflare blocks Training crawlers by default, which can take Googlebot and Bingbot with them. How to che](/guides/is-cloudflare-blocking-ai-crawlers-on-your-site)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit, and placement is never for sale.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [Status](/status) · [SDK](/sdk) · [Certificates](/certificates) | **Registry:** [MCP: Operator](/registry/mcp/operator) · [abcnews.go.com](/registry/abcnews.go.com) · [centralbuickgmctruck.com](/registry/centralbuickgmctruck.com) · [drom.ru](/registry/drom.ru) | **Glossary:** [External link](/glossary/external-link) · [Secret leakage](/glossary/secret-leakage) · [Claim signature](/glossary/claim-signature) · [homoglyph](/glossary/homoglyph) · [HTTPS DNS record](/glossary/https-dns-record)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Report an error](/report-error)

Same body as a file: /blog/one-cloudflare-setting-behind-most-new-ai-training-blocks-november-2026.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.