Guides · 2026-08-28 · By VSNARY | Emmanuel Orta
Is Cloudflare blocking AI crawlers on your site? What changes on 15 September 2026
A default is about to change, and the group it reaches hardest is the group that never opened the setting. How to find out which side of it you are on before the date, not after.
On 15 September 2026 Cloudflare's defaults for AI crawler access change. The headline part is simple enough: Training and Agent categories become blocked by default on monetised pages, while Search stays allowed. The part worth reading twice is the coupling. If a site blocks Training crawlers, multi-purpose crawlers — Googlebot, Applebot, Bingbot — are blocked too, even where Search crawlers are allowed, because those bots collect for both purposes and Cloudflare is no longer letting the dual role buy access under the search banner. Owners who do not want the new behaviour can opt out through Security settings before the date.
Two things follow. First, this is a live deadline, not a trend piece. Second, and more important for most readers: the population a default reaches is the population that never opened the setting. Nobody who deliberately configured their bot policy is affected by a default. Everybody who inherited one is.
Why this is easy to get wrong from the inside
An edge block does not look like a block from a browser. You visit your own site, it loads, everything is fine. The refusal happens to a different client, from a different address, and produces no error anyone on your team will ever see. On one site measured here, GPTBot and ClaudeBot both received HTTP 502 while Googlebot and PerplexityBot were served normally — with robots.txt explicitly allowing all four. Nothing in the site's own configuration said so. It was the edge, and almost nobody configures that layer deliberately.
That is the general shape of it: ANSWER_ENGINE_REFUSED fires on 3.4% of the scans in the public dataset, and 15.0% find an AI opt-out already set. A meaningful share of the second number was never a decision either. Web-wide, the eight largest opt-out groups sit within 12% of each other — AppleBot on 108,452 sites, GPTBot on 107,182, Common Crawl on 102,665, ClaudeBot on 101,917. Eight independent operators do not get blocked in near-lockstep by eight independent decisions. That is one copy-pasted block, one plugin default, one hosting toggle.
The three failure modes, and only one is a block
| What you see | What it is | Fires on |
|---|---|---|
| 403 or 502 for a crawler UA, 200 for a browser | An edge rule refusing by identity | 3.4% |
| 200 with an HTML challenge page | A bot defence answering in place of your file | 1.9% |
| 200 with a stale body | A cache holding something the origin no longer serves | 30.9% |
The second is the nastiest, because every uptime monitor calls it healthy. A robots.txt that answers 200 with a verification interstitial is parsed as no directives, which reads as everything allowed — the precise opposite of whatever the file said. And when a caching rule stores that response, a per-address defence becomes a site-wide outage of the machine layer, because a cache cannot tell that a 200 is a challenge page. CHALLENGE_PINNED_AT_EDGE fires on 1.4% of scans, ORIGIN_BLOCKED_BEHIND_CACHE on 0.7%.
How to check your own zone, in four requests
curl -sI https://yoursite.com/robots.txt | grep -i 'content-type\|cf-cache-status\|age'
Content type must be text/plain. Anything else means something is answering in place of your file. Then compare identities on the bare URL, never with a cache-buster, because a crawler never appends one:
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "Mozilla/5.0" https://yoursite.com/curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "GPTBot" https://yoursite.com/curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "ClaudeBot" https://yoursite.com/
Then read your zone's AI crawler controls directly — the dashboard exposes the Search, Agent and Training categories per zone, with per-crawler overrides and crawl analytics showing which bots actually arrived.
One honest caveat about the curl test, and most guides omit it. Your request sends a crawler's name from an address that operator does not publish. An edge that verifies identity properly should refuse it. So a refusal is not proof that Cloudflare is turning away OpenAI — it is proof that something at your edge treats that identity differently, which is worth knowing either way. Only your own logs, checked against the operator's published ranges, settle which one it is.
The parts of the change that got less coverage
Two further pieces matter more than the default itself for anyone thinking past September.
Content-use controls. Enterprise Bot Management customers can declare how bots may use content after crawling it, across three levels — Immediate, meaning no storage or reuse; Reference, meaning indexing, excerpts and links back; and Full, meaning summaries or reproduction. Cloudflare is extending the Content Signals vocabulary in robots.txt with a use parameter to express those preferences, and reporting whether verified bots comply. robots.txt still cannot enforce anything — but for the first time the compliance is being observed and published, which is a different lever than a block.
Attribution through intermediaries. Cloudflare added a transitive trust model that reads the HTTP Forwarded header to identify the original requester behind a proxy or browser-automation service, so policy can apply to the operator that initiated a request rather than the middleman carrying it. That is the piece to watch, because the entire agent economy is built on exactly that indirection, and identity-based rules have been trivially defeatable by one hop.
Both point the same direction: the useful unit of policy is shifting from who is asking to what they intend to do with it. A site owner who has only ever thought in terms of allow and block has no vocabulary for that distinction yet.
What to actually decide before the 15th
Blocking AI crawlers is a legitimate choice, and for some publishers it is the right one — the crawl-to-refer ratios behind the whole debate are genuinely lopsided. This page is not an argument for allowing everything. It is an argument for the decision being yours.
Three questions settle it. Do you want training crawlers, separately from search and agent crawlers? Are you prepared for the coupling — that blocking Training can take multi-purpose crawlers with it, including ones you rely on for search traffic? And if you allow them, has anyone verified they can actually complete a fetch, given that a third of sites in this dataset are serving crawlers a stale object regardless of policy?
A permission you granted and a request that succeeds are different things. Most sites that believe they allow answer engines have never checked the second.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.
Questions this post answers
Does Cloudflare block AI crawlers by default?
From 15 September 2026 the Training and Agent categories are blocked by default on monetised pages while Search remains allowed, and blocking Training also blocks multi-purpose crawlers such as Googlebot, Applebot and Bingbot. Owners can opt out through Security settings before that date. The change reaches sites that never opened the setting, not sites that configured a policy deliberately.
How do I check whether my site is blocking GPTBot or ClaudeBot?
Request your homepage on the bare URL with a browser user-agent and again with the crawler's user-agent, and compare status and byte count. Also check that robots.txt returns content type text/plain rather than HTML. A refusal in that test means your edge treats that identity differently; your server logs, checked against the operator's published IP ranges, confirm which.
My site loads fine, so nothing is blocked, right?
A browser is the client least likely to be refused. It executes JavaScript, clears challenges, carries cookies and often comes from a trusted address. Edge refusals apply to a different client from a different network and produce no error anyone on your team will see.