Guides · 2026-08-28 · By VSNARY | Emmanuel Orta
How to block AI crawlers, and what each block costs you
Blocking is a legitimate choice and this is how to do it properly. It is also three separate decisions that get made as one, and the one people regret is rarely the one they intended.
There are good reasons to block. The crawl-to-refer ratios behind the whole argument are lopsided — one platform's figure sat near 4,580 pages fetched for every visit sent back in mid-2026 — and if your business is publishing, giving that away for nothing is a real decision to weigh.
But block AI crawlers is three decisions, not one, and they have completely different costs.
| Category | What it does | Blocking it costs |
|---|---|---|
| Training | Collects text for model training | Nothing immediate; long-term presence in weights |
| Search indexing | Builds the index an assistant queries | Eligibility to be cited at all |
| User-triggered fetch | Fetches because someone asked about you now | The answer to a question about you, in the moment |
The third is the one blocked by accident most often. It is not a scraper harvesting your archive — it is a person asking an assistant about your business and the assistant going to look. Refusing it means the answer gets composed without you, from whatever else is available.
Layer 1: robots.txt
A request, not a control. Named crawlers from the major operators honour it; unnamed ones may not, and it has no enforcement power over anything.
Write it per-agent rather than copying a block wholesale, which is exactly how most sites ended up with a policy nobody chose. Web-wide, AppleBot is disallowed on 108,452 sites, GPTBot on 107,182, Common Crawl on 102,665 and ClaudeBot on 101,917 — eight independent operators blocked within 12% of each other, which is one copy-pasted snippet inherited by everyone downstream, not eight decisions. AI_OPTOUT_SET fires on 15.0% of scans here.
Two mistakes to avoid while editing. ROBOTS_RULES_SHADOWED fires on 1.3% of scans — a later group silently overriding an earlier one, so a rule you wrote never applies. And check the file still serves text/plain afterwards: a robots.txt returning HTML is parsed as no directives, which means everything allowed, the exact opposite of a block.
Layer 2: the network
This is where a block is actually enforced, against clients that never ask permission.
Block by verified identity, never by user-agent string. A rule that trusts a UA is a rule anyone satisfies by typing — and in this dataset 37% of checkable crawler claims came from outside the operator's published range, with some of that forged traffic requesting .env files and SSH keys while wearing a crawler's name. Allow-listing by user-agent hands those requests a pass.
If you are behind Cloudflare, the categories are exposed per zone and the defaults change on 15 September 2026: Training and Agent become blocked by default on monetised pages while Search stays allowed. Read the coupling before you set anything — blocking Training also blocks multi-purpose crawlers such as Googlebot, Applebot and Bingbot, which collect for both purposes. A block aimed at AI training can take your search traffic with it. Owners who do not want the new behaviour can opt out through Security settings before that date.
Layer 3: stating intent rather than access
The newer and more interesting option is declaring what may be done with content after it is fetched, rather than whether it may be fetched. Cloudflare's content-use controls express three levels — Immediate, meaning no storage or reuse; Reference, meaning indexing, excerpts and links back; and Full, meaning summaries or reproduction — with a use parameter extending the Content Signals vocabulary in robots.txt, and compliance by verified bots reported rather than enforced.
That is closer to what most site owners actually want. Very few want never mention us. Most want reference us and link back, do not train on us, which allow-and-block has no vocabulary for.
Check the block did what you meant
After any change, verify from outside your network, on the bare URL, as each identity:
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "GPTBot" https://yoursite.com/
Run it for every crawler you intended to allow as well as every one you intended to block. The common outcome is a policy that blocks more than intended, because a robots.txt block, a plugin default and an edge rule are three independent systems that nobody reads together. ANSWER_ENGINE_REFUSED fires on 3.4% of scans, and in most of those cases the refusal was not in the file the owner had been editing.
What a block does not do
Three limits worth being clear-eyed about before treating a block as protection.
It does not remove you from models already trained. Blocking a training crawler affects future collection, not weights that already exist.
It does not stop content reaching a model by another route. Your pages get quoted, syndicated and aggregated elsewhere, and those copies carry no policy of yours.
It does not stop anything that ignores you. robots.txt has no enforcement power, and an operator that publishes no verifiable IP range cannot be told apart from an impostor at the network layer either — which is why 37% of checkable crawler claims here came from outside a published range and five identities in the corpus have no feed to check against at all.
A block is a statement of intent that cooperative parties honour and a network rule that enforces against the ones you can identify. It is worth doing for those reasons and not worth mistaking for a wall.
The one thing worth keeping open
If you decide to block, consider allowing user-triggered fetches even while blocking training. Those requests exist because a real person is asking about you right now, and they are the cheapest attention on the internet: no crawl budget, no bulk harvesting, one page, in response to demand you already earned.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.
Questions this post answers
How do I block AI crawlers from my website?
Two layers. Write per-agent rules in robots.txt, which named crawlers honour but which enforces nothing. Then enforce at the network layer, blocking by verified identity rather than user-agent string, since a user-agent is a claim anyone can send.
Does blocking AI crawlers hurt my search traffic?
It can. From 15 September 2026, Cloudflare's defaults block Training and Agent crawlers on monetised pages, and blocking Training also blocks multi-purpose crawlers such as Googlebot, Applebot and Bingbot that collect for both search and training.
Should I block all AI crawlers?
Decide the three categories separately. Blocking training crawlers has no immediate cost. Blocking search indexing removes your eligibility to be cited. Blocking user-triggered fetches means that when someone asks an assistant about you, the answer is composed without you.