Guides · 2026-08-28 · By VSNARY | Emmanuel Orta · 3 views

How to block AI crawlers and agents, and what each block costs you

Blocking is a legitimate choice and this is how to do it properly. It is also three separate decisions that get made as one, and the one people regret is rarely the one they intended.

There are good reasons to block. The crawl-to-refer ratios behind the whole argument are lopsided — one platform's figure sat near 4,580 pages fetched for every visit sent back in mid-2026 — and if your business is publishing, giving that away for nothing is a real decision to weigh.

But block AI crawlers is three decisions, not one, and they have completely different costs.

The third is the one blocked by accident most often. It is not a scraper harvesting your archive — it is a person asking an assistant about your business and the assistant going to look. Refusing it means the answer gets composed without you, from whatever else is available.

Layer 1: robots.txt #

A request, not a control. Named crawlers from the major operators honour it; unnamed ones may not, and it has no enforcement power over anything.

Write it per-agent rather than copying a block wholesale, which is exactly how most sites ended up with a policy nobody chose. Web-wide, AppleBot is disallowed on 108,452 sites, GPTBot on 107,182, Common Crawl on 102,665 and ClaudeBot on 101,917 — eight independent operators blocked within 12% of each other, which is one copy-pasted snippet inherited by everyone downstream, not eight decisions. AI_OPTOUT_SET fires on 4.5% of scans here.

Two mistakes to avoid while editing. ROBOTS_RULES_SHADOWED fires on 12.9% of scans — a later group silently overriding an earlier one, so a rule you wrote never applies. And check the file still serves text/plain afterwards: a robots.txt returning HTML is parsed as no directives, which means everything allowed, the exact opposite of a block.

Layer 2: the network #

This is where a block is actually enforced, against clients that never ask permission.

Block by verified identity, never by user-agent string. A rule that trusts a UA is a rule anyone satisfies by typing — and in this dataset 37% of checkable crawler claims came from outside the operator's published range, with some of that forged traffic requesting .env files and SSH keys while wearing a crawler's name. Allow-listing by user-agent hands those requests a pass.

If you are behind Cloudflare, the categories are exposed per zone and the defaults change on 15 September 2026: Training and Agent become blocked by default on monetised pages while Search stays allowed. Read the coupling before you set anything — blocking Training also blocks multi-purpose crawlers such as Googlebot, Applebot and Bingbot, which collect for both purposes. A block aimed at AI training can take your search traffic with it. Owners who do not want the new behaviour can opt out through Security settings before that date.

Layer 3: stating intent rather than access #

The newer and more interesting option is declaring what may be done with content after it is fetched, rather than whether it may be fetched. Cloudflare's content-use controls express three levels — Immediate, meaning no storage or reuse; Reference, meaning indexing, excerpts and links back; and Full, meaning summaries or reproduction — with a use parameter extending the Content Signals vocabulary in robots.txt, and compliance by verified bots reported rather than enforced.

That is closer to what most site owners actually want. Very few want never mention us. Most want reference us and link back, do not train on us, which allow-and-block has no vocabulary for.

Check the block did what you meant #

After any change, verify from outside your network, on the bare URL, as each identity:

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "GPTBot" https://yoursite.com/

Run it for every crawler you intended to allow as well as every one you intended to block. The common outcome is a policy that blocks more than intended, because a robots.txt block, a plugin default and an edge rule are three independent systems that nobody reads together. ANSWER_ENGINE_REFUSED fires on 5.8% of scans, and in most of those cases the refusal was not in the file the owner had been editing.

What a block does not do #

Three limits worth being clear-eyed about before treating a block as protection.

It does not remove you from models already trained. Blocking a training crawler affects future collection, not weights that already exist.

It does not stop content reaching a model by another route. Your pages get quoted, syndicated and aggregated elsewhere, and those copies carry no policy of yours.

It does not stop anything that ignores you. robots.txt has no enforcement power, and an operator that publishes no verifiable IP range cannot be told apart from an impostor at the network layer either — which is why 37% of checkable crawler claims here came from outside a published range and five identities in the corpus have no feed to check against at all.

A block is a statement of intent that cooperative parties honour and a network rule that enforces against the ones you can identify. It is worth doing for those reasons and not worth mistaking for a wall.

The one thing worth keeping open #

If you decide to block, consider allowing user-triggered fetches even while blocking training. Those requests exist because a real person is asking about you right now, and they are the cheapest attention on the internet: no crawl budget, no bulk harvesting, one page, in response to demand you already earned.

The robots.txt rule that silently voids itself #

This is the single most common way a block written in good faith does nothing, and it fires on 12.9% of scans in this corpus.

robots.txt groups are not additive. A crawler obeys one group — the most specific one that names it — and ignores every other group in the file, including the wildcard. So the moment you add a named group for an agent, that agent stops seeing the rules under User-agent: * entirely.

User-agent: *
Disallow: /wp-admin/
Disallow: /checkout/

User-agent: GPTBot
Disallow: /pricing/

The intent there is obvious: keep GPTBot out of pricing, on top of the existing rules. The effect is the opposite. GPTBot now matches the second group, which says nothing about /wp-admin/ or /checkout/, so those become fair game for the one agent you singled out. Every rule you thought applied to everyone now applies to everyone except the agent you wrote a rule about.

The fix is unglamorous: every named group has to repeat the wildcard rules in full. If that sounds like duplication waiting to drift, it is, which is why a generated robots.txt beats a hand-edited one the moment you have more than one named group.

A name block only stops the crawlers that were already behaving #

Worth sitting with, because it inverts the usual intuition. Blocking by user-agent is a request, and a request is only honoured by an operator that chose to honour it. The operators most likely to comply are the ones that publish their address ranges, document their tokens and run a support page — which is to say, the ones whose traffic was never the problem.

Meanwhile anything impersonating those names ignores the file completely, and impersonation is not rare: across the checkable claims in this dataset, a substantial share came from addresses outside the range the named operator publishes. A robots.txt block sorts your traffic into compliant and non-compliant, and then removes the compliant half.

That is not an argument against writing the file. It is an argument for being clear about which problem each layer solves: robots.txt expresses policy, and only a network rule enforces it.

What a working block looks like from the outside #

A block changes what a fetch returns, so it is visible — and so is a block that is broader or narrower than you intended:

That last row deserves the space. A managed bot filter that answers a blocked client with an interstitial and a 200 status is telling the operator that the fetch succeeded, and handing it a page of challenge markup as if it were your content. You believe you blocked the crawler. The crawler believes it read your site. Neither is true, and an operator that caches that response has now stored a challenge page as your homepage.

If you want a block, return the status that means one. A 403 is unambiguous and costs a well-behaved operator one request.

Verify by outcome, on a forced miss #

Configuration is not evidence. The only proof a block works is a fetch that gets refused, made from outside your network, under the identity you blocked, on a request the edge cannot answer from cache:

curl -sI -A "GPTBot" "https://yoursite.com/?cachebust=1"

Read the status line, and read it on the final response rather than on a redirect. Three results are worth telling apart. A 403 or 401 is a block. A 200 with a short body is a challenge page, which is the row above. A 200 with your actual content means the rule did not apply — usually because it was written at a layer the request never reached, or because an edge cache answered before the rule ran.

Run the same request without the user-agent afterwards. If both come back identical, whatever you configured is not keying on identity at all.

The two blocks have different clocks #

The decision people regret is rarely the one they thought they were making, and the reason is timing.

Blocking retrieval takes effect immediately and reverses immediately. Answer-engine crawlers fetch at answer time, so refusing them removes you from answers about you starting today, and allowing them again restores it. It is a tap.

Blocking training works on a completely different clock. It affects what goes into the next model generation, which means the consequence is invisible for months and cannot be undone by changing your mind afterwards — a model trained without you stays trained without you, whatever your robots.txt says later.

Most people who set out to block AI crawlers want the second and accidentally buy the first, because the first is the one that shows up in logs. If you are unsure, block training, keep retrieval and user-triggered fetches open, and revisit in a quarter. That combination is reversible in the direction you are most likely to want to reverse.

Blocking an agent is a different decision from blocking a crawler

Each AI operator now runs separate clients for training, for its search index, and for the live fetch a person triggers. Blocking one is not blocking the others, and the cost lands in a different place each time.

Two of the three user-triggered fetchers document that robots.txt may not bind them, so a block that must hold for them is an edge rule on verified identity, not a Disallow line. Signed requests under Web Bot Auth are how agents increasingly identify themselves, and they are the identity to block or allow on.

Findings behind this #

Every figure above comes from a measurement written up in full, with the date, the method and the raw numbers.

• GPTBot, ClaudeBot, PerplexityBot: what each AI crawler is actually for
• How much AI crawler traffic is forged? Checked against IP ranges
• Which AI crawlers publish IP ranges, and which cannot be verified

Questions this post answers

How do I block AI crawlers from my website?

Two layers. Write per-agent rules in robots.txt, which named crawlers honour but which enforces nothing. Then enforce at the network layer, blocking by verified identity rather than user-agent string, since a user-agent is a claim anyone can send.

Does blocking AI crawlers hurt my search traffic?

It can. From 15 September 2026, Cloudflare's defaults block Training and Agent crawlers on monetised pages, and blocking Training also blocks multi-purpose crawlers such as Googlebot, Applebot and Bingbot that collect for both search and training.

Should I block all AI crawlers?

Decide the three categories separately. Blocking training crawlers has no immediate cost. Blocking search indexing removes your eligibility to be cited. Blocking user-triggered fetches means that when someone asks an assistant about you, the answer is composed without you.

Does blocking AI crawlers in robots.txt actually work?

It works on operators that choose to honour it, which is most of the large named ones, and on nothing else. Anything impersonating those names ignores the file entirely. If you need the request refused rather than discouraged, that has to happen at the network layer, where the server returns a status instead of stating a preference.

Will blocking AI crawlers hurt my SEO?

It depends which ones. Blocking training crawlers has no effect on search rankings. Blocking search-index and answer-engine crawlers removes your eligibility to be retrieved and cited, which is the visibility most people are trying to protect. The three are separate decisions that a single wildcard rule collapses into one.

How do I check my block is working?

Fetch your own site from outside your network under the blocked user-agent, with a cache-busting parameter, and read the status on the final response. A 403 is a block. A 200 carrying a short challenge page is not a block, and it tells the operator the fetch succeeded. A 200 with your real content means the rule never applied.

Should I block ChatGPT-User, Claude-User or Perplexity-User?

Only if you do not want an assistant to read your page at the moment a person asks about you. They are user-triggered fetchers, not training crawlers: blocking GPTBot or ClaudeBot keeps content out of training, while blocking these removes your own page from live answers about you.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works