Guides · 2026-08-28 · By VSNARY | Emmanuel Orta · 0 views

Do AI crawlers and agents respect robots.txt?

The named ones largely do. The problem in practice is almost never disobedience — it is that the file says one thing and the network in front of it does another, and nothing reports the conflict.

Asked plainly: the major named crawlers from OpenAI, Anthropic, Google, Microsoft and Apple publish their identities, publish IP ranges you can verify them against, and honour robots.txt directives. Some unnamed and some less scrupulous operators do not, and robots.txt has never had any enforcement power over them — it is a request, not a control.

But obedience is not where sites actually lose. In this dataset the far commoner failure is a robots.txt that is obeyed perfectly and says something its owner did not intend, or that no crawler can read in the first place.

Failure 1: the file cannot be parsed #

A robots.txt that answers 200 OK with a bot-challenge page is read as allow-everything, because a parser that receives HTML finds no directives and no directives means no restrictions. On one site, the underlying file was excellent — twenty-two agents allowed individually, a Content-Signal line, sitemap and llms.txt both declared. Every client that asked got 12KB of HTML saying one moment, please. Nobody read a word of it.

Three readings of the same file, and only the third is about the file:

ROBOTS_NOT_200 fires on 5.9% of scans, CHALLENGE_SERVED_200 on 0.6%, ROBOTS_IS_HTML on 0.2%. Check the content type, not the status: it must be text/plain.

Failure 2: the file is obeyed and the edge is not #

This is the one that costs the most and appears in no validator, because both halves are individually valid. Your robots.txt invites every answer engine. Your edge then refuses them. The crawler complied with your stated policy and was turned away by your infrastructure.

On one measured site, GPTBot and ClaudeBot both received 502 while Googlebot and PerplexityBot were served normally, with robots.txt allowing all four. That is a policy conflict — invited but refused — and ANSWER_ENGINE_REFUSED fires on 5.8% of scans. Nothing in robots.txt can cause it and nothing in robots.txt can fix it.

Failure 3: the file says something nobody chose #

AI_OPTOUT_SET fires on 4.5% of scans. Some of that is deliberate. Much of it is not. Web-wide, AppleBot is disallowed on 108,452 sites, GPTBot on 107,182, Common Crawl on 102,665, ClaudeBot on 101,917 — eight independent operators blocked within 12% of each other. That is not eight decisions. It is one copy-pasted snippet, one plugin default, one hosting toggle, inherited by everyone downstream of it.

Two related defects sit alongside: ROBOTS_RULES_SHADOWED (12.8%), where a later group silently overrides an earlier one so a rule you wrote never applies, and ROBOTS_DISALLOW_ALL (1.4%), which is occasionally a staging config that shipped.

Failure 4: the one job robots.txt has that is not about blocking #

Lost in the argument about obedience is that robots.txt is also the discovery entry point. It is where a crawler expects to be told where your sitemap is, and that declaration is broken far more often than any permission rule.

Taken together that is a discovery failure on a meaningful share of sites, and none of it is a crawler misbehaving. The fourth row deserves its own mention: declaring a sitemap and then disallowing the path to it is a self-cancelling instruction that both halves of your configuration consider correct. It is precisely the kind of contradiction that only appears if something reads the file the way a crawler does, in order, and follows what it says.

Why a user-agent string proves nothing #

The mirror image of do they obey is is it really them. Across 2,510 checkable claims here, 37% came from an address outside the range the operator publishes.

Some of that forged traffic is scanning for .env and credential files while wearing a crawler's name. Two consequences follow. Never write an allow rule that trusts a user-agent string, because that is a rule anyone satisfies by typing. And when you read your logs to judge whether crawlers obeyed your policy, verify identity against published ranges first — otherwise you are grading a stranger's behaviour as OpenAI's.

Read your own file the way a crawler does #

Four requests, in order, and each one answers a different question:

curl -sI https://yoursite.com/robots.txt — status and content type. It must be text/plain.
curl -s https://yoursite.com/robots.txt — the body, on the bare URL, with no cache-buster appended.
curl -s https://yoursite.com/robots.txt | grep -i sitemap — the declaration, then fetch what it points at.
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://yoursite.com/ — whether the edge agrees with the file.

The second of those is the one people skip. Adding ?v=1 to the URL asks the origin a fresh question and bypasses whatever the edge is holding, so the file comes back perfect and the actual defect stays invisible. A crawler never appends anything.

The order that actually matters #

Whether crawlers obey your robots.txt is the fourth question, not the first. Ahead of it: can the file be parsed, does your edge agree with it, and did you choose what it says. Every one of those is checkable from outside your network in under a minute, and each is more likely to be broken than a major crawler is to ignore you.

Agents acting for a person play by different rules

robots.txt was written for crawlers that decide on their own what to fetch. A user-triggered fetcher opens one page because a person asked it to, and the three largest operators document it three different ways.

So a Disallow line is a dependable control over training and index crawlers and a partial one over agents. If a page must not be reached by an agent acting for someone, robots.txt is a request; the enforcement is at your edge, by verified identity, and the cost is that the person asking about you gets no answer from your own page. Operator documentation read on 8 October 2026.

Findings behind this #

Every figure above comes from a measurement written up in full, with the date, the method and the raw numbers.

• robots.txt returned 200 and no crawler could read it: challenge pages
• How much AI crawler traffic is forged? Checked against IP ranges
• Is your host blocking ClaudeBot? Group the 403s by IP first

Questions this post answers

Do GPTBot and ClaudeBot obey robots.txt?

The major named crawlers publish their identities and IP ranges and honour robots.txt directives. Unnamed operators may not, and robots.txt has no enforcement power over them. In practice the commoner failure is not disobedience but a file that cannot be parsed, or an edge rule that refuses a crawler robots.txt invited.

Why would a crawler be blocked if my robots.txt allows it?

Because robots.txt is a request to the crawler and your edge is a control over the connection. They are separate layers and can disagree. That conflict is measured on every scan here (the live share is on the dataset page), and no validator reports it, because both halves are individually valid.

Can I trust the user-agent in my server logs?

Not without checking the source address against the operator's published range. Of 2,510 checkable claims in this dataset, 37% came from outside it, including 47% of traffic calling itself GPTBot.

Do AI agents respect robots.txt?

It depends on the operator. OpenAI says robots.txt may not apply to ChatGPT-User because a user started the fetch, Perplexity says Perplexity-User generally ignores it, and Anthropic says Claude-User honours it. Training and index crawlers such as GPTBot and ClaudeBot honour it.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works