Guides · 2026-08-28 · By VSNARY | Emmanuel Orta
Do AI crawlers respect robots.txt?
The named ones largely do. The problem in practice is almost never disobedience — it is that the file says one thing and the network in front of it does another, and nothing reports the conflict.
Asked plainly: the major named crawlers from OpenAI, Anthropic, Google, Microsoft and Apple publish their identities, publish IP ranges you can verify them against, and honour robots.txt directives. Some unnamed and some less scrupulous operators do not, and robots.txt has never had any enforcement power over them — it is a request, not a control.
But obedience is not where sites actually lose. In this dataset the far commoner failure is a robots.txt that is obeyed perfectly and says something its owner did not intend, or that no crawler can read in the first place.
Failure 1: the file cannot be parsed
A robots.txt that answers 200 OK with a bot-challenge page is read as allow-everything, because a parser that receives HTML finds no directives and no directives means no restrictions. On one site, the underlying file was excellent — twenty-two agents allowed individually, a Content-Signal line, sitemap and llms.txt both declared. Every client that asked got 12KB of HTML saying one moment, please. Nobody read a word of it.
Three readings of the same file, and only the third is about the file:
| Uptime monitor | 200 — healthy |
| Parsing validator | no directives, so everything allowed |
| What a crawler received | 12KB of HTML |
ROBOTS_NOT_200 fires on 3.9% of scans, CHALLENGE_SERVED_200 on 1.9%, ROBOTS_IS_HTML on 0.4%. Check the content type, not the status: it must be text/plain.
Failure 2: the file is obeyed and the edge is not
This is the one that costs the most and appears in no validator, because both halves are individually valid. Your robots.txt invites every answer engine. Your edge then refuses them. The crawler complied with your stated policy and was turned away by your infrastructure.
On one measured site, GPTBot and ClaudeBot both received 502 while Googlebot and PerplexityBot were served normally, with robots.txt allowing all four. That is a policy conflict — invited but refused — and ANSWER_ENGINE_REFUSED fires on 3.4% of scans. Nothing in robots.txt can cause it and nothing in robots.txt can fix it.
Failure 3: the file says something nobody chose
AI_OPTOUT_SET fires on 15.0% of scans. Some of that is deliberate. Much of it is not. Web-wide, AppleBot is disallowed on 108,452 sites, GPTBot on 107,182, Common Crawl on 102,665, ClaudeBot on 101,917 — eight independent operators blocked within 12% of each other. That is not eight decisions. It is one copy-pasted snippet, one plugin default, one hosting toggle, inherited by everyone downstream of it.
Two related defects sit alongside: ROBOTS_RULES_SHADOWED (1.3%), where a later group silently overrides an earlier one so a rule you wrote never applies, and ROBOTS_DISALLOW_ALL (2.1%), which is occasionally a staging config that shipped.
Failure 4: the one job robots.txt has that is not about blocking
Lost in the argument about obedience is that robots.txt is also the discovery entry point. It is where a crawler expects to be told where your sitemap is, and that declaration is broken far more often than any permission rule.
| Defect | What it means | Fires on |
|---|---|---|
NO_SITEMAP_FOUND | No sitemap discoverable at all | 5.6% |
ROBOTS_NO_SITEMAP | A sitemap exists; robots.txt does not name it | 3.2% |
DECLARED_SITEMAP_BROKEN | robots.txt points at a sitemap that does not resolve | 2.3% |
SITEMAP_BLOCKED | The sitemap is declared and then disallowed | 1.8% |
SITEMAP_IS_HTML | An HTML page sits where the XML should be | 1.5% |
Taken together that is a discovery failure on a meaningful share of sites, and none of it is a crawler misbehaving. The fourth row deserves its own mention: declaring a sitemap and then disallowing the path to it is a self-cancelling instruction that both halves of your configuration consider correct. It is precisely the kind of contradiction that only appears if something reads the file the way a crawler does, in order, and follows what it says.
Why a user-agent string proves nothing
The mirror image of do they obey is is it really them. Across 2,510 checkable claims here, 37% came from an address outside the range the operator publishes.
| Claimed to be | Claims | Forged |
|---|---|---|
| Googlebot | 567 | 9.5% |
| ClaudeBot | 500 | 16.4% |
| GPTBot | 300 | 47% |
| PerplexityBot | 189 | 51.9% |
| ChatGPT-User | 141 | 90.1% |
Some of that forged traffic is scanning for .env and credential files while wearing a crawler's name. Two consequences follow. Never write an allow rule that trusts a user-agent string, because that is a rule anyone satisfies by typing. And when you read your logs to judge whether crawlers obeyed your policy, verify identity against published ranges first — otherwise you are grading a stranger's behaviour as OpenAI's.
Read your own file the way a crawler does
Four requests, in order, and each one answers a different question:
curl -sI https://yoursite.com/robots.txt — status and content type. It must be text/plain.curl -s https://yoursite.com/robots.txt — the body, on the bare URL, with no cache-buster appended.curl -s https://yoursite.com/robots.txt | grep -i sitemap — the declaration, then fetch what it points at.curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://yoursite.com/ — whether the edge agrees with the file.
The second of those is the one people skip. Adding ?v=1 to the URL asks the origin a fresh question and bypasses whatever the edge is holding, so the file comes back perfect and the actual defect stays invisible. A crawler never appends anything.
The order that actually matters
Whether crawlers obey your robots.txt is the fourth question, not the first. Ahead of it: can the file be parsed, does your edge agree with it, and did you choose what it says. Every one of those is checkable from outside your network in under a minute, and each is more likely to be broken than a major crawler is to ignore you.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.
Questions this post answers
Do GPTBot and ClaudeBot obey robots.txt?
The major named crawlers publish their identities and IP ranges and honour robots.txt directives. Unnamed operators may not, and robots.txt has no enforcement power over them. In practice the commoner failure is not disobedience but a file that cannot be parsed, or an edge rule that refuses a crawler robots.txt invited.
Why would a crawler be blocked if my robots.txt allows it?
Because robots.txt is a request to the crawler and your edge is a control over the connection. They are separate layers and can disagree. That conflict fires on 3.4% of scans here, and no validator reports it, because both halves are individually valid.
Can I trust the user-agent in my server logs?
Not without checking the source address against the operator's published range. Of 2,510 checkable claims in this dataset, 37% came from outside it, including 47% of traffic calling itself GPTBot.