Findings · 2026-08-13 · By VSNARY | Emmanuel Orta · 5 views
robots.txt returns 200 but no crawler can read it
Every monitor said the file was fine. Every client, from Googlebot to curl, received a 12 KB verification page instead, and an llms.txt elsewhere turned out to be a login wall. How it happens, and the two-request test that catches it.
A 200 status says the server answered, not that it served the file. One site's robots.txt returned 200 to a browser, Googlebot, GPTBot and curl, and every one received a 12 KB verification page instead of the rules. A bot defence had challenged one address and a cache rule stored that page for everyone. Most parsers read an unreadable robots.txt as everything allowed. Check the content type and the body, not the status: robots.txt and llms.txt should be text/plain or text/markdown, and should not match the page a nonexistent path returns.
Three readings of one file. Only the third is about the file.
A site’s /robots.txt answered HTTP 200 to every client we sent: a browser, Googlebot, GPTBot, curl and our own crawler. All of them got the same 12 KB of HTML, a verification interstitial that reads one moment, please, your request is being verified.
Underneath it, the real file was excellent: twenty-two agents allowed individually, a Content-Signal line, a sitemap and an llms.txt declared. Nobody could read a word of it, and every monitor on the site reported it healthy.
Why did every monitor say the file was fine? #
Uptime checks look at the status code, and the status code was 200. Validators fetch the file and parse what they get; what they got was HTML, so they found no directives, which reads as everything is allowed, the exact opposite of what the file said. Our own scan reported it as no robots.txt served. That is the trap: an absence and a refusal look identical unless you check what the body is.
| Reader | What it concluded from the same 200 |
|---|---|
| HTTP uptime monitor | healthy |
| Validator that parses the file | no directives, so everything is allowed |
| A crawler | 12 KB of HTML: a verification interstitial |
| Clients that received exactly this | a browser, Googlebot, GPTBot, curl and our own crawler |
How did a defence against one address reach every crawler? #
The challenge came from the origin, aimed at one address making rapid requests. On its own that is reasonable and would have inconvenienced only that client. The origin even marked its answer correctly, with Cache-Control: private, no-store: this response is for one requester only.
The damage came from an edge cache rule written to make machine files fast. It overrode the origin’s caching headers, which is what an override does by design, so the page written for one suspicious address was stored against the URL and handed to every crawler for the next five minutes. A cache key is built from the URL and a few headers; it has no way to know a 200 is a challenge.
Follow the consequence through and it is worse than a missing file. Most parsers treat an unreadable robots.txt as permission, so the cached challenge removed the site’s crawl restrictions for every well-behaved operator, while the client that triggered it was never reading robots.txt in good faith anyway. The compliant crawlers lost the policy; the non-compliant one lost nothing. Operators that keep their own copy of your robots.txt then hold a challenge page as your policy for as long as their copy lives. Challenge pages answered with 200 turn up on 0.3% of scanned sites here.
Two changes remove the amplifier. Send Cache-Control: no-store on every interstitial and make sure no edge rule overrides it, and answer a refusal with a status that means refusal, so it is never eligible for a rule that caches 200s. The machine-file caching guide shows how to see which copy a crawler is getting.
The same trap at llms.txt: a login wall answering 200 #
A large community site answers /llms.txt with HTTP 200 and 8,401 bytes of HTML: a script that submits a form on load, a favicon link, a page title, and nothing a language model could use. It is the same page the site returns for a path that cannot exist. The site has no llms.txt at all; it has a login wall wearing a status code.
We got this one half wrong. On the same scan, one part of CrawlCheck read the content type and called the file absent, while another saw the 200 and called it present. Our self-audit flags any two readings that describe one path differently, which is how the disagreement surfaced before it reached a report. The reading that trusted the status was corrected: an llms.txt answered with HTML is now reported as missing, whatever the status says.
Why a list of known challenge pages is not enough #
On the day the robots.txt case happened, its interstitial wording was not on our list of known challenge pages, so a wording match could not flag the page that caused it. Matching known wording is a lagging instrument by construction: it recognises only walls somebody has already met, and every new vendor, version or translation produces a page it has never seen. Walls also vary far more than a wording list expects:
| Wall seen | Status | Body | What gave it away |
|---|---|---|---|
| Verification interstitial at robots.txt | 200 | 12 KB of HTML | HTML where a text file belongs |
| Login wall at llms.txt | 200 | 8,401 bytes of HTML | identical to the page for a nonexistent path |
| Shield answering every path | 202 | 190-byte meta-refresh | nothing 404s, every machine file fails |
| Block page on every path, homepage included | 200 | 75 KB, matched no known wording | identical byte counts on every path |
| A large retailer, every path | not stated | ten bytes, empty content type | two identical tiny answers |
Walls of this kind, where every path gets the same answer, turn up on 16% of scanned sites here.
The two-request test that needs no signature #
Fetch the page you care about. Then fetch a path that cannot exist, a random string nobody would ever have published, and compare the two answers. A real site gives two different responses: a page, and a 404 that looks nothing like it. A wall gives the same response to both, because it is deciding about the client and never let the request reach the site. The same body at both paths is one interstitial served twice, whatever the status or content type says.
This matters for any tool that grades your site. A checker that scores a wall as a D with no sitemap and a perfect speed score has measured its own rejection, and every number on that report describes the firewall. We shipped exactly that report once; the site that scored 15 because it refused us is the write-up. If a report on your site looks implausibly empty, run the two-request test before acting on it.
Three failures, three different repairs #
| What you see | What crawlers get | Where the repair is |
|---|---|---|
File parses but is served as something other than text/plain | Strict parsers may reject it | A response header |
| Body is an HTML page, no directives readable | No file, so everything allowed | A firewall or cache rule, not the file |
| robots.txt answers the same as a nonexistent path | A shield, not the site | A firewall or cache rule, not the file |
No edit to the file touches the second or third. Only the first is a fix you make in the file’s own configuration.
Which monitor checks would have caught it? #
Every monitor on that site checked the same thing, the status line. These assertions are cheap and none of them looks at the status:
| Assertion | What it catches |
|---|---|
Content type is text/plain | HTML served in place of the file, whatever the status |
| Body length within an expected band | A 12 KB interstitial where a 2 KB file belongs, and an empty file |
Body contains the literal User-agent: | Anything that is not a robots.txt, including a wall nobody has seen before |
The third is the strongest and the least used. It does not need to recognise the wall, only your own file, which you control. Any monitor that supports a body match supports it; AI crawler monitoring vs uptime monitors covers which tools do.
How to check your own in ten seconds #
Look at the content type, not the status, for each machine file:
curl -sI https://yoursite.com/robots.txt | grep -i content-type
curl -sI https://yoursite.com/llms.txt | grep -i content-typerobots.txt should say text/plain; llms.txt text/plain or text/markdown. Then compare sizes against a path that cannot exist:
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" https://yoursite.com/robots.txt
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" https://yoursite.com/this-path-does-not-exist-q7xIf the two answers are within a few hundred bytes of each other, you are reading a wall, not a file. The same test applies to your sitemap. A machine file that returns 200 and cannot be parsed is worse than one that 404s, because a 404 at least tells the truth. The llms.txt guide covers the rest of that file’s shape, and the terms are in the glossary.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
Can robots.txt return 200 and still be broken?
Yes. A cached bot-protection challenge page can sit at that path and answer 200 with HTML. Uptime monitors call it healthy because they check the status code, not the content type or the body.
What does a crawler do when robots.txt is unreadable?
Most parsers treat a file they cannot read as no restrictions at all, so a robots.txt that arrives as challenge markup is read as everything allowed. Either way, your actual rules are not the ones being applied.
Can a bot protection page break robots.txt for everyone?
Yes, if the response gets cached. A challenge generated for one address is stored against the URL, and every later request is answered from that stored copy, so a per-client control becomes a site-wide condition.
Can an llms.txt be served as HTML?
No. The convention is plain text or markdown. A 200 that carries HTML at /llms.txt is a catch-all page, a challenge or a login form, and none of those is the file. Check the content type and the first line, not the status.
How do I detect a verification wall without a signature list?
Request the real path and an impossible one, then compare the two responses. A working site answers them differently. A wall answers both with the same body, because it is deciding about the client rather than reading the path. That test works on walls nobody has catalogued yet.
What should a robots.txt monitor check instead of the status?
Three things: that the content type is text/plain, that the body length is in the range you expect, and that the body contains the literal text User-agent:. The last one recognises your own file rather than trying to recognise every possible wall.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.