Findings · 2026-08-13 · By VSNARY | Emmanuel Orta · 0 views
robots.txt returned 200 and no crawler could read it: challenge pages
A robots.txt that answers HTTP 200 with a bot-challenge page is read as allow-everything by most crawlers. Why every validator missed it, and the two-request test that catches it.
Three readings of one file. Only the third is about the file.
A site’s /robots.txt answered HTTP 200 to every client we sent — a browser, Googlebot, GPTBot, curl, our own crawler. All of them got the same 12 KB of HTML: a verification interstitial that reads one moment, please, your request is being verified.
Underneath it, the real file was excellent: twenty-two agents allowed individually, a Content-Signal line, sitemap and llms.txt declared. Nobody could read a word of it.
Why every monitor said it was fine
Uptime checks look at the status code. The status code was 200. Validators fetch the file and parse what they get; what they got was HTML, so they reported no robots directives — which reads as everything is allowed, the exact opposite of what the file said.
Our own scan reported it as no robots.txt served, every agent allowed by default. That is the trap: an absence and a refusal look identical unless you check the content type.
What was actually happening
The challenge came from the origin, not the CDN — a bot defence firing at one address making rapid requests. Ordinary, and it would have been harmless on its own.
The damage came from a caching rule that stored the response. A per-address defence became a site-wide outage of the machine layer, because a cache cannot tell that a 200 is a challenge page. Every crawler for the next five minutes was served the cached interstitial.
The origin had done the right thing. Its response carried Cache-Control: private, no-store — the standard way of saying this answer is for one requester only. A cache rule at the edge overrode the origin’s TTL to make machine files fast, and an override ignores the origin’s opinion by design. So a page written for one suspicious address was pinned at the edge and handed to everyone. We now name that state as its own finding, CHALLENGE_PINNED_AT_EDGE: the origin marked a response uncacheable, a TTL override stored it anyway, and the verification page is being served from the cache until it expires or is purged.
What each reader made of the same 200
| Reader | What it concluded |
|---|---|
| HTTP uptime monitor | healthy |
| Parsed validator | no directives, so everything is allowed |
| A crawler | 12 KB of HTML — a verification interstitial |
| Clients that received exactly this | a browser, Googlebot, GPTBot, curl, and our own crawler |
A per-address defence became a site-wide outage of the machine layer, because a cache cannot tell that a 200 is a challenge page.
Our own signature list did not contain this page
The scanner keeps a list of body signatures that mark interstitial and verification pages: checking your browser before accessing, verifying you are human, just a moment, the class names a well-known CDN puts on its challenge, and a dozen more. This page said one moment, please. Different words. The list had the CDN’s phrasing and not the origin-side shield’s, so on the day this happened the challenge check could not fire on the very page that caused it. It was added five days later, along with a second shield found on a different site that answers every path with a 190-byte meta-refresh at HTTP 202 — nothing 404s, every machine file fails, and a scan of the wall gets scored as a scan of the site.
A signature list is only ever as good as the last wall it met. That is why it is not the check we rely on.
The two-request test that does not need a signature
Fetch the homepage. Then fetch a path that cannot exist — a random string nobody would ever have published. Compare the two answers.
A real site gives two different responses: a page, and a 404 that looks nothing like it. A wall gives the same response to both, because it never let the request reach the site at all. The scanner records the pair as refusalProbe: both statuses, both byte counts, whether the sizes are close, and whether either body carries a challenge signature. Same status, close size, and a signature on both is one interstitial served twice. Identical byte counts on two unrelated URLs is the same thing with no signature needed — one site answered every path, including the homepage, with a 75 KB block page, and no phrase in our list matched it.
This test has been refined once since. The first version also required both bodies to be HTML. A large retailer then answered every path, including its homepage, with a ten-byte Not found and an empty content-type, and the HTML requirement let that wall through to be graded as a site with no sitemap. Two identical tiny answers to the homepage and to a path that cannot exist are a wall whatever the content-type says; a page type is not what makes it one. The requirement was removed.
Why a wall is recorded and not graded
When the probe says wall, the scan stops grading. There is no site behind the answer to grade. The record states what was received and who refused it, and the refusal is held for two days before it becomes a finding at all, on the rule that one observation is an anecdote. A tool that scores a challenge page D with no sitemap found and a perfect speed score has measured its own rejection, and every number on that report describes the firewall. We shipped exactly that report once, and it is the class of error the self-audit exists to catch.
The three findings this can raise, and what each one means
ROBOTS_CTYPE is the mild one: the file parsed, but it was served as something other than text/plain, and some parsers are strict. ROBOTS_IS_HTML is the outage: the body is a page, no directive and no sitemap reference can be read, and every crawler proceeds as if the file did not exist. ROBOTS_IS_CATCHALL is the diagnosis underneath it: robots.txt is being answered with the same page as a path that does not exist, so nothing at that address is really being served — the shield is. Each names a different repair. The first is a header. The second and third are a firewall rule or a cache rule, and no edit to the file will touch them.
How to check your own in ten seconds
Fetch your robots.txt and look at the content type, not the status:
curl -sI https://yoursite.com/robots.txt | grep -i content-typeIt should say text/plain. If it says text/html, something is answering in place of your file, and the 200 beside it means nothing.
Then run the second request: fetch /this-path-does-not-exist- followed by a few random characters, and compare its size to the robots.txt answer. If they are within a few hundred bytes of each other, you are reading a wall, not a file. The same pair of tests applies to your sitemap and any llms.txt. A file that returns 200 and cannot be parsed is worse than one that 404s, because a 404 at least tells the truth — the terms are in the glossary.
What this scanner does about it now
Every scan fetches a randomised path that cannot exist alongside the homepage and the machine files. When the nonexistent path answers 200 with the same HTML a machine file answers, that file is recorded as absent, not present, and the report says the site answered with its catch-all page. When every path answers identically in the way challenge walls do, the scan stops and reports that it was refused rather than grading the wall. The same rule reached llms.txt only after the scanner's own self-audit noticed two of its layers disagreeing about one file.
The cache rule that turns a nuisance into an outage
If your CDN caches machine files, and it should, the rule must not store a response whose content type is wrong for the path. A robots.txt that arrives as text/html is never a robots.txt, and caching it hands the challenge to every crawler for the length of the TTL. The purge that returned 200 and evicted nothing and a cache rule does not evict what is already cached are the two follow-ups; the glossary defines challenge page at 200 and edge cache pinning.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.
Questions this post answers
Can robots.txt return 200 and still be broken?
Yes. A cached bot-protection challenge page can sit at that path and answer 200 with HTML. Uptime monitors call it healthy because they check the status code, not the content type or the body.
How do I check my own robots.txt in ten seconds?
Fetch the bare URL from outside your own session and read two things: the content type, which must be text/plain, and the first line of the body. HTML there means no crawler can read your rules.
What does a crawler do when robots.txt is unreadable?
Parsers that cannot read the file may treat the site as fully disallowed or fully allowed depending on the parser. Either way your actual rules are not the ones being applied.
Related findings
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.