Findings · 2026-08-13 · By VSNARY | Emmanuel Orta · 2 views

robots.txt returned 200 and no crawler could read it: challenge pages

A robots.txt that answers HTTP 200 with a bot-challenge page is read as allow-everything by most crawlers. Why every validator missed it, and the two-request test that catches it.

A site’s /robots.txt answered HTTP 200 to every client we sent — a browser, Googlebot, GPTBot, curl, our own crawler. All of them got the same 12 KB of HTML: a verification interstitial that reads one moment, please, your request is being verified.

Underneath it, the real file was excellent: twenty-two agents allowed individually, a Content-Signal line, sitemap and llms.txt declared. Nobody could read a word of it.

Why every monitor said it was fine #

Uptime checks look at the status code. The status code was 200. Validators fetch the file and parse what they get; what they got was HTML, so they reported no robots directives — which reads as everything is allowed, the exact opposite of what the file said.

Our own scan reported it as no robots.txt served, every agent allowed by default. That is the trap: an absence and a refusal look identical unless you check the content type.

What was actually happening #

The challenge came from the origin, not the CDN — a bot defence firing at one address making rapid requests. Ordinary, and it would have been harmless on its own.

The damage came from a caching rule that stored the response. A per-address defence became a site-wide outage of the machine layer, because a cache cannot tell that a 200 is a challenge page. Every crawler for the next five minutes was served the cached interstitial.

The origin had done the right thing. Its response carried Cache-Control: private, no-store — the standard way of saying this answer is for one requester only. A cache rule at the edge overrode the origin’s TTL to make machine files fast, and an override ignores the origin’s opinion by design. So a page written for one suspicious address was pinned at the edge and handed to everyone. We now name that state as its own finding, CHALLENGE_PINNED_AT_EDGE: the origin marked a response uncacheable, a TTL override stored it anyway, and the verification page is being served from the cache until it expires or is purged.

What each reader made of the same 200 #

A per-address defence became a site-wide outage of the machine layer, because a cache cannot tell that a 200 is a challenge page.

Our own signature list did not contain this page #

The scanner keeps a list of body signatures that mark interstitial and verification pages: checking your browser before accessing, verifying you are human, just a moment, the class names a well-known CDN puts on its challenge, and a dozen more. This page said one moment, please. Different words. The list had the CDN’s phrasing and not the origin-side shield’s, so on the day this happened the challenge check could not fire on the very page that caused it. It was added five days later, along with a second shield found on a different site that answers every path with a 190-byte meta-refresh at HTTP 202 — nothing 404s, every machine file fails, and a scan of the wall gets scored as a scan of the site.

A signature list is only ever as good as the last wall it met. That is why it is not the check we rely on.

The two-request test that does not need a signature #

Fetch the homepage. Then fetch a path that cannot exist — a random string nobody would ever have published. Compare the two answers.

A real site gives two different responses: a page, and a 404 that looks nothing like it. A wall gives the same response to both, because it never let the request reach the site at all. The scanner records the pair as refusalProbe: both statuses, both byte counts, whether the sizes are close, and whether either body carries a challenge signature. Same status, close size, and a signature on both is one interstitial served twice. Identical byte counts on two unrelated URLs is the same thing with no signature needed — one site answered every path, including the homepage, with a 75 KB block page, and no phrase in our list matched it.

This test has been refined once since. The first version also required both bodies to be HTML. A large retailer then answered every path, including its homepage, with a ten-byte Not found and an empty content-type, and the HTML requirement let that wall through to be graded as a site with no sitemap. Two identical tiny answers to the homepage and to a path that cannot exist are a wall whatever the content-type says; a page type is not what makes it one. The requirement was removed.

Why a wall is recorded and not graded #

When the probe says wall, the scan stops grading. There is no site behind the answer to grade. The record states what was received and who refused it, and the refusal is held for two days before it becomes a finding at all, on the rule that one observation is an anecdote. A tool that scores a challenge page D with no sitemap found and a perfect speed score has measured its own rejection, and every number on that report describes the firewall. We shipped exactly that report once, and it is the class of error the self-audit exists to catch.

The three findings this can raise, and what each one means #

ROBOTS_CTYPE is the mild one: the file parsed, but it was served as something other than text/plain, and some parsers are strict. ROBOTS_IS_HTML is the outage: the body is a page, no directive and no sitemap reference can be read, and every crawler proceeds as if the file did not exist. ROBOTS_IS_CATCHALL is the diagnosis underneath it: robots.txt is being answered with the same page as a path that does not exist, so nothing at that address is really being served — the shield is. Each names a different repair. The first is a header. The second and third are a firewall rule or a cache rule, and no edit to the file will touch them.

The cache is what turned one address into every address #

The bot defence was not the failure. Challenging a client making rapid requests from one address is a reasonable thing for an origin to do, and on its own it would have inconvenienced exactly that client.

The failure is that the response was cached. A cache key is built from the URL and a small set of headers. It does not include the reason the origin produced that particular body, because the origin has no way to say so in a 200. So a response generated for one address, because of that address’s behaviour became the canonical answer at that URL for everybody who asked afterwards.

That is the amplifier, and it is a configuration choice rather than an accident. A rule that caches 200s at the edge without qualification will store challenge pages, soft error pages and consent walls with equal enthusiasm, because all three are 200s carrying HTML.

Two changes remove it. Make the challenge uncacheable — Cache-Control: no-store on any interstitial, which costs nothing because nobody wants a challenge served from cache anyway. And return a status that means refusal, so the response is not eligible for the caching rule in the first place. Challenge pages fire on 0.8% of scans here, and a cacheable one is the version that turns a per-client control into a site-wide condition.

What a cached challenge does to the crawlers that behave #

Follow the consequence through and it gets worse than a missing file.

Most robots.txt parsers treat an unparseable or empty file as permission. That is the correct default — a site that has not stated restrictions has not restricted anything — but it means a robots.txt that arrives as 12 KB of challenge markup is read as everything allowed. The file underneath listed twenty-two agents individually, carried a Content-Signal line and declared a sitemap and an llms.txt. None of it was delivered, and its absence did not read as an error. It read as consent.

So a bot defence, cached, had the net effect of removing the site’s crawl restrictions for every well-behaved operator on the internet, while the client that triggered the challenge was unaffected because it was never reading robots.txt in good faith to begin with. The control inverted: the compliant crawlers lost the policy, and the non-compliant one lost nothing.

Operators that keep their own cached copy of your robots.txt compound it further, because they are now holding a challenge page as your stated policy for however long their cache lives.

The three monitors that would have caught it, none of which check status #

Every monitor on that site reported healthy, and all of them were checking the same thing. The checks that would have fired are cheap and none of them looks at the status line:

The third is the strongest and the least used. A known-string assertion does not need to recognise the wall, only to recognise your own file, which you control and can therefore make an assumption about safely. Any monitor that supports a body match supports it.

Status-only monitoring is the default because it is the cheapest thing to offer and it works for the failure everyone imagines: the server being down. It is silent on every failure where the server is up and answering with the wrong thing, which is most of what actually goes wrong at the machine layer.

Why the signature list is the wrong primary instrument #

Our own scanner keeps a list of body signatures that mark verification and interstitial pages. It did not contain this one, which is how the finding surfaced in the first place.

The honest conclusion is not that the list needed another entry. It is that signature matching is a lagging instrument by construction: it can only recognise walls somebody has already met and written down, and every new vendor, version or localisation produces a page it has never seen. Maintaining it is an arms race against strings.

The contrast test needs no prior knowledge at all. Ask for a path that cannot exist, compare that response to the real one, and a wall gives itself away by answering both identically — because a wall is not reading the path, it is deciding about the client. That property holds for every wall that has ever been built and every one that will be, which is what makes it the primary check. It fires as 3.2% in this corpus.

The signature list stays, demoted: it turns something is in the way into this is a Cloudflare interstitial, which is useful for the reader and worthless as the thing the detection depends on.

How to check your own in ten seconds #

Fetch your robots.txt and look at the content type, not the status:

curl -sI https://yoursite.com/robots.txt | grep -i content-type

It should say text/plain. If it says text/html, something is answering in place of your file, and the 200 beside it means nothing.

Then run the second request: fetch /this-path-does-not-exist- followed by a few random characters, and compare its size to the robots.txt answer. If they are within a few hundred bytes of each other, you are reading a wall, not a file. The same pair of tests applies to your sitemap and any llms.txt. A file that returns 200 and cannot be parsed is worse than one that 404s, because a 404 at least tells the truth — the terms are in the glossary.

What this scanner does about it now #

Every scan fetches a randomised path that cannot exist alongside the homepage and the machine files. When the nonexistent path answers 200 with the same HTML a machine file answers, that file is recorded as absent, not present, and the report says the site answered with its catch-all page. When every path answers identically in the way challenge walls do, the scan stops and reports that it was refused rather than grading the wall. The same rule reached llms.txt only after the scanner's own self-audit noticed two of its layers disagreeing about one file.

The cache rule that turns a nuisance into an outage #

If your CDN caches machine files, and it should, the rule must not store a response whose content type is wrong for the path. A robots.txt that arrives as text/html is never a robots.txt, and caching it hands the challenge to every crawler for the length of the TTL. The purge that returned 200 and evicted nothing and a cache rule does not evict what is already cached are the two follow-ups; the glossary defines challenge page at 200 and edge cache pinning.

Questions this post answers

Can robots.txt return 200 and still be broken?

Yes. A cached bot-protection challenge page can sit at that path and answer 200 with HTML. Uptime monitors call it healthy because they check the status code, not the content type or the body.

How do I check my own robots.txt in ten seconds?

Fetch the bare URL from outside your own session and read two things: the content type, which must be text/plain, and the first line of the body. HTML there means no crawler can read your rules.

What does a crawler do when robots.txt is unreadable?

Parsers that cannot read the file may treat the site as fully disallowed or fully allowed depending on the parser. Either way your actual rules are not the ones being applied.

Why does my robots.txt return 200 but crawlers ignore it?

Because a 200 describes the transaction, not the content. The likely causes are a bot challenge or interstitial being served in place of the file, a soft error page, or the file being returned as HTML. Check the content type and the body length rather than the status, and confirm the body actually contains a User-agent line.

Can a bot protection page break robots.txt for everyone?

Yes, if the response gets cached. A challenge generated for one address is stored against the URL, and every later request is answered from that stored copy. Most parsers read an unparseable robots.txt as permission, so the net effect is that the site's crawl restrictions disappear for well-behaved operators while the client that triggered the challenge is unaffected.

How do I detect a verification wall without a signature list?

Request the real path and an impossible one, then compare the two responses. A working site answers them differently, with a page and a 404 that look nothing alike. A wall answers both with the same body, because it is deciding about the client rather than reading the path. That test works on walls nobody has catalogued yet.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works