Findings · 2026-08-13

The file that returned 200 to every crawler and could not be read

A site’s /robots.txt answered HTTP 200 to every client we sent — a browser, Googlebot, GPTBot, curl, our own crawler. All of them got the same 12KB of HTML: a verification interstitial that reads one moment, please, your request is being verified.

Underneath it, the real file was excellent: twenty-two agents allowed individually, a Content-Signal line, sitemap and llms.txt declared. Nobody could read a word of it.

Why every monitor said it was fine

Uptime checks look at the status code. The status code was 200. Validators fetch the file and parse what they get; what they got was HTML, so they reported no robots directives — which reads as everything is allowed, the exact opposite of what the file said.

Our own scan reported it as no robots.txt served, every agent allowed by default. That is the trap: an absence and a refusal look identical unless you check the content type.

What was actually happening

The challenge came from the origin, not the CDN — a bot defence firing at one address making rapid requests. Ordinary, and it would have been harmless on its own.

The damage came from a caching rule that stored the response. A per-address defence became a site-wide outage of the machine layer, because a cache cannot tell that a 200 is a challenge page. Every crawler for the next five minutes was served the cached interstitial.

How to check your own in ten seconds

Fetch your robots.txt and look at the content type, not the status:

curl -sI https://yoursite.com/robots.txt | grep -i content-type

It should say text/plain. If it says text/html, something is answering in place of your file, and the 200 beside it means nothing.

The same test applies to your sitemap and any llms.txt. A file that returns 200 and cannot be parsed is worse than one that 404s, because a 404 at least tells the truth.

All findings · The dataset · How the dataset works