CrawlCheck

Agent view · Findings · 2026-08-13

robots.txt returns 200 but no crawler can read it

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/a-file-that-answered-200-and-could-not-be-read: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/a-file-that-answered-200-and-could-not-be-read>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes19,516
words2,931
sha-256c8344c5b7bbed10e58b9ac279486f6c07c3722046e5e233b8d4a7877d8eed4fa
repr-digestsha-256=:yDRMW3u+0Q5YuawnlIb2wHw3IgRuXiM7jUp4d9ju1Po=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795182384;expires=1795185984;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="poA_POfyVKx4ZcHmpAkyzg";tag="crawlcheck-page"
signaturesig1=:fvRerHnBjGIXeNPnLLD/ruAosWn8HGGvnwq0XWnjL2gQNfchPdYIxFqoKAdoy1U7bxx98jdfEzhFxn30zp7xDA==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Findings · 2026-08-13 · By [VSNARY | Emmanuel Orta](/author) · 5 views

# robots.txt returns 200 but no crawler can read it

Every monitor said the file was fine. Every client, from Googlebot to curl, received a 12 KB verification page instead, and an llms.txt elsewhere turned out to be a login wall. How it happens, and the two-request test that catches it.

A 200 status says the server answered, not that it served the file. One site's robots.txt returned 200 to a browser, Googlebot, GPTBot and curl, and every one received a 12 KB verification page instead of the rules. A bot defence had challenged one address and a cache rule stored that page for everyone. Most parsers read an unreadable robots.txt as everything allowed. Check the content type and the body, not the status: robots.txt and llms.txt should be text/plain or text/markdown, and should not match the page a nonexistent path returns.

On this page

- Why did every monitor say the file was fine?
- How did a defence against one address reach every crawler?
- The same trap at llms.txt: a login wall answering 200
- Why a list of known challenge pages is not enough
- The two-request test that needs no signature
- Three failures, three different repairs
- Which monitor checks would have caught it?
- How to check your own in ten seconds

What each checker saw on the same robots.txt

**HTTP 200**uptime monitor: healthy

**Parsed**validator: no directives, so everything allowed

**12KB of HTML**what a crawler actually received

Three readings of one file. Only the third is about the file.

A site’s `/robots.txt` answered **HTTP 200** to every client we sent: a browser, Googlebot, GPTBot, curl and our own crawler. All of them got the same 12 KB of HTML, a verification interstitial that reads *one moment, please, your request is being verified*.

Underneath it, the real file was excellent: twenty-two agents allowed individually, a [Content-Signal](/glossary/content-signal) line, a sitemap and an [llms.txt](/glossary/llms-txt) declared. **Nobody could read a word of it**, and every monitor on the site reported it healthy.

## Why did every monitor say the file was fine? #

Uptime checks look at the status code, and the status code was 200. Validators fetch the file and parse what they get; what they got was HTML, so they found no directives, which reads as *everything is allowed*, the exact opposite of what the file said. Our own scan reported it as no [robots.txt](/glossary/robots-txt) served. That is the trap: **an absence and a [refusal](/glossary/refusal) look identical unless you check what the body is.**

| Reader | What it concluded from the same 200 |
| --- | --- |
| HTTP uptime monitor | healthy |
| Validator that parses the file | no directives, so everything is allowed |
| A crawler | 12 KB of HTML: a verification interstitial |
| Clients that received exactly this | a browser, Googlebot, GPTBot, curl and our own crawler |

## How did a defence against one address reach every crawler? #

The challenge came from the origin, aimed at one address making rapid requests. On its own that is reasonable and would have inconvenienced only that client. The origin even marked its answer correctly, with `Cache-Control: private, no-store`: this response is for one requester only.

The damage came from an [edge cache](/glossary/edge-cache) rule written to make machine files fast. It overrode the origin’s caching headers, which is what an override does by design, so the page written for one suspicious address was stored against the [URL](/glossary/url) and handed to every crawler for the next five minutes. A cache key is built from the URL and a few headers; it has no way to know a 200 is a challenge.
A defence aimed at one client became the site-wide answer because a cache rule stored a response the origin had marked as for one requester only.
Follow the consequence through and it is worse than a missing file. Most parsers treat an unreadable robots.txt as permission, so the cached challenge removed the site’s crawl restrictions for every well-behaved operator, while the client that triggered it was never reading robots.txt in good faith anyway. **The compliant crawlers lost the policy; the non-compliant one lost nothing.** Operators that keep their own copy of your robots.txt then hold a challenge page as your policy for as long as their copy lives. Challenge pages answered with 200 turn up on 0.3% of scanned sites here.

Two changes remove the amplifier. Send `Cache-Control: no-store` on every interstitial and make sure no edge rule overrides it, and answer a refusal with a status that means refusal, so it is never eligible for a rule that caches 200s. [The machine-file caching guide](/guides/machine-file-caching-stale-split-and-blocked-behind-the-edge) shows how to see which copy a crawler is getting.

## The same trap at llms.txt: a login wall answering 200 #

A large community site answers `/llms.txt` with HTTP 200 and 8,401 bytes of HTML: a script that submits a form on load, a favicon link, a page title, and nothing a language model could use. It is the same page the site returns for a path that cannot exist. The site has no llms.txt at all; it has a login wall wearing a status code.

We got this one half wrong. On the same scan, one part of CrawlCheck read the content type and called the file absent, while another saw the 200 and called it present. Our [self-audit](/glossary/self-audit) flags any two readings that describe one path differently, which is how the disagreement surfaced before it reached a report. The reading that trusted the status was corrected: an llms.txt answered with HTML is now reported as missing, whatever the status says.

## Why a list of known challenge pages is not enough #

On the day the robots.txt case happened, its interstitial wording was not on our list of known challenge pages, so a wording match could not flag the page that caused it. Matching known wording is a lagging instrument by construction: it recognises only walls somebody has already met, and every new vendor, version or translation produces a page it has never seen. Walls also vary far more than a wording list expects:

| Wall seen | Status | Body | What gave it away |
| --- | --- | --- | --- |
| Verification interstitial at robots.txt | 200 | 12 KB of HTML | HTML where a text file belongs |
| Login wall at llms.txt | 200 | 8,401 bytes of HTML | identical to the page for a nonexistent path |
| Shield answering every path | 202 | 190-byte meta-refresh | nothing 404s, every machine file fails |
| Block page on every path, homepage included | 200 | 75 KB, matched no known wording | identical byte counts on every path |
| A large retailer, every path | not stated | ten bytes, empty content type | two identical tiny answers |

Walls of this kind, where every path gets the same answer, turn up on 16% of scanned sites here.

## The two-request test that needs no signature #

Fetch the page you care about. Then fetch a path that cannot exist, a random string nobody would ever have published, and compare the two answers. A real site gives two different responses: a page, and a 404 that looks nothing like it. A wall gives the same response to both, because it is deciding about the client and never let the request reach the site. The same body at both paths is one interstitial served twice, whatever the status or content type says.

This matters for any tool that grades your site. A checker that scores a wall as a D with no sitemap and a perfect speed score has measured its own rejection, and every number on that report describes the firewall. We shipped exactly that report once; [the site that scored 15 because it refused us](/blog/the-site-that-scored-15-because-it-refused-us) is the write-up. If a report on your site looks implausibly empty, run the two-request test before acting on it.

## Three failures, three different repairs #

| What you see | What crawlers get | Where the repair is |
| --- | --- | --- |
| File parses but is served as something other than `text/plain` | Strict parsers may reject it | A response header |
| Body is an HTML page, no directives readable | No file, so everything allowed | A firewall or cache rule, not the file |
| robots.txt answers the same as a nonexistent path | A shield, not the site | A firewall or cache rule, not the file |

No edit to the file touches the second or third. Only the first is a fix you make in the file’s own configuration.

## Which monitor checks would have caught it? #

Every monitor on that site checked the same thing, the status line. These assertions are cheap and none of them looks at the status:

| Assertion | What it catches |
| --- | --- |
| [Content type](/glossary/content-type) is `text/plain` | HTML served in place of the file, whatever the status |
| Body length within an expected band | A 12 KB interstitial where a 2 KB file belongs, and an empty file |
| Body contains the literal `User-agent:` | Anything that is not a robots.txt, including a wall nobody has seen before |

The third is the strongest and the least used. It does not need to recognise the wall, only your own file, which you control. Any monitor that supports a body match supports it; [AI crawler monitoring vs uptime monitors](/guides/monitoring-ai-crawler-access-vs-uptime-and-change-monitors) covers which tools do.

## How to check your own in ten seconds #

Look at the content type, not the status, for each machine file:

```
curl -sI https://yoursite.com/robots.txt | grep -i content-type
curl -sI https://yoursite.com/llms.txt | grep -i content-type
```

robots.txt should say `text/plain`; llms.txt `text/plain` or `text/markdown`. Then compare sizes against a path that cannot exist:

```
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" https://yoursite.com/robots.txt
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" https://yoursite.com/this-path-does-not-exist-q7x
```

If the two answers are within a few hundred bytes of each other, you are reading a wall, not a file. The same test applies to your sitemap. A machine file that returns 200 and cannot be parsed is worse than one that 404s, because a 404 at least tells the truth. [The llms.txt guide](/guides/how-to-write-an-llms-txt-and-verify-it) covers the rest of that file’s shape, and the terms are in the [glossary](/glossary).

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fa-file-that-answered-200-and-could-not-be-read%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fa-file-that-answered-200-and-could-not-be-read%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=robots.txt%20returns%20200%20but%20no%20crawler%20can%20read%20it)[Email](mailto:?subject=robots.txt%20returns%20200%20but%20no%20crawler%20can%20read%20it&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fa-file-that-answered-200-and-could-not-be-read)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### Can robots.txt return 200 and still be broken?

Yes. A cached bot-protection challenge page can sit at that path and answer 200 with HTML. Uptime monitors call it healthy because they check the status code, not the content type or the body.

### What does a crawler do when robots.txt is unreadable?

Most parsers treat a file they cannot read as no restrictions at all, so a robots.txt that arrives as challenge markup is read as everything allowed. Either way, your actual rules are not the ones being applied.

### Can a bot protection page break robots.txt for everyone?

Yes, if the response gets cached. A challenge generated for one address is stored against the URL, and every later request is answered from that stored copy, so a per-client control becomes a site-wide condition.

### Can an llms.txt be served as HTML?

No. The convention is plain text or markdown. A 200 that carries HTML at /llms.txt is a catch-all page, a challenge or a login form, and none of those is the file. Check the content type and the first line, not the status.

### How do I detect a verification wall without a signature list?

Request the real path and an impossible one, then compare the two responses. A working site answers them differently. A wall answers both with the same body, because it is deciding about the client rather than reading the path. That test works on walls nobody has catalogued yet.

### What should a robots.txt monitor check instead of the status?

Three things: that the content type is text/plain, that the body length is in the range you expect, and that the body contains the literal text User-agent:. The last one recognises your own file rather than trying to recognise every possible wall.

## Related findings

[2026-09-16 HTTP status codes for crawlers: what a 200 can hide A 200 can carry a bot challenge and a 403 can be a wall on every path. How to read status codes the way crawlers get them, with cu](/guides/http-status-codes-what-a-crawler-actually-gets)[2026-10-03 How to test robots.txt for GPTBot and ClaudeBot: 5 tools compared How to test robots.txt for GPTBot, ClaudeBot and PerplexityBot: five tools compared, a two-minute test, and why delivery matters m](/guides/how-to-test-robots-txt-for-gptbot-and-claudebot)[2026-08-28 How to check if AI crawlers can read your site Five curl checks that show whether GPTBot, ClaudeBot and PerplexityBot can read your site: content type, cache, crawler vs browser](/guides/how-to-check-if-ai-crawlers-can-read-your-site)

Where this is used

Guides that cite this finding:

- [Do AI crawlers and agents respect robots.txt?](/guides/do-ai-crawlers-respect-robots-txt)
- [How to check if AI crawlers can read your site](/guides/how-to-check-if-ai-crawlers-can-read-your-site)
- [Is Cloudflare blocking AI crawlers on your site?](/guides/is-cloudflare-blocking-ai-crawlers-on-your-site)
- [XML sitemap audit: check the file and every URL in it](/guides/xml-sitemap-audit)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [/](/) · [Listing check](/listing-check) · [SDK](/sdk) · [Certificates](/certificates) | **Registry:** [crestviewbuickgmc.com](/registry/crestviewbuickgmc.com) · [gosuslugi.ru](/registry/gosuslugi.ru) · [meta-council.com](/registry/meta-council.com) · [studioxnews.ca](/registry/studioxnews.ca) · [Place: Tucson](/registry/place/az/tucson) | **Datasets:** [South Carolina](/datasets/state/south-carolina) · [Technologies: Termly](/datasets/technologies/termly) · [Problems: Robots group name typo](/datasets/problems/robots-group-name-typo)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /blog/a-file-that-answered-200-and-could-not-be-read.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.