Findings · 2026-09-01 · By VSNARY | Emmanuel Orta · 0 views

llms.txt returned 200 and was a login wall: why status codes lie about machine files

The same page that answered a nonexistent path answered /llms.txt. One layer of CrawlCheck noticed; the other called it present.

A large community site answers /llms.txt with HTTP 200. The body is 8,401 bytes of HTML: a script that submits a form on load, a favicon link, a page title, and nothing a language model could use. It is the same shell the site returns for a path that cannot exist. It is a login wall wearing a status code.

Two parts of CrawlCheck looked at that response on the same scan. One read the content type, saw text/html, and said the file was not there: an llms.txt is text by definition, and this was not text. The other saw the 200 and marked the file present. One response, two verdicts.

The contradiction is what got reported, not the file #

We did not find this by reading the site. We found it because CrawlCheck audits its own output and flags any two of its own readings that describe the same path differently. A disagreement there is never a finding about the site. It is a bug in CrawlCheck, and it means one of the two readings is wrong.

The second reading was wrong. It already treated a sitemap answered with the site’s catch-all page as absent, but had never applied that to llms.txt, so identical bytes were called absent at one path and present at the next.

What a 200 is allowed to mean #

A status code is a claim by the server about the request, not about the content. For a machine file the content is the whole point, so the test has to be on the body. An llms.txt that answers 200 with HTML is absent. Whether it is a catch-all page, a challenge, or a login form does not matter; none of them is the file. We corrected the check: the file is now reported as missing, and both readings agree.

This is the same shape as the robots.txt that answered 200 with a challenge page, and the same lesson: an absence and a refusal look identical unless you check the content type. What was new here is that the disagreement between two of our own readings was the alarm, which is exactly what the self-audit exists to do.

One response, two readings #

Two of our own readings described the same path differently, which is the only reason this was caught before it reached a report.

How to check your own #

Fetch the file and look at the content type, then at the first line of the body:

curl -s -D - https://yoursite.com/llms.txt -o /tmp/llms.txt | grep -i content-type
head -c 200 /tmp/llms.txt

It should say text/plain or text/markdown, and the first characters should be a heading or a sentence, not <!DOCTYPE html>. If your platform answers every unknown path with your homepage, it answers /llms.txt that way too, and every checker that trusts the status code will tell you the file exists. The guide to writing and verifying an llms.txt covers the rest of the shape.

Questions this post answers

Can an llms.txt be served as HTML?

No. The convention is plain text or markdown. A 200 that carries HTML at /llms.txt is the site's catch-all page, a challenge, or a login form, and none of those is the file. Check the content type, not the status.

Why did two parts of CrawlCheck disagree?

One reading tested the content type and treated HTML as missing; the other had learned that rule for sitemaps but not for llms.txt. The self-audit flags any two of CrawlCheck's own readings that describe one path differently, which is how the gap was found.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works