Findings · 2026-11-19 · By VSNARY | Emmanuel Orta
Who reads llms.txt? Our check called 8 of 10 unverified
Our llms.txt reader check marked 8 of 10 readers 'unverified' and counted two of our own fetches as a confirmed AI assistant. Here is what changed and what the readers were.
On 19 November 2026, CrawlCheck's llms.txt reader check for crawlcheck.io listed 12 fetches in 30 days, each labelled by kind of network and country, and none confirmed as coming from the operator it names. The day before, the same panel marked 8 of 10 readers only 'unverified' and listed 2 fetches carrying a CrawlCheck tool name as a confirmed AI assistant. A tool name is now counted as a claim; only proven requests count as ours.
What the panel said #
Every CrawlCheck report has a small panel called the llms.txt reader check. llms.txt is a plain text file a site publishes to tell AI tools what to read. The reader check is a single link we give each site to paste into that file: anything that fetches the link has read the file and gone on to follow what it lists. The panel counts those fetches and says who made them.
On 18 November, the panel on our own site's report said this: 10 fetches in the last 30 days, 0 from an address the operator publishes. Under that, three rows. A browser-style user agent, 5 fetches. A client calling itself GeneralRetrieval-DocsFetch, 2. A client calling itself python-requests, 1. Each of the three ended in the same word: (unverified).
So 8 of the 10 readers carried one label, and the label said almost nothing. A user agent is the name a client gives itself in each request; anyone can type any name. "Unverified" told you the name was not proven. It did not tell you what the reader actually was, where it came from, or whether it was the same kind of thing as the other seven.
The other two were ours, and we labelled them wrong #
The remaining 2 fetches were worse. They carried the name CrawlCheck-Owner-Audit, a CrawlCheck tool name, and the panel listed them as a confirmed AI assistant. Our own audit traffic was being counted as a reader of our own file, and dressed up as the most trustworthy row on the panel.
Two errors, then. Eight readers with a label that explained nothing, and two fetches with a name we recognised, counted as if a name were proof.
What we changed on 18 and 19 November #
The owner of CrawlCheck saw the panel on the evening of 18 November and set a standing rule: every client that reads or even tries to read the site gets an identity. No bare "unverified" rows.
The first change went live that night. Every reader row now says what kind of thing it is and, where the reading has it, what kind of network it came from and the country. The rows changed from a verdict to a description.
The first version of those labels went further than it should. For about fifty minutes on the night of 18 November, the new labels on public reports printed the names of the companies that own the networks readers came from. A network's owner is not the same as the reader: a cloud provider rents its addresses to thousands of customers, and naming the provider next to a claim it never made is an accusation. We replaced the names with the kind of network and the country, and re-labelled stored reports at the moment they are shown so no older copy carries a name. The detail is still available to a site's own owner, for their own site only.
The second change went live just after midnight on 19 November and undid half of our first fix. The first fix had simply removed our own tool name from the count. But a tool name is a claim like any other, and anyone can send ours. The rule now: a fetch counts as CrawlCheck's own only when it carries proof, such as a signature with our own published key. A fetch that merely says it is CrawlCheck is counted like anyone else's and labelled claims a CrawlCheck name. The same sentence is on our policy page: our own signed requests are kept out of public counts, and a request that only says it is ours is recorded like any other.
What the readers turned out to be #
This is the panel on a report on crawlcheck.io scanned on 18 November, as it reads on 19 November. 12 fetches in 30 days, 0 confirmed as coming from the operator named.
| Reader, as it named itself | Fetches | Before | Now |
|---|---|---|---|
| Browser-style user agent | 5 | unverified | that day's fetches came from a cloud or hosting network (5); a company, home or mobile network (2) |
GeneralRetrieval-DocsFetch | 2 | unverified | claims this name; that day's fetches came from a cloud or hosting network (2); a company, home or mobile network (1) |
python-requests | 1 | unverified | claims this name, from a company, home or mobile network, US |
CrawlCheck-Owner-Audit | 4 | listed as a confirmed AI assistant | claims a CrawlCheck name, from a company, home or mobile network, US (2) |
Read row by row, the answer to "who reads our llms.txt" is now specific, and mostly unflattering. Nothing that followed the link proved it was the company or crawler it named. The largest group was browser-style: it used the opening a web browser sends, from networks that rent servers. People do not normally browse from a data centre, so those are most likely scripts dressed as browsers. python-requests is the default name of a common programming library, sent by whoever wrote the script. GeneralRetrieval-DocsFetch names a fetcher but offers nothing that proves who runs it.
Two things in the table are honest limits, not errors. Where a row says "that day's fetches came from", the reading covers days before 18 November, and for those days the panel can only say where all of that day's fetches came from, not which reader came from which network. And the counts on the original report, scanned before the change, show 10 fetches with "the network is not in this reading": a newer scan is needed to show it.
Why a label is not an identity #
The old panel treated identity as a yes-or-no question: is this the operator it claims to be? For the major search and AI crawlers that question has an answer, because their operators publish the addresses they crawl from or sign their requests, and anyone can check. The method is generic and we wrote it up in checking a crawler against its operator's own evidence.
Most readers of an llms.txt are not major crawlers. They are scripts, small tools and browser automation, and none of them publish anything to check against. For those, "unverified" is always true and never useful. What a site owner can actually learn is the kind of network the request came from, the country, and whether the name it gave matches anything. That is what the panel now prints.
There is a limit we state rather than hide. A browser-style request from a home or mobile network can be tied to its network and country, but not to a named company, unless it signs its requests, uses a published address range or resolves by DNS to its operator.
What did not change #
The panel still never prints a full address. Public reports show the kind of network and the country, nothing narrower. The reader check still costs a site nothing to add: one line in llms.txt. If you have not added yours, the steps are in how to write an llms.txt and verify it. And a reading of zero fetches is still a measurement, not a defect: it means nothing that fetched your llms.txt went on to follow what it lists.
What to take from this #
If a tool shows you a list of visitors marked "unverified", ask what the word is doing. It usually means the tool checked one thing, found no match, and stopped. The useful question is the next one: claimed what, from what kind of network, and is there anything that could prove it. Our panel was answering the first question for 8 readers of 10 and getting the other 2 wrong. It now answers the second for all of them.
CrawlCheck’s dataset when this page loaded: 570,959 sites measured.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
What is the llms.txt reader check?
A single link CrawlCheck gives each site to paste into its llms.txt file. Anything that fetches that link has read the file and followed what it lists. The report counts those fetches and labels each reader by the name it gave, the kind of network it came from and the country.
What did 'unverified' mean on the old panel?
Only that the reader's name could not be matched to its operator's published addresses. It did not say what the reader was or where it came from. On 18 November it was the only label on 8 of the 10 readers of crawlcheck.io's llms.txt.
What do the reader labels say now?
Each row says whether the reader claims a name or uses a browser-style user agent, then the kind of network (a cloud or hosting network, or a company, home or mobile network) and the country. A reader whose operator can be proven is marked as confirmed as coming from the operator it names.
Why are company names not shown for each network?
A network's owner is not the reader. Cloud providers rent addresses to many customers, so naming the provider next to a claim it never made would be an accusation. Public reports show the kind of network and the country. A site's owner can see more detail for their own site only.
Are CrawlCheck's own fetches counted?
Only when they cannot be proven. Requests signed with CrawlCheck's own published key are kept out of public counts. A fetch that only carries a CrawlCheck name is counted like any other reader and labelled 'claims a CrawlCheck name'.
Does the panel show full IP addresses?
No. Public reports show the kind of network and the country, nothing narrower. Full addresses are never printed on a report page, its text copy or its data file.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.