CrawlCheck

Agent view · Findings · 2026-11-19

Who reads llms.txt? Our check called 8 of 10 unverified

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/our-llms-txt-reader-check-called-8-of-10-readers-unverified: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/our-llms-txt-reader-check-called-8-of-10-readers-unverified>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes17,398
words2,656
sha-256f7a013409c651662de09ef314ca96b364386ee2a4da28c4d063c4effc725b530
repr-digestsha-256=:96ATQJxlFmLeCe8xTKlrNkOG7ipNooxNBjxO/8cltTA=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795183084;expires=1795186684;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="0hbGEUprZ7YyuTgSXQxDTg";tag="crawlcheck-page"
signaturesig1=:wEMmFFjb++C3TyOFuQArbQZobYSTgYkVXH1qDvPThwBZvb/aPbZClEs8MSnXiuOIzFtL9NIsr4dFl4zyhm0UCw==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Findings · 2026-11-19 · By [VSNARY | Emmanuel Orta](/author)

# Who reads llms.txt? Our check called 8 of 10 unverified

Our llms.txt reader check marked 8 of 10 readers 'unverified' and counted two of our own fetches as a confirmed AI assistant. Here is what changed and what the readers were.

On 19 November 2026, CrawlCheck's llms.txt reader check for crawlcheck.io listed 12 fetches in 30 days, each labelled by kind of network and country, and none confirmed as coming from the operator it names. The day before, the same panel marked 8 of 10 readers only 'unverified' and listed 2 fetches carrying a CrawlCheck tool name as a confirmed AI assistant. A tool name is now counted as a claim; only proven requests count as ours.

On this page

- What the panel said
- The other two were ours, and we labelled them wrong
- What we changed on 18 and 19 November
- What the readers turned out to be
- Why a label is not an identity
- What did not change
- What to take from this

## What the panel said #

Every CrawlCheck report has a small panel called the **[llms.txt](/glossary/llms-txt) reader check**. llms.txt is a plain text file a site publishes to tell AI tools what to read. The reader check is a single link we give each site to paste into that file: anything that fetches the link has read the file and gone on to follow what it lists. The panel counts those fetches and says who made them.

On 18 November, the panel on our own site's report said this: 10 fetches in the last 30 days, 0 from an address the operator publishes. Under that, three rows. A browser-style [user agent](/glossary/user-agent), 5 fetches. A client calling itself `GeneralRetrieval-DocsFetch`, 2. A client calling itself `python-requests`, 1. Each of the three ended in the same word: *(unverified)*.

So 8 of the 10 readers carried one label, and the label said almost nothing. A user agent is the name a client gives itself in each request; anyone can type any name. "Unverified" told you the name was not proven. It did not tell you what the reader actually was, where it came from, or whether it was the same kind of thing as the other seven.

## The other two were ours, and we labelled them wrong #

The remaining 2 fetches were worse. They carried the name `CrawlCheck-Owner-Audit`, a CrawlCheck tool name, and the panel listed them as a confirmed AI assistant. Our own audit traffic was being counted as a reader of our own file, and dressed up as the most trustworthy row on the panel.

Two errors, then. Eight readers with a label that explained nothing, and two fetches with a name we recognised, counted as if a name were proof.

## What we changed on 18 and 19 November #

The owner of CrawlCheck saw the panel on the evening of 18 November and set a standing rule: every client that reads or even tries to read the site gets an identity. No bare "unverified" rows.

The first change went live that night. Every reader row now says what kind of thing it is and, where the reading has it, what kind of network it came from and the country. The rows changed from a verdict to a description.

The first version of those labels went further than it should. For about fifty minutes on the night of 18 November, the new labels on public reports printed the names of the companies that own the networks readers came from. A network's owner is not the same as the reader: a cloud provider rents its addresses to thousands of customers, and naming the provider next to a claim it never made is an accusation. We replaced the names with the kind of network and the country, and re-labelled stored reports at the moment they are shown so no older copy carries a name. The detail is still available to a site's own owner, for their own site only.

The second change went live just after midnight on 19 November and undid half of our first fix. The first fix had simply removed our own tool name from the count. But a tool name is a claim like any other, and anyone can send ours. The rule now: a fetch counts as CrawlCheck's own only when it carries proof, such as a signature with our own published key. A fetch that merely says it is CrawlCheck is counted like anyone else's and labelled *claims a CrawlCheck name*. The same sentence is on [our policy page](/policy): our own signed requests are kept out of public counts, and a request that only says it is ours is recorded like any other.

## What the readers turned out to be #

This is the panel on [a report on crawlcheck.io scanned on 18 November](/r/dr7fdudgya), as it reads on 19 November. 12 fetches in 30 days, 0 confirmed as coming from the operator named.

| Reader, as it named itself | Fetches | Before | Now |
| --- | --- | --- | --- |
| Browser-style user agent | 5 | unverified | that day's fetches came from a cloud or hosting network (5); a company, home or mobile network (2) |
| `GeneralRetrieval-DocsFetch` | 2 | unverified | claims this name; that day's fetches came from a cloud or hosting network (2); a company, home or mobile network (1) |
| `python-requests` | 1 | unverified | claims this name, from a company, home or mobile network, US |
| `CrawlCheck-Owner-Audit` | 4 | listed as a confirmed AI assistant | claims a CrawlCheck name, from a company, home or mobile network, US (2) |

Read row by row, the answer to "who reads our llms.txt" is now specific, and mostly unflattering. Nothing that followed the link proved it was the company or crawler it named. The largest group was browser-style: it used the opening a web browser sends, from networks that rent servers. People do not normally browse from a data centre, so those are most likely scripts dressed as browsers. `python-requests` is the default name of a common programming library, sent by whoever wrote the script. `GeneralRetrieval-DocsFetch` names a fetcher but offers nothing that proves who runs it.

Two things in the table are honest limits, not errors. Where a row says "that day's fetches came from", the reading covers days before 18 November, and for those days the panel can only say where all of that day's fetches came from, not which reader came from which network. And the counts on the original report, scanned before the change, show 10 fetches with "the network is not in this reading": a newer scan is needed to show it.

## Why a label is not an identity #

The old panel treated identity as a yes-or-no question: is this the operator it claims to be? For the major search and AI crawlers that question has an answer, because their operators publish the addresses they crawl from or sign their requests, and anyone can check. The method is generic and we wrote it up in [checking a crawler against its operator's own evidence](/guides/how-to-verify-googlebot-bingbot-and-ai-crawlers).

Most readers of an llms.txt are not major crawlers. They are scripts, small tools and browser automation, and none of them publish anything to check against. For those, "unverified" is always true and never useful. What a site owner can actually learn is the kind of network the request came from, the country, and whether the name it gave matches anything. That is what the panel now prints.

There is a limit we state rather than hide. A browser-style request from a home or mobile network can be tied to its network and country, but not to a named company, unless it signs its requests, uses a published address range or resolves by DNS to its operator.

## What did not change #

The panel still never prints a full address. Public reports show the kind of network and the country, nothing narrower. The reader check still costs a site nothing to add: one line in llms.txt. If you have not added yours, the steps are in [how to write an llms.txt and verify it](/guides/how-to-write-an-llms-txt-and-verify-it). And a reading of zero fetches is still a measurement, not a defect: it means nothing that fetched your llms.txt went on to follow what it lists.

## What to take from this #

If a tool shows you a list of visitors marked "unverified", ask what the word is doing. It usually means the tool checked one thing, found no match, and stopped. The useful question is the next one: claimed what, from what kind of network, and is there anything that could prove it. Our panel was answering the first question for 8 readers of 10 and getting the other 2 wrong. It now answers the second for all of them.

CrawlCheck’s dataset when this page loaded: 570,959 sites measured.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Four-llms-txt-reader-check-called-8-of-10-readers-unverified%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Four-llms-txt-reader-check-called-8-of-10-readers-unverified%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Who%20reads%20llms.txt%3F%20Our%20check%20called%208%20of%2010%20unverified)[Email](mailto:?subject=Who%20reads%20llms.txt%3F%20Our%20check%20called%208%20of%2010%20unverified&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Four-llms-txt-reader-check-called-8-of-10-readers-unverified)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### What is the llms.txt reader check?

A single link CrawlCheck gives each site to paste into its llms.txt file. Anything that fetches that link has read the file and followed what it lists. The report counts those fetches and labels each reader by the name it gave, the kind of network it came from and the country.

### What did 'unverified' mean on the old panel?

Only that the reader's name could not be matched to its operator's published addresses. It did not say what the reader was or where it came from. On 18 November it was the only label on 8 of the 10 readers of crawlcheck.io's llms.txt.

### What do the reader labels say now?

Each row says whether the reader claims a name or uses a browser-style user agent, then the kind of network (a cloud or hosting network, or a company, home or mobile network) and the country. A reader whose operator can be proven is marked as confirmed as coming from the operator it names.

### Why are company names not shown for each network?

A network's owner is not the reader. Cloud providers rent addresses to many customers, so naming the provider next to a claim it never made would be an accusation. Public reports show the kind of network and the country. A site's owner can see more detail for their own site only.

### Are CrawlCheck's own fetches counted?

Only when they cannot be proven. Requests signed with CrawlCheck's own published key are kept out of public counts. A fetch that only carries a CrawlCheck name is counted like any other reader and labelled 'claims a CrawlCheck name'.

### Does the panel show full IP addresses?

No. Public reports show the kind of network and the country, nothing narrower. Full addresses are never printed on a report page, its text copy or its data file.

## Related findings

[2026-09-01 Dead link or blind spot? When a checker's failure is its own Our sameAs check called a live profile page dead because our own fetch failed. It now reports such targets as unverifiable. How to](/blog/the-scanner-that-cannot-fetch-its-own-host)[2026-09-01 Seven site types, and the report checks we corrected We ran seven very different sites through every graded check and found some failing sites for non-defects. Which checks changed, a](/blog/we-read-every-scored-row-against-seven-sites-here-is-what-we-unscored)[2026-09-30 Stale cached robots.txt or llms.txt? Two fetches tell you CDNs cache robots.txt and llms.txt hardest. Two fetches, bare and cache-busted, show when edge and origin disagree, and why a new](/guides/machine-file-caching-stale-split-and-blocked-behind-the-edge)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Registry:** [AI policy](/registry/aipolicy) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [/](/) · [Conduct: Duckduckbot](/agents/conduct/duckduckbot) | **AI crawlers:** [Ccbot vs GPTBot](/ai-crawlers/compare/ccbot-vs-gptbot) | **Blog:** [Six in ten websites already use post-quantum encryption. Whether yours does mostly depends on one thing](/blog/post-quantum-key-exchange-six-in-ten-websites-november-2026) | **Glossary:** [E-E-A-T](/glossary/e-e-a-t) · [Normalized discounted cumulative gain](/glossary/normalized-discounted-cumulative-gain) · [Search intent](/glossary/search-intent) · [Data exfiltration](/glossary/data-exfiltration) · [canonical origin](/glossary/canonical-origin)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /blog/our-llms-txt-reader-check-called-8-of-10-readers-unverified.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.