Findings · 2026-09-20 · By VSNARY | Emmanuel Orta · 4 views

Linked from every page, our llms.txt was fetched 8 times

In two weeks, sixteen AI and search crawler names made 12,831 requests to crawlcheck.io and asked for llms.txt eight times. Per URL, that is the rate of an ordinary page, not a front door.

Every page on crawlcheck.io links to our llms.txt. Our robots.txt names it twice. The home page links it in the body as well as the head. If any site gives a crawler every chance to find its llms.txt, it is this one, because telling people to publish one is part of what we do.

So we counted how often crawlers actually fetched it. Between 6 and 19 September 2026, sixteen AI and search crawler names made 12,831 requests to crawlcheck.io. Eight of them were for /llms.txt.

Our first reading of these numbers was wrong #

The first time we looked, we compared totals: 3,472 requests for our registry pages against 8 for llms.txt, and concluded that crawlers read pages and ignore the file. That comparison is unfair. The registry is 1,119 separate pages and llms.txt is one file. Divided per URL, an average registry page was fetched 3.1 times in the fortnight. llms.txt was fetched 8 times. On that measure it did better than the average page, not worse.

We say this first because the corrected number is less dramatic and more useful, and because a finding that only survives when the denominator is left out is not a finding. The count below is per URL throughout.

How often each kind of URL was fetched #

The sixteen names counted are GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Amazonbot, Meta-ExternalAgent, CCBot and Bytespider. Days are UTC, and the window stops at 19 September, the last full day before this post.

Two groups separate cleanly. The home page, robots.txt and the sitemap were fetched hundreds of times each: they are where a crawl starts. Everything else, including every machine file we publish, sits between three and eight requests per URL, which is the rate of an ordinary page. llms.txt is not treated as a front door. It is treated as one more page.

Discoverable is not the same as fetched #

The low count is not a discovery problem. Our robots.txt names llms.txt, agents.md and entitymap.json. Every page carries <link rel="alternate"> tags to all four machine files, and the home page links them in its body. Crawlers fetched the home page 307 times and robots.txt 299 times in the window, and every one of those responses pointed at llms.txt. The pointer was read hundreds of times; the file it pointed at, eight.

We checked those routes against the site as it stood before the window opened, because a finding about fetching is only as strong as the claim that the file could be found. The robots.txt pointers and the head links to llms.txt, agents.md and entitymap.json were already there. The media kit is the exception: the file and its link went live on 10 September, so its seven requests come from ten days, not fourteen.

One detail counts against us. Our robots.txt carries two non-standard lines, Llms: and EntityMap:. The Robots Exclusion Protocol, RFC 9309, defines user-agent, allow and disallow rules and leaves any other line to each crawler's discretion, and no crawler operator we know of documents reading either of ours. They cost nothing and, as far as these counts can show, do nothing.

Which crawlers fetched the files #

No crawler fetched any of these files more than three times in fourteen days. ClaudeBot never requested llms.txt; in the same window it requested robots.txt 35 times. GPTBot touched four of the five files, twice at most each. None of this says a file was ignored after it was fetched. It says the files were fetched about as often as a page that nobody links to specially.

What this does not show #

It is one site and fourteen days. A site with a larger crawl budget may see a different pattern, and so may a site that lists its machine files in its sitemap. Ours lists pages only.

A fetch is not use. These are requests that reached our server. What an answer engine does with a file after fetching it, and what it then says in an answer, cannot be seen from the server side at all.

The counts are by claimed name. We verify crawler identities against the address ranges operators publish, and the rates are on the dataset page, but the per-file counts here are not yet split by verification. Most of the names above verify at high rates on this site, and a share of any crawler's traffic is forged, so a small count by name can include requests that were not the crawler at all.

What to do with it #

If a fact matters, put it in the pages. A page is fetched on the same crawl that discovers it; a side file is fetched on its own schedule, which on this evidence is the schedule of an ordinary page. Structured data inside the HTML travels with the page. The same data in a separate file travels only when that file is chosen.

Keep the llms.txt. It is cheap, some clients read it, and it does no harm. Just do not rely on it as the only place a fact appears, and check your own logs before assuming it is read. If a cache sits in front of it, also check that crawlers get the current copy, as in the machine-file caching guide. What the file can and cannot do is in our guide to whether llms.txt works, and how to write one that verifies is in the llms.txt writing guide.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Related findings

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works