Findings · 2026-09-20 · By VSNARY | Emmanuel Orta · 0 views

Our llms.txt is linked from every page. Crawlers fetch it about as often as a blog post

Sixteen AI and search crawlers made 12,831 requests to crawlcheck.io in two weeks. Eight were for llms.txt, linked from every page. Per URL, that is an ordinary page's rate, not a front door's.

Every page on crawlcheck.io links to our llms.txt. Our robots.txt names it twice. The home page links it in the body as well as the head. If any site gives a crawler every chance to find its llms.txt, it is this one, because telling people to publish one is part of what we do.

So we counted how often crawlers actually fetched it. Between 6 and 19 September 2026, sixteen AI and search crawler names made 12,831 requests to crawlcheck.io. Eight of them were for /llms.txt.

Our first reading of these numbers was wrong #

The first time we looked, we compared totals: 3,472 requests for our registry pages against 8 for llms.txt, and concluded that crawlers read pages and ignore the file. That comparison is unfair. The registry is 1,119 separate pages and llms.txt is one file. Divided per URL, an average registry page was fetched 3.1 times in the fortnight. llms.txt was fetched 8 times. On that measure it did better than the average page, not worse.

We say this first because the corrected number is less dramatic and more useful, and because a finding that only survives when the denominator is left out is not a finding. The count below is per URL throughout.

How often each kind of URL was fetched #

The sixteen names counted are GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Amazonbot, Meta-ExternalAgent, CCBot and Bytespider. Days are UTC, and the window stops at 19 September, the last full day before this post.

Two groups separate cleanly. The home page, robots.txt and the sitemap were fetched hundreds of times each: they are where a crawl starts. Everything else, including every machine file we publish, sits between three and eight requests per URL, which is the rate of an ordinary page. llms.txt is not treated as a front door. It is treated as one more page.

Discoverable is not the same as fetched #

The low count is not a discovery problem. Our robots.txt names llms.txt, agents.md and entitymap.json. Every page carries <link rel="alternate"> tags to all four machine files, and the home page links them in its body. Crawlers fetched the home page 307 times and robots.txt 299 times in the window, and every one of those responses pointed at llms.txt. The pointer was read hundreds of times; the file it pointed at, eight.

We checked those routes against the code we archived before the window opened, because a finding about fetching is only as strong as the claim that the file could be found. The robots.txt pointers and the head links to llms.txt, agents.md and entitymap.json were already there. The media kit is the exception: the file and its link went live on 10 September, so its seven requests come from ten days, not fourteen.

One detail counts against us. Our robots.txt carries two non-standard lines, Llms: and EntityMap:. The Robots Exclusion Protocol, RFC 9309, defines user-agent, allow and disallow rules and leaves any other line to each crawler's discretion, and no crawler operator we know of documents reading either of ours. They cost nothing and, as far as these counts can show, do nothing.

Which crawlers fetched the files #

No crawler fetched any of these files more than three times in fourteen days. ClaudeBot never requested llms.txt; in the same window it requested robots.txt 35 times. GPTBot touched four of the five files, twice at most each. None of this says a file was ignored after it was fetched. It says the files were fetched about as often as a page that nobody links to specially.

What this does not show #

It is one site and fourteen days. A site with a larger crawl budget may see a different pattern, and so may a site that lists its machine files in its sitemap. Ours does not: the sitemap lists pages only, and that is the next variable we will change and measure.

A fetch is not use. These are requests that reached our server. What an answer engine does with a file after fetching it, and what it then says in an answer, cannot be seen from the server side at all.

The counts are by claimed name. We verify crawler identities against the address ranges operators publish, and the rates are on the dataset page, but the per-file counts here are not yet split by verification. Most of the names above verify at high rates on this site; the hosting provider's own flag verified none of the PerplexityBot requests, although our check against Perplexity's published ranges places most PerplexityBot traffic to this site inside them. The two methods consult different lists; from 20 September our log records both, and verification per file.

What to do with it #

If a fact matters, put it in the pages. A page is fetched on the same crawl that discovers it; a side file is fetched on its own schedule, which on this evidence is the schedule of an ordinary page. Structured data inside the HTML travels with the page. The same data in a separate file travels only when that file is chosen.

Keep the llms.txt. It is cheap, some clients read it, and it does no harm. Just do not rely on it as the only place a fact appears, and check your own logs before assuming it is read. What the file can and cannot do is in our guide to whether llms.txt works, and how to write one that verifies is in the llms.txt writing guide.

Next we list the machine files in our sitemap and record verification per path, then compare the same fortnight again. If the rate changes, we will say by how much.

Questions this post answers

Do AI crawlers read llms.txt?

On crawlcheck.io between 6 and 19 September 2026, sixteen AI and search crawler names fetched it 8 times, about as often as an average blog post on the site and far less than robots.txt or the home page. That is one site; check your own logs before assuming either way.

Why compare requests per URL instead of totals?

Because a site has one llms.txt and many pages. Our registry has 1,119 pages; its 3,472 requests work out to about three per page, fewer than llms.txt received. Totals made the file look ignored. Per URL it is fetched like an ordinary page.

Was the file hard to find?

No. Our robots.txt names it, every page links it in the head, and the home page links it in the body. Crawlers fetched the home page and robots.txt hundreds of times in the same window.

Should I remove my llms.txt?

No. It is cheap and some clients read it. Do not rely on it as the only place a fact appears: put the facts you need crawlers to use in the pages themselves, which are fetched on every crawl that reaches them.

Did ClaudeBot fetch llms.txt?

Not on our site in this window: zero requests for llms.txt and 35 for robots.txt. One site over two weeks is a measurement, not a statement about any operator's policy.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works