CrawlCheck

Guides · 2026-09-30 · By · 2 views

Stale cached robots.txt or llms.txt? Two fetches tell you

Why one fetch cannot tell whether a machine file is healthy, the two-fetch check that can, the four outcomes, why a new cache rule did not fix a one-year robots.txt, and the purge order that works.

Machine files such as robots.txt, llms.txt and sitemap.xml are cached hardest by CDNs, so one fetch cannot show whether the edge and the origin agree. Fetch each file twice: at the bare URL, the way a crawler asks, and with a cache-busting query string. Different status codes mean a stale copy; HTML on one side means some crawlers get a page instead of the file; a cached 200 against an origin 403 means the cache is hiding a block. A new cache rule does not touch copies already stored, so purge, origin cache first, then CDN.

Check your own site free

About a minute. First scan free with your email, nothing installed.

robots.txt, llms.txt, sitemap.xml and an entity map are small files that rarely change, which is exactly what a CDN caches hardest. That is fine until one changes, or the origin starts refusing a crawler, because the edge keeps answering with whatever it stored and nobody fetching the bare URL can tell. CrawlCheck found this by being fooled by it: a bare request to one of our machine files returned a cached HTML interstitial at HTTP 200 while a cache-busted request to the same URL returned valid JSON. Then we found the files on our own sites carrying a one-year cache header nobody had set on purpose. This guide is the two-fetch check, the four outcomes, and the fixes that actually took effect.

GET /robots.txtbare, as a crawler asksGET /robots.txt?cc=x7k2pcache-bustedEdge cachestored copy answersEdge cachekey misses, passes onOriginthe file as it is nowComparestatus, type, bytesIf the two answers differ, the edge and the origin disagree, and only the bare answer is what crawlers get.
The two-fetch method. One fetch shows only the edge; the pair shows whether the edge still agrees with the origin.

Why is one fetch not enough? #

An edge cache is keyed on the URL. Ask for /robots.txt and you get the stored copy if there is one; the origin is not consulted. Ask for /robots.txt?cc=x7k2p and, unless the cache ignores query strings, the key misses and the request reaches the origin. Comparing the two tells you whether the edge and the origin agree about that file, in status, in content type and in bytes.

The four outcomes #

Status disagrees. The bare URL answers 200 and the origin 404, or the reverse. The edge is outliving a deleted or moved file, or a new file has not reached the edge yet. Seen on 0.4% of scanned sites. Compare status before content type: a 404 body is HTML by nature, so a status difference always looks like a type difference too.

Same status, different kind. Both answer 200, but one is HTML and the other is not. Some crawlers are getting a page where the file should be, usually a bot-challenge or maintenance interstitial cached under the file’s URL. Seen on 0% of scanned sites. It is rare, but when it happens a crawler is reading its crawl policy from an HTML page.

Edge answers, origin refuses. The bare URL answers 200 from cache, and the cache-busted request gets a 403 or 503 from the origin. The file looks healthy only because the edge is serving a copy stored before the origin started refusing; the refusal surfaces when that copy expires. Seen on 0.2% of scanned sites.

Homepage served from an old copy. The age header says how many seconds ago the stored copy was built. An old copy alone proves nothing. If a fresh copy differs, visitors are getting an edit stuck behind the cache; if the two are identical, the age is harmless. Seen on 13.4% of scanned sites.

Bare URLCache-bustedWhat it meansWhat to do
200404edge outlives a deleted filepurge the file, origin first
404200new file not at the edge yetpurge, or wait for expiry
200 HTML200 text or JSONcached interstitial under the file’s URLpurge; stop challenge pages being cacheable
200 from cache403 or 503origin refusing, edge hiding itfix the origin block before the copy expires
homepage, old, bytes differfresh copy differsan edit stuck behind the cachepurge the page
homepage, old, bytes equalfresh copy identicalold but correctnothing

Where does the split come from? #

Almost always from a bot-management or maintenance layer answering a request for the machine file with an HTML page that was allowed to be cached. The request that triggered it usually came from an unfamiliar address, which is exactly what a crawler is. For as long as the edge kept that page, every crawler asking for the file was told to enable JavaScript, while the origin was fine the whole time. The same failure, seen from the crawler’s side, is in a file that answered 200 and could not be read.

Why did a new cache rule not fix it? #

Because a cache rule applies when an object is next fetched and stored. Copies already in the cache keep the lifetime they were stored with. We found a robots.txt on one of our own sites served like this:

Header or settingValue
cache-control sent to clientspublic, max-age=31536000 (one year)
age of the copy served658,413 s (7.6 days)
Edge cache limit set by the zone’s rule3,600 s (one hour)

The rule was enabled and matched the path. The object was seven and a half days old under a one-hour limit, because it had been stored before the rule existed, under the origin’s one-year directive. Nothing in the dashboard showed that. On that directive the rule would have taken effect in 2027.

A purge is what makes a new cache rule real. After purging, the file came back with age: 0 and a cache miss, and the rule governed it. The one-year header to browsers still went out, though: the browser-cache override in the same rule did not rewrite the origin’s header for these responses, including when set to 300 seconds. What worked was a response-header rule rewriting cache-control to public, max-age=300 on the machine-file paths. It took effect immediately and was still holding when re-checked a day later.

Why is a year on these files worse than it sounds? #

A one-year cache on a stylesheet is normal because the stylesheet has a fingerprinted filename and you ship a new one. robots.txt and llms.txt cannot be renamed; their URLs are fixed by convention. A client that honours a one-year directive may not re-read your crawl policy for a year, so if you block a crawler by mistake, the correction cannot reach it. These are files whose whole purpose is to be re-read. Keep them on minutes.

Check every file on every site #

Having found one, we checked ten of our sites, three machine files each, treating any directive longer than one day as a finding:

SiteFileDirective found
Site A/robots.txtmax-age=31536000
Site A/llms.txtmax-age=31536000
Site B/llms.txtmax-age=31536000
Site C/llms.txtmax-age=31536000

Four files across three sites, on the same stack, none set deliberately, with no pattern we could reconstruct for which site got which. Fixing the one file we first noticed would have left three in place.

In what order should you purge? #

A WordPress site on managed hosting behind a CDN has at least two caches in series: a server-side page cache such as Varnish, then the CDN edge. Purge the edge first and the next request pulls the still-stale copy out of the server cache and stores it at the edge again; the purge looked like it worked and changed nothing. Purge the origin-side cache first, confirm a direct request returns the new bytes, then purge the CDN. The purge that returned 200 and evicted nothing is the case that established the rule.

How to run the check yourself #

  1. For each machine file, fetch the bare URL and the same URL with a random query string. Use GET, not HEAD; the two do not always report the same cache status.
  2. Compare status first, then content type, then size.
  3. Read cf-cache-status or x-cache on the bare response. A hit with a mismatch is the edge; a miss with a mismatch is something in front of the origin that keys on the query.
  4. Read age next to the cache lifetime you configured. An age larger than that lifetime proves your rule is not governing that copy.
  5. Read cache-control on robots.txt and llms.txt. Anything measured in days or longer needs fixing.
  6. Run it from a network that is not your office: a challenge page served to unfamiliar addresses is exactly the copy that gets cached.

A scan on crawlcheck.io runs this comparison and shows both answers side by side. Once the cache is out of the way, the robots.txt status guide and the sitemap guide cover what each file should return.

How do you prevent it? #

  • Give machine files a cache lifetime of minutes, or exclude them from the page cache; they are tiny and cost the origin nothing.
  • Send Cache-Control: no-store on every challenge and maintenance response. That one setting prevents the split case.
  • After adding or changing a cache rule, purge the paths it covers, origin first, then re-fetch and confirm age reset.
  • When the origin changes what it serves crawlers, a new firewall rule or a new hosting plan, purge the machine files explicitly rather than waiting for expiry.
  • When you find one bad directive, sweep every machine file on every site that shares the stack.
Share of scanned sites with each finding in this articleStale cached copy served to crawlers13.4%Stale cached robots.txt or llms.txt0.4%Origin refuses requests the cache cannot answer0.2%Cache serves different copies of a machine file0%Share of scanned sites with eachfinding in this articleStale cached copy served to crawlers13.4%Stale cached robots.txt or llms.txt0.4%Origin refuses requests the cache cannot answer0.2%Cache serves different copies of a machine file0%
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

Why fetch each machine file twice?

Once bare, as a crawler asks for it, and once with a random query string that misses the cache and reaches the origin. A single fetch cannot tell whether the edge and the origin agree; the pair can.

What is the difference between a stale and a split machine-file cache?

Stale means the two fetches returned different status codes, usually an edge outliving a deleted file. Split means the same status but one answer is HTML and the other is not, so some crawlers get a page where the file should be. Compare status first, because a 404 body is HTML by nature.

Why did my new cache rule not change what is served?

Cache rules apply to objects stored after the rule exists. A copy already in the cache keeps the lifetime it was stored with. Purge the paths the rule covers, then re-fetch and confirm the age header reset.

How long should robots.txt and llms.txt be cached?

Minutes, not days. Their URLs cannot be renamed or versioned, so a long cache directive means a correction to your crawl policy cannot reach clients that already stored the old file.

In what order should I purge caches?

Origin-side page cache first, then the CDN edge. Purging the edge first lets the next request re-store the still-stale server copy, so the purge changes nothing.

How do I stop a challenge page being cached under robots.txt?

Send Cache-Control: no-store on every challenge or maintenance response, and keep machine files on a short cache lifetime or out of the page cache.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works