CrawlCheck

Agent view · Guides · 2026-09-30

Stale cached robots.txt or llms.txt? Two fetches tell you

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/machine-file-caching-stale-split-and-blocked-behind-the-edge: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/machine-file-caching-stale-split-and-blocked-behind-the-edge>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes18,892
words2,904
sha-256e0c023acca8b09f13764f9b13db93e28c6af9b837ea10ad5caa0473130e398c2
repr-digestsha-256=:4MAjrMqLCfE3ZPmxPbk+KMavm4N+oQrVyqBHMTDjmMI=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795182375;expires=1795185975;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="XGMal46KejksRLmfekjtng";tag="crawlcheck-page"
signaturesig1=:jkc9PYrR/if/Laloiz72bMbxP3XSQ4bmGxYevm6tqs2NH69LZUu6DuyAo/m1o/y/HFqsKvMJP3LbtF6dWyGeBg==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Guides · 2026-09-30 · By [VSNARY | Emmanuel Orta](/author) · 2 views

# Stale cached robots.txt or llms.txt? Two fetches tell you

Why one fetch cannot tell whether a machine file is healthy, the two-fetch check that can, the four outcomes, why a new cache rule did not fix a one-year robots.txt, and the purge order that works.

Machine files such as robots.txt, llms.txt and sitemap.xml are cached hardest by CDNs, so one fetch cannot show whether the edge and the origin agree. Fetch each file twice: at the bare URL, the way a crawler asks, and with a cache-busting query string. Different status codes mean a stale copy; HTML on one side means some crawlers get a page instead of the file; a cached 200 against an origin 403 means the cache is hiding a block. A new cache rule does not touch copies already stored, so purge, origin cache first, then CDN.

Check your own site free

Check my site

About a minute. First scan free with your email, nothing installed.

On this page

- Why is one fetch not enough?
- The four outcomes
- Where does the split come from?
- Why did a new cache rule not fix it?
- Why is a year on these files worse than it sounds?
- Check every file on every site
- In what order should you purge?
- How to run the check yourself
- How do you prevent it?

[robots.txt](/glossary/robots-txt), [llms.txt](/glossary/llms-txt), sitemap.xml and an [entity map](/glossary/entity-map) are small files that rarely change, which is exactly what a [CDN](/glossary/cdn) caches hardest. That is fine until one changes, or the origin starts refusing a crawler, because the edge keeps answering with whatever it stored and nobody fetching the bare [URL](/glossary/url) can tell. CrawlCheck found this by being fooled by it: a bare request to one of our machine files returned a cached HTML interstitial at HTTP 200 while a cache-busted request to the same URL returned valid JSON. Then we found the files on our own sites carrying a one-year cache header nobody had set on purpose. This guide is the two-fetch check, the four outcomes, and the fixes that actually took effect.
The two-fetch method. One fetch shows only the edge; the pair shows whether the edge still agrees with the origin.

## Why is one fetch not enough? #

An [edge cache](/glossary/edge-cache) is keyed on the URL. Ask for `/robots.txt` and you get the stored copy if there is one; the origin is not consulted. Ask for `/robots.txt?cc=x7k2p` and, unless the cache ignores query strings, the key misses and the request reaches the origin. Comparing the two tells you whether the edge and the origin agree about that file, in status, in content type and in bytes.

## The four outcomes #

**Status disagrees.** The bare URL answers 200 and the origin 404, or the reverse. The edge is outliving a deleted or moved file, or a new file has not reached the edge yet. Seen on 0.4% of scanned sites. Compare status before content type: a 404 body is HTML by nature, so a status difference always looks like a type difference too.

**Same status, different kind.** Both answer 200, but one is HTML and the other is not. Some crawlers are getting a page where the file should be, usually a bot-challenge or maintenance interstitial cached under the file’s URL. Seen on 0% of scanned sites. It is rare, but when it happens a crawler is reading its crawl policy from an HTML page.

**Edge answers, origin refuses.** The bare URL answers 200 from cache, and the cache-busted request gets a 403 or 503 from the origin. The file looks healthy only because the edge is serving a copy stored before the origin started refusing; the [refusal](/glossary/refusal) surfaces when that copy expires. Seen on 0.2% of scanned sites.

**Homepage served from an old copy.** The `age` header says how many seconds ago the stored copy was built. An old copy alone proves nothing. If a fresh copy differs, visitors are getting an edit stuck behind the cache; if the two are identical, the age is harmless. Seen on 13.4% of scanned sites.

| Bare URL | Cache-busted | What it means | What to do |
| --- | --- | --- | --- |
| 200 | 404 | edge outlives a deleted file | purge the file, origin first |
| 404 | 200 | new file not at the edge yet | purge, or wait for expiry |
| 200 HTML | 200 text or JSON | cached interstitial under the file’s URL | purge; stop challenge pages being cacheable |
| 200 from cache | 403 or 503 | origin refusing, edge hiding it | fix the origin block before the copy expires |
| homepage, old, bytes differ | fresh copy differs | an edit stuck behind the cache | purge the page |
| homepage, old, bytes equal | fresh copy identical | old but correct | nothing |

## Where does the split come from? #

Almost always from a bot-management or maintenance layer answering a request for the machine file with an HTML page that was allowed to be cached. The request that triggered it usually came from an unfamiliar address, which is exactly what a crawler is. For as long as the edge kept that page, every crawler asking for the file was told to enable JavaScript, while the origin was fine the whole time. The same failure, seen from the crawler’s side, is in [a file that answered 200 and could not be read](/blog/a-file-that-answered-200-and-could-not-be-read).

## Why did a new cache rule not fix it? #

Because a cache rule applies when an object is next fetched and stored. Copies already in the cache keep the lifetime they were stored with. We found a robots.txt on one of our own sites served like this:

| Header or setting | Value |
| --- | --- |
| `cache-control` sent to clients | public, max-age=31536000 (one year) |
| `age` of the copy served | 658,413 s (7.6 days) |
| Edge cache limit set by the zone’s rule | 3,600 s (one hour) |

The rule was enabled and matched the path. The object was seven and a half days old under a one-hour limit, because it had been stored before the rule existed, under the origin’s one-year directive. Nothing in the dashboard showed that. On that directive the rule would have taken effect in 2027.

**A purge is what makes a new cache rule real.** After purging, the file came back with `age: 0` and a cache miss, and the rule governed it. The one-year header to browsers still went out, though: the browser-cache override in the same rule did not rewrite the origin’s header for these responses, including when set to 300 seconds. What worked was a response-header rule rewriting `cache-control` to `public, max-age=300` on the machine-file paths. It took effect immediately and was still holding when re-checked a day later.

## Why is a year on these files worse than it sounds? #

A one-year cache on a stylesheet is normal because the stylesheet has a fingerprinted filename and you ship a new one. robots.txt and llms.txt cannot be renamed; their URLs are fixed by convention. A client that honours a one-year directive may not re-read your crawl policy for a year, so if you block a crawler by mistake, the correction cannot reach it. These are files whose whole purpose is to be re-read. Keep them on minutes.

## Check every file on every site #

Having found one, we checked ten of our sites, three machine files each, treating any directive longer than one day as a finding:

| Site | File | Directive found |
| --- | --- | --- |
| Site A | /robots.txt | max-age=31536000 |
| Site A | /llms.txt | max-age=31536000 |
| Site B | /llms.txt | max-age=31536000 |
| Site C | /llms.txt | max-age=31536000 |

Four files across three sites, on the same stack, none set deliberately, with no pattern we could reconstruct for which site got which. Fixing the one file we first noticed would have left three in place.

## In what order should you purge? #

A WordPress site on managed hosting behind a CDN has at least two caches in series: a server-side page cache such as Varnish, then the CDN edge. Purge the edge first and the next request pulls the still-stale copy out of the server cache and stores it at the edge again; the purge looked like it worked and changed nothing. Purge the origin-side cache first, confirm a direct request returns the new bytes, then purge the CDN. [The purge that returned 200 and evicted nothing](/blog/the-purge-returned-200-and-evicted-nothing) is the case that established the rule.

## How to run the check yourself #

- For each machine file, fetch the bare URL and the same URL with a random query string. Use GET, not HEAD; the two do not always report the same cache status.
- Compare status first, then content type, then size.
- Read `cf-cache-status` or `x-cache` on the bare response. A hit with a mismatch is the edge; a miss with a mismatch is something in front of the origin that keys on the query.
- Read `age` next to the cache lifetime you configured. An age larger than that lifetime proves your rule is not governing that copy.
- Read `cache-control` on robots.txt and llms.txt. Anything measured in days or longer needs fixing.
- Run it from a network that is not your office: a challenge page served to unfamiliar addresses is exactly the copy that gets cached.

A scan on [crawlcheck.io](/) runs this comparison and shows both answers side by side. Once the cache is out of the way, [the robots.txt status guide](/guides/robots-txt-status-codes-what-404-403-and-503-tell-a-crawler) and [the sitemap guide](/guides/xml-sitemap-audit) cover what each file should return.

## How do you prevent it? #

- Give machine files a cache lifetime of minutes, or exclude them from the page cache; they are tiny and cost the origin nothing.
- Send `Cache-Control: no-store` on every challenge and maintenance response. That one setting prevents the split case.
- After adding or changing a cache rule, purge the paths it covers, origin first, then re-fetch and confirm `age` reset.
- When the origin changes what it serves crawlers, a new firewall rule or a new hosting plan, purge the machine files explicitly rather than waiting for expiry.
- When you find one bad directive, sweep every machine file on every site that shares the stack.

Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fmachine-file-caching-stale-split-and-blocked-behind-the-edge%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fmachine-file-caching-stale-split-and-blocked-behind-the-edge%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Stale%20cached%20robots.txt%20or%20llms.txt%3F%20Two%20fetches%20tell%20you)[Email](mailto:?subject=Stale%20cached%20robots.txt%20or%20llms.txt%3F%20Two%20fetches%20tell%20you&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fmachine-file-caching-stale-split-and-blocked-behind-the-edge)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### Why fetch each machine file twice?

Once bare, as a crawler asks for it, and once with a random query string that misses the cache and reaches the origin. A single fetch cannot tell whether the edge and the origin agree; the pair can.

### What is the difference between a stale and a split machine-file cache?

Stale means the two fetches returned different status codes, usually an edge outliving a deleted file. Split means the same status but one answer is HTML and the other is not, so some crawlers get a page where the file should be. Compare status first, because a 404 body is HTML by nature.

### Why did my new cache rule not change what is served?

Cache rules apply to objects stored after the rule exists. A copy already in the cache keeps the lifetime it was stored with. Purge the paths the rule covers, then re-fetch and confirm the age header reset.

### How long should robots.txt and llms.txt be cached?

Minutes, not days. Their URLs cannot be renamed or versioned, so a long cache directive means a correction to your crawl policy cannot reach clients that already stored the old file.

### In what order should I purge caches?

Origin-side page cache first, then the CDN edge. Purging the edge first lets the next request re-store the still-stale server copy, so the purge changes nothing.

### How do I stop a challenge page being cached under robots.txt?

Send Cache-Control: no-store on every challenge or maintenance response, and keep machine files on a short cache lifetime or out of the page cache.

## Related findings

[2026-08-13 Your cache purge returned 200 and evicted nothing Four cache purges returned 200 and evicted nothing. The two-request test that catches a stale cache, how to read the headers, and](/blog/the-purge-returned-200-and-evicted-nothing)[2026-10-03 How to test robots.txt for GPTBot and ClaudeBot: 5 tools compared How to test robots.txt for GPTBot, ClaudeBot and PerplexityBot: five tools compared, a two-minute test, and why delivery matters m](/guides/how-to-test-robots-txt-for-gptbot-and-claudebot)[2026-09-23 robots.txt status codes: what a 404, 403, 429 or 503 tells Google and AI crawlers What a 404, 403, 429 or 503 on robots.txt means to Google and other crawlers under RFC 9309: when crawling is allowed, paused or b](/guides/robots-txt-status-codes-what-404-403-and-503-tell-a-crawler)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [Certificates](/certificates) · [Conduct: Ahrefsbot](/agents/conduct/ahrefsbot) | **Registry:** [MCP: Operator](/registry/mcp/operator) · [MCP](/registry/mcp) · [AI policy](/registry/aipolicy) · [producthunt.com](/registry/producthunt.com) · [yandex.ru](/registry/yandex.ru) · [theagenttimes.com](/registry/capabilities/theagenttimes.com) | **Datasets:** [Maine](/datasets/state/maine) · [Technologies: Wix](/datasets/technologies/wix) | **Guides:** [Indexnow what it does who reads it and how to check it worked](/guides/indexnow-what-it-does-who-reads-it-and-how-to-check-it-worked) | **Glossary:** [cosmetic row](/glossary/cosmetic-row)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /guides/machine-file-caching-stale-split-and-blocked-behind-the-edge.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.