Findings · 2026-08-21 · By VSNARY | Emmanuel Orta
Your new cache rule did not fix the object it was written for
A robots.txt served with a one-year browser cache, 7.6 days old, on a zone whose cache rule capped the edge TTL at one hour. The rule was fine. It just does not apply to anything already in the cache.
We found a robots.txt being served with a one-year browser cache. Not a long cache. The maximum practical one.
cache-control | public, max-age=31536000 |
age | 658,413 s (7.6 days) |
| Edge rule in place | edge TTL 3,600 s |
Look at the last two rows together. There was a cache rule on that zone explicitly capping the edge TTL at one hour, and the object being served was seven and a half days old. The rule was enabled. The rule matched the path. And the rule was doing nothing.
A cache rule does not evict what is already cached
The object had entered the cache before the rule existed, under the origin’s own one-year directive. Cache rules apply when an object is fetched and stored. They are not retroactive. Nothing in the dashboard indicates this: the rule shows as active, its expression matches, and the age of the object it is failing to govern is not displayed anywhere near it.
So a rule you added to fix a caching mistake will not fix any copy of that mistake already in the cache. On a one-year TTL, it would have taken effect in 2027.
The action that makes a new cache rule real is a purge. After purging, the object came back with age: 0 and a cache MISS, and the rule finally governed it.
The fix that did not work
The obvious next move was the browser-TTL control that sits in the same rule. It was already set to override the origin. It had been set that way the whole time, and the file still went out with a one-year directive. Changing the override to a real value of 300 seconds did not help either.
We were fixing an origin response header with a cache rule, and a cache rule’s browser TTL did not rewrite it for these responses. Two attempts, both reasonable, both verified as not working before moving on — which is the only part of this we would repeat.
What worked was a response header transform rule: rewrite cache-control to public, max-age=300 on the machine-file paths. That took effect immediately and is still holding a day later, with content type and body unchanged.
It was never one file
Having found one, we swept ten sites across three files each rather than fixing the one we happened to notice.
| Site | File | Directive found |
| Site A | /robots.txt | max-age=31536000 |
| Site A | /llms.txt | max-age=31536000 |
| Site B | /llms.txt | max-age=31536000 |
| Site C | /llms.txt | max-age=31536000 |
Four files across three sites, on the same stack, none of which anyone had set deliberately. One site had it on robots.txt and another did not, for no reason we could reconstruct. A single-file fix would have left three of them in place and produced a satisfying, false, sense of having handled it.
Why a year on these files specifically is worse than it sounds
A one-year cache on a stylesheet is normal practice, because a stylesheet has a fingerprinted filename and you ship a new one. robots.txt and llms.txt cannot be fingerprinted. Their URLs are fixed by convention. There is no cache-busting mechanism available to them at all.
So the directive means what it says: a client that honours it may not re-read your crawl policy for a year. If you block a crawler you did not mean to block, or allow one you meant to exclude, the correction cannot reach any client that already cached the file. You cannot rename the file. You cannot version it. You can only wait.
And these are the files whose entire purpose is to be re-read. A sitemap tells a crawler what changed. A robots policy tells it what it may do now. Both are statements about the present, published under a header that says they are true until next year.
What transfers
- After adding a cache rule, purge the paths it covers. Otherwise the rule governs future objects and the existing bad object keeps being served for its original TTL.
- Read
agealongside your rule. An age larger than your configured TTL is proof the rule is not governing that object, and it is the fastest available check. - Convention-named files should have short TTLs. Anything that cannot be fingerprinted cannot be corrected in a hurry. Minutes, not months.
- Sweep the estate, not the file. A default that produced this on one site produced it on others, and the pattern of which sites were affected made no sense — which is exactly why guessing does not work.
Method
All headers were read with an ordinary HTTP client from outside the networks serving the sites, using GET rather than HEAD — the two do not always report the same cache status. The sweep covered ten sites and three files each; only directives longer than one day were treated as findings. Post-fix figures were re-checked a day later.
Every figure above came out of this scanner.
Point it at your own domain and see the same measurements, free.