Findings · 2026-08-15 · By VSNARY | Emmanuel Orta
Site audits go stale: why one clean scan proves little
Measuring the same sites every day produces a number a one-off audit cannot: how many were broken at least once. The gap between “broken now” and “ever broken” is the case for a series over a snapshot.
A one-off audit reports what was true when it ran, so it misses intermittent faults: a cache serving stale robots rules for an hour a day, a deploy that briefly drops a header, a rate limiter that refuses crawlers only under load. Scanning the same sites daily gives two figures: how many are broken on their latest reading and how many were broken on at least one day. The gap is what a single audit cannot see. Stale caches and edge challenges come and go; a missing sitemap stays until fixed.
Every audit answers one question: what was true at the moment it looked. The defects that make that answer misleading are the intermittent ones. Check on Tuesday and the site is clean. It was broken on Sunday and will be broken on Thursday, and no Tuesday audit at any budget can see either.
How do you measure what an audit misses?
CrawlCheck scans a fixed panel of sites every day and keeps each day's reading. That makes two percentages computable over the same sites and the same window: how many carry a defect on their latest reading, and how many carried one on at least one day. A single crawl, anyone's including ours, can only produce the first. The second needs the sites to have been measured before the question was asked.
What a single crawl cannot see #
87.5% of measured sites carry a defect today. 100% carried one at least once between 2026-08-14 and 2026-11-20. 6 look clean now and did not for at least one day in that window.
| Across 48 sites measured every day | Share |
|---|---|
| Carry a defect today | 87.5% |
| Carried one at least once in the window | 100% |
| The gap | 12.5 points |
| Look clean now and did not, at least once | 6 |
Both numbers describe the same sites over 99 days, 2026-08-14 to 2026-11-20. A one-off crawl at any budget can only ever produce the first row — the second needs the same sites to have been measured before the question was asked. Sites added after the window opened are excluded (2161) because they have had fewer chances to be seen broken. No scanned domain is named here or anywhere else on this site.
Citing this figure. The numbers above are recomputed from the live record when this page loads, so link the anchor rather than freezing a copy: https://crawlcheck.io/blog/your-site-was-clean-the-day-you-audited-it#the-gap. Sentence form, as of 2026-11-20: Among the 48 sites CrawlCheck measured daily between 2026-08-14 and 2026-11-20, 87.5% carried a machine-layer defect on the latest reading, while 100% carried one at least once — a gap of 12.5 points invisible to any single-day audit.
Which defects come and go, and which just sit there?
The headline gap says how many sites have a hidden bad day. The more useful question is which defects hide. A large gap between “ever” and “now” marks an intermittent fault; no gap marks a defect that has simply been there the whole time.
| Defect | How it behaves | Share of scanned sites (live) | How to catch it yourself |
|---|---|---|---|
| Stale cached copy of a page or machine file | Comes and goes with cache lifetimes and purges | 13.4% | Fetch the bare URL and a cache-busted URL; compare bytes |
| Machine file (robots.txt, llms.txt) stale at the edge | Comes and goes; often appears right after an edit | 0.4% | Fetch the file after every change, from outside your network |
| Challenge page served with status 200 | Comes and goes with bot-management rule updates and load | 0.3% | Fetch as a crawler user-agent and read the body, not just the status |
| A named answer engine refused | Often intermittent: rate limits and datacentre filters under load | 5.8% | Repeat the crawler fetch at different times of day |
| Missing sitemap | Stays until someone adds one: “ever” equals “now” | One check is enough |
The intermittent rows are also the ones a site owner is most likely to be told do not exist, because the auditor looked on a good day. For the cache rows, how to check what crawlers really get from a cached robots.txt or llms.txt walks through the test; for refusals, how to tell a refusal from a hiccup covers a single failed fetch.
Which exclusions move the number?
A site added after the window opened has had fewer chances to be seen broken, so late joiners are excluded from both percentages rather than allowed to drag the “ever” figure down. A site whose latest reading has no usable findings count is left out of both denominators and counted openly. With too few days of data, “at least once” and “right now” are the same set by construction, so the table is withheld until the window is long enough, and the reason travels with it.
A cached number is a Tuesday audit too
The same failure applies to statistics. An answer engine once summarised this site and quoted our public scan counter accurately for the day it read the page, which put it about 140 scans behind the live figure by the time the summary was shown. Nothing lied; a snapshot aged. That is why the figures on this page are recomputed from the live record when it loads, and why the citable unit is the URL, not the quote. A number that agrees with itself is not the same as a number that is still true.
What is this sample?
The panel is seeded and submitted: sites we chose to watch and sites people asked us to scan, not a random sample of the web. Quote it as “among the sites CrawlCheck measures daily”, nothing broader. No scanned domain is named here.
One limit bounds every figure. A defect is counted from what CrawlCheck could detect on that date. When a check is added, sites that carried that defect for months first appear to acquire it the day the check arrived. The figures are a floor on how much was broken, never a ceiling, and a trend in them is a trend in what we can see, not in the web's health.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
Why is a one-off audit not enough?
It can only report the state on the day it ran. It cannot say how many sites were broken at least once, which is the number that describes reliability.
What is the gap between broken now and broken ever?
Scanning the same sites daily produces both figures. The difference between them is the set of sites with a defect that a single crawl on a good day would miss.
Which crawler defects are most often intermittent?
Stale cached copies of pages and machine files, challenge pages served at the edge, and refusals that only happen under load or from datacentre addresses. A missing sitemap, by contrast, stays until it is fixed.
How can I catch an intermittent defect myself?
Repeat the same crawler-style fetch at different times of day and after every deploy or cache change, and compare status, size and body each time. One clean result is one data point.
Which exclusions matter when reading these figures?
Sites that joined part-way through the window are excluded, because counting them would shrink the ever-broken share for reasons that have nothing to do with the sites.
Can the figures be read as a trend in website health?
No. They are a floor on what CrawlCheck could detect, and adding a new check makes old defects appear new. They describe the measured panel, not the web.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.