Agent view · Findings · 2026-08-15
Site audits go stale: why one clean scan proves little
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/your-site-was-clean-the-day-you-audited-it: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/your-site-was-clean-the-day-you-audited-it>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 15,898 |
| words | 2,336 |
| sha-256 | cbaeafb3cee6b6ba95943d46c98825d83dbc76cd07f7d414f3ffb2c979b45a90 |
| repr-digest | sha-256=:y66vs87mtrqVlD1GyYgl2D28ds0H99QU8/+yyXm0WpA=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795182071;expires=1795185671;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="YA-fFGR7iOPCtDHHkjftDQ";tag="crawlcheck-page" |
| signature | sig1=:hvPkl9Lf1mIlZsfBMQ/xG6adk/7/cYdtMlcFB0YzEBzLQxd1oW+8FuZTuHGOA4rXQtQcylJfpy83c/lRViL2DA==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Findings · 2026-08-15 · By [VSNARY | Emmanuel Orta](/author) # Site audits go stale: why one clean scan proves little Measuring the same sites every day produces a number a one-off audit cannot: how many were broken at least once. The gap between “broken now” and “ever broken” is the case for a series over a snapshot. A one-off audit reports what was true when it ran, so it misses intermittent faults: a cache serving stale robots rules for an hour a day, a deploy that briefly drops a header, a rate limiter that refuses crawlers only under load. Scanning the same sites daily gives two figures: how many are broken on their latest reading and how many were broken on at least one day. The gap is what a single audit cannot see. Stale caches and edge challenges come and go; a missing sitemap stays until fixed. On this page - How do you measure what an audit misses? - What a single crawl cannot see - Which defects come and go, and which just sit there? - Which exclusions move the number? - A cached number is a Tuesday audit too - What is this sample? Every audit answers one question: what was true at the moment it looked. The defects that make that answer misleading are the intermittent ones. Check on Tuesday and the site is clean. It was broken on Sunday and will be broken on Thursday, and no Tuesday audit at any budget can see either. ## How do you measure what an audit misses? CrawlCheck scans a fixed panel of sites every day and keeps each day's reading. That makes two percentages computable over the *same* sites and the *same* window: how many carry a defect on their latest reading, and how many carried one on at least one day. A single crawl, anyone's including ours, can only produce the first. The second needs the sites to have been measured before the question was asked. ## What a single crawl cannot see # 87.5% of measured sites carry a defect today. 100% carried one at least once between 2026-08-14 and 2026-11-20. 6 look clean now and did not for at least one day in that window. | Across 48 sites measured every day | Share | | --- | --- | | Carry a defect today | 87.5% | | Carried one at least once in the window | 100% | | The gap | 12.5 points | | Look clean now and did not, at least once | 6 | Both numbers describe the same sites over 99 days, 2026-08-14 to 2026-11-20. **A one-off crawl at any budget can only ever produce the first row** — the second needs the same sites to have been measured before the question was asked. Sites added after the window opened are excluded (2161) because they have had fewer chances to be seen broken. No scanned domain is named here or anywhere else on this site. **Citing this figure.** The numbers above are recomputed from the live record when this page loads, so link the anchor rather than freezing a copy: `https://crawlcheck.io/blog/your-site-was-clean-the-day-you-audited-it#the-gap`. Sentence form, as of 2026-11-20: *Among the 48 sites CrawlCheck measured daily between 2026-08-14 and 2026-11-20, 87.5% carried a machine-layer defect on the latest reading, while 100% carried one at least once — a gap of 12.5 points invisible to any single-day audit.* Illustration of the gap, not panel data. The live figures for the panel are in the table above. ## Which defects come and go, and which just sit there? The headline gap says how many sites have a hidden bad day. The more useful question is which defects hide. A large gap between “ever” and “now” marks an intermittent fault; no gap marks a defect that has simply been there the whole time. | Defect | How it behaves | Share of scanned sites (live) | How to catch it yourself | | --- | --- | --- | --- | | Stale cached copy of a page or machine file | Comes and goes with cache lifetimes and purges | 13.4% | Fetch the bare [URL](/glossary/url) and a cache-busted URL; compare bytes | | Machine file ([robots.txt](/glossary/robots-txt), [llms.txt](/glossary/llms-txt)) stale at the edge | Comes and goes; often appears right after an edit | 0.4% | Fetch the file after every change, from outside your network | | Challenge page served with status 200 | Comes and goes with bot-management rule updates and load | 0.3% | Fetch as a crawler user-agent and read the body, not just the status | | A named [answer engine](/glossary/answer-engine) refused | Often intermittent: rate limits and datacentre filters under load | 5.8% | Repeat the crawler fetch at different times of day | | Missing sitemap | Stays until someone adds one: “ever” equals “now” | | One check is enough | The intermittent rows are also the ones a site owner is most likely to be told do not exist, because the auditor looked on a good day. For the cache rows, [how to check what crawlers really get from a cached robots.txt or llms.txt](/guides/machine-file-caching-stale-split-and-blocked-behind-the-edge) walks through the test; for refusals, [how to tell a refusal from a hiccup](/blog/one-503-is-not-a-blocked-site-how-to-tell-a-refusal-from-a-hiccup) covers a single failed fetch. ## Which exclusions move the number? A site added after the window opened has had fewer chances to be seen broken, so late joiners are excluded from both percentages rather than allowed to drag the “ever” figure down. A site whose latest reading has no usable findings count is left out of both denominators and counted openly. With too few days of data, “at least once” and “right now” are the same set by construction, so the table is withheld until the window is long enough, and the reason travels with it. ## A cached number is a Tuesday audit too The same failure applies to statistics. An answer engine once summarised this site and quoted our public scan counter accurately for the day it read the page, which put it about 140 scans behind the live figure by the time the summary was shown. Nothing lied; a snapshot aged. That is why the figures on this page are recomputed from the live record when it loads, and why the citable unit is the URL, not the quote. [A number that agrees with itself](/blog/our-audit-labelled-one-page-and-measured-another) is not the same as a number that is still true. ## What is this sample? The panel is seeded and submitted: sites we chose to watch and sites people asked us to scan, not a random sample of the web. Quote it as “among the sites CrawlCheck measures daily”, nothing broader. No scanned domain is named here. One limit bounds every figure. A defect is counted from what CrawlCheck could detect on that date. When a check is added, sites that carried that defect for months first appear to acquire it the day the check arrived. The figures are a floor on how much was broken, never a ceiling, and a trend in them is a trend in what we can see, not in the web's health. Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%. Written by [Emmanuel Orta](/author) · VSNARY I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post. [Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta) Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fyour-site-was-clean-the-day-you-audited-it%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fyour-site-was-clean-the-day-you-audited-it%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Site%20audits%20go%20stale%3A%20why%20one%20clean%20scan%20proves%20little)[Email](mailto:?subject=Site%20audits%20go%20stale%3A%20why%20one%20clean%20scan%20proves%20little&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fyour-site-was-clean-the-day-you-audited-it)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta) ## Questions this post answers ### Why is a one-off audit not enough? It can only report the state on the day it ran. It cannot say how many sites were broken at least once, which is the number that describes reliability. ### What is the gap between broken now and broken ever? Scanning the same sites daily produces both figures. The difference between them is the set of sites with a defect that a single crawl on a good day would miss. ### Which crawler defects are most often intermittent? Stale cached copies of pages and machine files, challenge pages served at the edge, and refusals that only happen under load or from datacentre addresses. A missing sitemap, by contrast, stays until it is fixed. ### How can I catch an intermittent defect myself? Repeat the same crawler-style fetch at different times of day and after every deploy or cache change, and compare status, size and body each time. One clean result is one data point. ### Which exclusions matter when reading these figures? Sites that joined part-way through the window are excluded, because counting them would shrink the ever-broken share for reasons that have nothing to do with the sites. ### Can the figures be read as a trend in website health? No. They are a floor on what CrawlCheck could detect, and adding a new check makes old defects appear new. They describe the measured panel, not the web. ## Related findings [2026-10-15 Robots.txt allows it, the server refuses: 57 top sites At 57 of 418 top sites, the server refused GPTBot, ClaudeBot, OAI-SearchBot or PerplexityBot even though robots.txt allows it. How](/blog/robots-txt-allows-server-refuses-ai-crawlers-october-2026)[2026-10-18 Crawl budget: measure what a crawler reads in 50 fetches Replay a crawl over your own links to see which pages a crawler reaches in 25, 50 or 100 fetches, which come last, and which fetch](/guides/crawl-budget-how-much-of-your-site-a-crawler-reads-in-50-fetches)[2026-10-06 State of AI visibility, October 2026: 139,788 homepages 139,788 homepages measured 3–6 October 2026: 6.3% block an AI crawler, 15.4% publish llms.txt, the median homepage is 3.6% text. L](/blog/state-of-ai-visibility-october-2026-139788-homepages) Where this is used Guides that cite this finding: - [GEO vs AEO vs LLM SEO: the terms, and what is measurable](/guides/geo-aeo-llm-seo-what-the-terms-mean) - [How to check if AI crawlers can read your site](/guides/how-to-check-if-ai-crawlers-can-read-your-site) - [How to measure AI visibility yourself: three checks](/guides/how-to-measure-ai-visibility) - [How to rank in ChatGPT: the part that is in your control](/guides/how-to-rank-in-chatgpt) The main product Found this on your own site? **We fix it for $100.** Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after. [Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v) [What it covers](/pricing#fix) · [scan first, it is free](/) Free browser extension Check this on any site you open, **without leaving the page.** CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account. [Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post) [What it does](/extension) Write for the library Measured something yourself? **Submit an article.** Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published. [Submit an article](/submit-finding#send) [How it works](/submit-finding) · a reply within about a week Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections). ## Comments Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*. Send for review Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan. [All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy) We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io) #### How we measure, and our own machine files CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io). **Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing. **Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours. **Pages:** [SDK](/sdk) · [Certificates](/certificates) · [SEO nap checker](/seo-nap-checker) · [Conduct: Baiduspider](/agents/conduct/baiduspider) | **Registry:** [MCP: Operator](/registry/mcp/operator) · [MCP](/registry/mcp) · [trakkr.ai](/registry/capabilities/trakkr.ai) | **Datasets:** [Technologies: Mailchimp](/datasets/technologies/mailchimp) · [Problems: ARD manifest warnings](/datasets/problems/ard-manifest-warnings) | **Guides:** [How to tell if a website fix moved traffic fixed vs untouched pages](/guides/how-to-tell-if-a-website-fix-moved-traffic-fixed-vs-untouched-pages) | **Glossary:** [population baseline](/glossary/population-baseline) · [Top-k](/glossary/top-k) [Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /blog/your-site-was-clean-the-day-you-audited-it.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.