Agent view · Findings · 2026-08-20
Why caching only successes kept our scans slow
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/cache-your-misses-a-lookup-that-only-caches-on-success: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/cache-your-misses-a-lookup-that-only-caches-on-success>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 11,232 |
| words | 1,556 |
| sha-256 | 3723469e9b9d5ccc73e31b68cdec2a9f0950f9fc4aabd428b0741ce7aca2de70 |
| repr-digest | sha-256=:NyNGnpudXMxz4xtozewqnwlQ+fxKq9QosHQc56yi3nA=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311496;expires=1795315096;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="gRcz2mgDE4qsTKVvv4MO_Q";tag="crawlcheck-page" |
| signature | sig1=:M8lgFBRDfGqrviKyo7GIbfobBUggHwiINpsv30RMypDpbzWS/N1X5s9PKuyzW5B3nJXSDabIgdjSdtJuYGbiBw==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Findings · 2026-08-20 · By [VSNARY | Emmanuel Orta](/author) # Why caching only successes kept our scans slow Scans of small business sites ran slow and varied from run to run, because we kept asking a question whose answer was always no. Our first fix made one case worse. Scans of small business sites were slower than they needed to be, and the same site could take nearly three times longer on one run than another. The cause: a lookup for public real-world performance data, which small sites almost never have, remembered its answer only when data came back. A "no data" answer was asked again on every scan, costing up to 8.2 seconds. Remembering the "no" cut it to 3 ms. The rule: cache facts about what you measured, including "nothing there", never your own failed attempts. On this page - What was wrong? - How did our first fix make it worse? - What carries over to your own tools? The repeated “no data” lookup, before and after Before up to 8.2 s on every scan Once the “no” was remembered 3 ms Three scans of the same site, with no change in between, took 12.9 seconds, 20.6 seconds, and nearly three times the fastest. An average would have sent us hunting for something twenty seconds long. There was nothing twenty seconds long: there was a fast path and a slow path, and averaging them hid both. ## What was wrong? # Part of a scan asks whether public real-world performance data exists for the site. For most local businesses it does not, and it will not: the Chrome UX Report only publishes a site once it has enough Chrome visits, and a regional business rarely gets them. We remembered the answer only when data came back. A "no" was never remembered, so every scan of a small site paid again for the same empty answer. | Measure | Value | | --- | --- | | Fastest of three runs, same site | 12.9 s | | Middle run | 20.6 s | | Slowest run | nearly three times the fastest | | Repeated "no data" lookup, before | up to 8.2 s on every scan | | Same lookup, once the "no" was remembered | 3 ms | | Side effect of our first fix | an intermittent failure frozen for six hours | ## How did our first fix make it worse? # The first fix remembered every outcome, including a temporary "still waiting" state. That is not an answer about the site at all; it is our own request not having finished. Remembering it stopped the retry from running, and an intermittent failure became a permanent one for six hours until we corrected it. The scan results themselves never changed: the measurement was the same, it just stopped repeating an empty lookup. ## What carries over to your own tools? # - **Time each step, not the total.** A total cannot tell one slow step from one occasionally slow step, and they need opposite fixes. The same trap applies to [single Lighthouse runs](/blog/one-lighthouse-run-cannot-support-a-finding-225-percent-variance). - **Remember "the answer is no".** It is a fact about the thing you measured and stays true as long as a yes would. - **Never remember "we could not ask".** A rejected key or a request still pending is a fact about you. Retry it. If your own site is one of the many with no Chrome UX Report data, [here is how to measure Core Web Vitals without it](/guides/core-web-vitals-field-data-for-low-traffic-sites-without-crux). CrawlCheck’s dataset when this page loaded: 582,334 sites measured. Written by [Emmanuel Orta](/author) · VSNARY I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post. [Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta) Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fcache-your-misses-a-lookup-that-only-caches-on-success%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fcache-your-misses-a-lookup-that-only-caches-on-success%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Why%20caching%20only%20successes%20kept%20our%20scans%20slow)[Email](mailto:?subject=Why%20caching%20only%20successes%20kept%20our%20scans%20slow&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fcache-your-misses-a-lookup-that-only-caches-on-success)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta) ## Questions this post answers ### What is a negative cache? A stored record that a lookup found nothing, with its own expiry, so the same empty request is not repeated on every run. A cache that only stores successes repeats the expensive miss forever. ### Why do small business sites have no Chrome UX Report data? The Chrome UX Report only publishes a site once it has enough real Chrome visits, and a regional business rarely reaches that level. ### Did this change any scan result? No. The measurement was the same before and after. Scans of small sites simply stopped repeating a lookup whose answer was already known. ### What should never be cached? Anything that describes your own attempt rather than the thing you measured, such as a rejected credential or a request that has not finished. Those must be retried. ## Related findings [2026-11-06 Our AI accuracy check ran for six days and measured nothing Our daily AI-accuracy check made no successful Gemini call for six days after Google changed key authorisation, and logged clean r](/blog/our-ai-accuracy-check-ran-for-six-days-and-measured-nothing)[2026-11-13 Booking links counted as missing sitemap pages CrawlCheck's index alert counted Google booking-link addresses as unfetched sitemap pages on five sites. What caused it and the fi](/blog/booking-links-counted-as-missing-sitemap-pages)[2026-10-30 AI answer corrections: three clocks, not one update time Two archived notes bracket a corrected AI answer about CrawlCheck. They do not time the fix. What went wrong in our note, and the](/blog/ai-business-corrections-three-clocks-not-one) The main product Found this on your own site? **We fix it for $100.** Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after. [Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v) [What it covers](/pricing#fix) · [scan first, it is free](/) Free browser extension Check this on any site you open, **without leaving the page.** CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account. [Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post) [What it does](/extension) Write for the library Measured something yourself? **Submit an article.** Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published. [Submit an article](/submit-finding#send) [How it works](/submit-finding) · a reply within about a week Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections). ## Comments Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*. Send for review Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan. [All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy) We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io) #### How we measure, and our own machine files CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io). **Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing. **Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours. **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [/](/) · [Listing check](/listing-check) · [Entity graph](/tools/entity-graph) · [Conduct: Adsbot google](/agents/conduct/adsbot-google) | **Datasets:** [Missouri](/datasets/state/missouri) · [Technologies: Cookiebot](/datasets/technologies/cookiebot) · [Problems: Robots sitemap relative](/datasets/problems/robots-sitemap-relative) | **Guides:** [Robots txt allow list cancels your disallow rules](/guides/robots-txt-allow-list-cancels-your-disallow-rules) | **Glossary:** [anchoring](/glossary/anchoring) · [Authority weighting](/glossary/authority-weighting) [Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /blog/cache-your-misses-a-lookup-that-only-caches-on-success.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.