Agent view · Findings · 2026-11-06

Our crawler counts missed about 45% of hits

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/our-crawler-counts-missed-45-percent-of-hits: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

Findings · 2026-11-06 · By [VSNARY | Emmanuel Orta](/author) · 1 view

# Our crawler counts missed about 45% of hits

Every published crawler hit count before 6 November 2026 is too low. Concurrent writes to one counter dropped nearly half of all hits. Here is the measurement, the cause and the fix.

On 6 November 2026, measured against Cloudflare's own request log, CrawlCheck's per-crawler hit counts on the sites it watches had recorded about 55% of the requests that actually arrived. The counter had worked the same way since it was built in August 2026; the loss before 6 November was not measured, and was probably smaller on lighter-traffic days. The cause was a read-modify-write on one storage key per crawler per site: two hits landing in the same second read the same value, both added one, and one write won. Counts are now atomic adds in a single coordinated object. Published conduct counts from before the fix undercount real traffic by roughly that share; the rates built from them are less affected because every crawler lost hits at a similar rate.

On this page

- What the counter did
- The measurement
- Why no check caught it
- The fix
- What this changes in published figures

On 6 November 2026 we put our own crawler telemetry next to Cloudflare's request log for the same twelve minutes and the same eight crawlers. Cloudflare saw 96 requests. We had recorded 53. Amazonbot alone sent 64 requests and we had written down 29.

That is not a sampling difference. Both systems see every request to the same hosts. One of them was losing about 45 percent of the hits, and it was ours.

## What the counter did #

Each [verified crawler](/glossary/verified-crawler) hit incremented a key named for the crawler and the site, `tel:<bot>:<host>`, plus a shared daily budget key. The increment was a read, an add and a write. Cloudflare Workers run many copies of the code at once, and a KV store has no atomic add. When two hits for the same crawler on the same site arrived close together, both copies read the same number, both added one, and both wrote it back. The second write replaced the first. One hit vanished.

Busy crawlers lose the most, because they are the ones whose hits overlap. Amazonbot, which fetches in bursts, lost 55 percent in the sample. A crawler that visits once an hour lost almost nothing. The loss is proportional to burstiness, not to volume.

## The measurement #

| Crawler | Cloudflare requests | Our count before the fix | Our count after the fix |
| --- | --- | --- | --- |
| Amazonbot | 64 | 29 | 37 of 37 in the after window |
| Eight crawlers together | 96 | 53 | 47 of 46 (one Cloudflare sample boundary) |
| Lost share | | about 45% | none measurable |

The after window is a second twelve-minute sample taken eleven minutes after the fix went live, read the same way. The 47 against 46 is one request that fell on the edge of Cloudflare's minute bucket; the point is that the two systems now agree.

A test harness that fires 50 concurrent hits plus one more makes the defect reproducible: before the fix, 5 of 51 were recorded. After it, 51 of 51.

## Why no check caught it #

Every check we had asked whether telemetry was running. It was. Rows were written every minute, the dashboards moved, the daily budget counted down. Nothing compared our total to an independent total, because until this week nothing read Cloudflare's own request counts for the same hosts. A counter that loses half its increments looks exactly like a counter on a quieter day.

The same class of defect produced [26 days of pending proofs](/blog/our-proofs-said-pending-for-26-days) last September: the process ran, every check confirmed the process ran, and nobody opened the output.

## The fix #

Hit counts, verdict counts and the daily budget are now atomic adds inside one coordinated object per site, seeded from the last stored row. The KV key is written from that object, not raced by every request. The budget key is no longer read or written per hit, which also removes about two storage operations per crawler hit. The daily cap moved from 20,000 to 50,000 because the true hit rate, around 18,000 a day across the watched sites, would have hit the old cap mid-afternoon once every hit was counted.

## What this changes in published figures #

The counter worked this way from the day it was built in August 2026. We measured the loss once, on 6 November, at that day's traffic; on quieter days fewer hits overlapped, so the share lost was probably smaller, but we have no measurement of it. Every absolute crawler hit count we published before 6 November 2026, including the per-crawler totals in [the eight-week conduct post](/blog/eight-weeks-of-agent-conduct-who-was-really-who) and the [conduct ratings page](/agents/conduct), is low: by up to roughly 45 percent for bursty crawlers on busy days, and by less for steady ones. We are not restating those numbers, because the lost hits were never stored; read every count from before that date as a lower bound.

Shares and rates are less affected. [Robots.txt](/glossary/robots-txt) obedience is judged per request that was recorded, and a dropped hit was dropped whether it obeyed or not. The verified-versus-forged split in [how much AI crawler traffic is forged](/blog/how-much-ai-crawler-traffic-is-forged) was computed from the recorded rows and holds as a share. Where a crawler's grade depended on a count crossing a threshold, the threshold was reached later than it should have been, which can only have made a grade kinder.

From 6 November the counts are complete. The first full day under the new counter is 7 November, and that is the first day we would quote as a daily total.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Four-crawler-counts-missed-45-percent-of-hits%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Four-crawler-counts-missed-45-percent-of-hits%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Our%20crawler%20counts%20missed%20about%2045%25%20of%20hits)[Email](mailto:?subject=Our%20crawler%20counts%20missed%20about%2045%25%20of%20hits&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Four-crawler-counts-missed-45-percent-of-hits)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### How much crawler traffic did CrawlCheck undercount?

About 45% of verified hits across eight crawlers in a twelve-minute comparison against Cloudflare's request log on 6 November 2026, and 55% for Amazonbot, the burstiest. Steady, low-volume crawlers lost almost nothing.

### Why did the counter lose hits?

Each hit read a stored count, added one and wrote it back. Two hits arriving close together read the same value and the second write replaced the first. The store has no atomic add, so overlapping hits were lost.

### Are the published conduct ratings wrong?

The absolute hit counts behind them are low. The rates, such as the share of requests that obeyed robots.txt, are much less affected because hits were lost regardless of what they did. A grade that needed a count threshold was reached later, never earlier.

### What changed on 6 November 2026?

Hit, verdict and budget counts became atomic adds inside one coordinated object per site. A test of 50 concurrent hits plus one recorded 5 of 51 before the fix and 51 of 51 after. A second live comparison against Cloudflare matched within one request.

### Will the old numbers be corrected?

No. The dropped hits were never stored, so there is nothing to restate. Counts quoted from before 6 November 2026 should be read as lower bounds.

## Related findings

[2026-10-03 Verify GPTBot, ClaudeBot and PerplexityBot: tools compared How to verify GPTBot, ClaudeBot and PerplexityBot requests: published IP lists, CrawlCheck's free log check and Cloudflare verifie](/guides/how-to-verify-gptbot-claudebot-and-perplexitybot-tools-compared)[2026-08-18 Which AI crawlers can't be verified by IP range? Which crawler operators publish IP ranges and which publish nothing, why an unverifiable claim is not a forgery, and what you can](/blog/the-crawlers-nobody-can-check)[2026-10-15 Agent conduct, October 2026: 34,676 verified requests Agent conduct on crawlcheck.io, 12 Aug to 15 Oct 2026: 34,676 verified requests, 9 of 12 crawlers read robots.txt, 3 never did, 84](/blog/agent-conduct-october-2026)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit, and placement is never for sale.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [Listing check](/listing-check) · [Entity map validator](/entity-map-validator) · [Status](/status) · [SDK](/sdk) | **Glossary:** [CAWG training and data mining assertion](/glossary/cawg-training-assertion) · [Sitemap URL set](/glossary/sitemap-url-set) · [Information architecture](/glossary/information-architecture) · [Tool permission](/glossary/tool-permission) · [Data sheet](/glossary/data-sheet) · [k-anonymity](/glossary/k-anonymity) · [Retry-After](/glossary/retry-after) | **Directory:** [Zilliz](/directory/zilliz)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Report an error](/report-error)

Same body as a file: /blog/our-crawler-counts-missed-45-percent-of-hits.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.