Findings · 2026-11-06 · By VSNARY | Emmanuel Orta · 1 view
Our crawler counts missed about 45% of hits
Every published crawler hit count before 6 November 2026 is too low. Concurrent writes to one counter dropped nearly half of all hits. Here is the measurement, the cause and the fix.
On 6 November 2026, measured against Cloudflare's own request log, CrawlCheck's per-crawler hit counts on the sites it watches had recorded about 55% of the requests that actually arrived. The counter had worked the same way since it was built in August 2026; the loss before 6 November was not measured, and was probably smaller on lighter-traffic days. The cause was a read-modify-write on one storage key per crawler per site: two hits landing in the same second read the same value, both added one, and one write won. Counts are now atomic adds in a single coordinated object. Published conduct counts from before the fix undercount real traffic by roughly that share; the rates built from them are less affected because every crawler lost hits at a similar rate.
On 6 November 2026 we put our own crawler telemetry next to Cloudflare's request log for the same twelve minutes and the same eight crawlers. Cloudflare saw 96 requests. We had recorded 53. Amazonbot alone sent 64 requests and we had written down 29.
That is not a sampling difference. Both systems see every request to the same hosts. One of them was losing about 45 percent of the hits, and it was ours.
What the counter did #
Each verified crawler hit incremented a key named for the crawler and the site, tel:<bot>:<host>, plus a shared daily budget key. The increment was a read, an add and a write. Cloudflare Workers run many copies of the code at once, and a KV store has no atomic add. When two hits for the same crawler on the same site arrived close together, both copies read the same number, both added one, and both wrote it back. The second write replaced the first. One hit vanished.
Busy crawlers lose the most, because they are the ones whose hits overlap. Amazonbot, which fetches in bursts, lost 55 percent in the sample. A crawler that visits once an hour lost almost nothing. The loss is proportional to burstiness, not to volume.
The measurement #
| Crawler | Cloudflare requests | Our count before the fix | Our count after the fix |
|---|---|---|---|
| Amazonbot | 64 | 29 | 37 of 37 in the after window |
| Eight crawlers together | 96 | 53 | 47 of 46 (one Cloudflare sample boundary) |
| Lost share | about 45% | none measurable |
The after window is a second twelve-minute sample taken eleven minutes after the fix went live, read the same way. The 47 against 46 is one request that fell on the edge of Cloudflare's minute bucket; the point is that the two systems now agree.
A test harness that fires 50 concurrent hits plus one more makes the defect reproducible: before the fix, 5 of 51 were recorded. After it, 51 of 51.
Why no check caught it #
Every check we had asked whether telemetry was running. It was. Rows were written every minute, the dashboards moved, the daily budget counted down. Nothing compared our total to an independent total, because until this week nothing read Cloudflare's own request counts for the same hosts. A counter that loses half its increments looks exactly like a counter on a quieter day.
The same class of defect produced 26 days of pending proofs last September: the process ran, every check confirmed the process ran, and nobody opened the output.
The fix #
Hit counts, verdict counts and the daily budget are now atomic adds inside one coordinated object per site, seeded from the last stored row. The KV key is written from that object, not raced by every request. The budget key is no longer read or written per hit, which also removes about two storage operations per crawler hit. The daily cap moved from 20,000 to 50,000 because the true hit rate, around 18,000 a day across the watched sites, would have hit the old cap mid-afternoon once every hit was counted.
What this changes in published figures #
The counter worked this way from the day it was built in August 2026. We measured the loss once, on 6 November, at that day's traffic; on quieter days fewer hits overlapped, so the share lost was probably smaller, but we have no measurement of it. Every absolute crawler hit count we published before 6 November 2026, including the per-crawler totals in the eight-week conduct post and the conduct ratings page, is low: by up to roughly 45 percent for bursty crawlers on busy days, and by less for steady ones. We are not restating those numbers, because the lost hits were never stored; read every count from before that date as a lower bound.
Shares and rates are less affected. Robots.txt obedience is judged per request that was recorded, and a dropped hit was dropped whether it obeyed or not. The verified-versus-forged split in how much AI crawler traffic is forged was computed from the recorded rows and holds as a share. Where a crawler's grade depended on a count crossing a threshold, the threshold was reached later than it should have been, which can only have made a grade kinder.
From 6 November the counts are complete. The first full day under the new counter is 7 November, and that is the first day we would quote as a daily total.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
How much crawler traffic did CrawlCheck undercount?
About 45% of verified hits across eight crawlers in a twelve-minute comparison against Cloudflare's request log on 6 November 2026, and 55% for Amazonbot, the burstiest. Steady, low-volume crawlers lost almost nothing.
Why did the counter lose hits?
Each hit read a stored count, added one and wrote it back. Two hits arriving close together read the same value and the second write replaced the first. The store has no atomic add, so overlapping hits were lost.
Are the published conduct ratings wrong?
The absolute hit counts behind them are low. The rates, such as the share of requests that obeyed robots.txt, are much less affected because hits were lost regardless of what they did. A grade that needed a count threshold was reached later, never earlier.
What changed on 6 November 2026?
Hit, verdict and budget counts became atomic adds inside one coordinated object per site. A test of 50 concurrent hits plus one recorded 5 of 51 before the fix and 51 of 51 after. A second live comparison against Cloudflare matched within one request.
Will the old numbers be corrected?
No. The dropped hits were never stored, so there is nothing to restate. Counts quoted from before 6 November 2026 should be read as lower bounds.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit, and placement is never for sale.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.