Which agents really reached the sites that report, and how many were impersonators.
Every agent request a reporting site receives is checked against the address ranges the agent’s operator publishes, at that site’s own origin. Verified requests count for the agent; requests using its name from anywhere else are forged claims. No reporting site is named.
Definition
- Verified
- The request came from the operator's published network ranges, or carried a valid Web Bot Auth signature.
- Forged claim
- The user agent named a known crawler but the request did not come from that operator.
- Unverifiable
- The operator publishes no way to check its traffic.
- Reporting site
- A site whose edge or reporter sends its crawler log to the index. An agent seen on fewer than 5 sites is not published.
What this does not prove
- A handful of reporting sites is a panel, not the web: shares describe these sites.
- Forged shares depend on who targets these sites; they are not a global spoofing rate.
- A month-to-date edition changes until the month closes; cite the closed edition.
- Fetches say a crawler came, not what it did with the page.
Check it yourself
The numbers on this page come from the signed JSON at /api/v1/traffic-index, rebuilt every few minutes; each copy carries its own sha256 and an Ed25519 signature by key 1hHmE3fo1rg5…. For a copy that never changes, cite a weekly edition. Verify either with no call to us:
node crawlcheck-verify.mjs --online traffic-index.json
The verifier: /crawlcheck-verify.mjs, zero dependencies.
Editions
One sealed edition per ISO week: the signed reading as first taken that week, never edited. Feed: Atom · all instruments: /api/v1/instruments.
Final edition: the month is over and these counts no longer change. Signed 2026-10-09 09:20 UTC. Other months: 2026-10 · 2026-09 · 2026-08
Only 3 sites reported in 2026-09: below the 5-site floor, so totals are shown and no per-agent rate is published.
Seen, below the floor
Named so you know they were seen; their counts stay private until 5 sites report them.
ClaudeBot (below the floor), Googlebot (below the floor), GPTBot (below the floor), Bingbot (below the floor), AhrefsBot (below the floor), Applebot (below the floor), Amazonbot (below the floor), ChatGPT-User (below the floor), meta-externalagent (below the floor), SemrushBot (below the floor), OAI-SearchBot (below the floor), Claude-User (below the floor), CCBot (below the floor), Perplexity-User (below the floor), Bytespider (below the floor), PerplexityBot (below the floor), Google-Extended (below the floor), YouBot (below the floor), cohere-ai (below the floor), DuckAssistBot (below the floor)
Use the numbers
The signed edition is /api/v1/traffic-index/2026-09 (kind crawlcheck-traffic-index). Save it and run node crawlcheck-verify.mjs --online edition.json with the public verifier: the signature and the published key are checked on your machine. Licence: CC BY 4.0.
Cite as: CrawlCheck (2026). Agent Traffic Index, 2026-09. https://crawlcheck.io/traffic-index?month=2026-09
Add your site to the index
A site on Cloudflare reports with three lines in its Worker: the snippet in /spec/visitor-resolve checks each agent request with Visitor Resolve and sends one arrival per hit after the response is out. The site is proven by the CF-Worker header Cloudflare sets, so no key is needed, and the site is never named here. Each new site lifts more agents over the 5-site floor. Per-agent behaviour on one property is in Agent conduct.