Which agents really reached the sites that report, and how many were impersonators.
Every agent request a reporting site receives is checked against the address ranges the agent’s operator publishes, at that site’s own origin. Verified requests count for the agent; requests using its name from anywhere else are forged claims. No reporting site is named.
Definition
- Verified
- The request came from the operator's published network ranges, or carried a valid Web Bot Auth signature.
- Forged claim
- The user agent named a known crawler but the request did not come from that operator.
- Unverifiable
- The operator publishes no way to check its traffic.
- Reporting site
- A site whose edge or reporter sends its crawler log to the index. An agent seen on fewer than 5 sites is not published.
What this does not prove
- A handful of reporting sites is a panel, not the web: shares describe these sites.
- Forged shares depend on who targets these sites; they are not a global spoofing rate.
- A month-to-date edition changes until the month closes; cite the closed edition.
- Fetches say a crawler came, not what it did with the page.
Check it yourself
The numbers on this page come from the signed JSON at /api/v1/traffic-index, rebuilt every few minutes; each copy carries its own sha256 and an Ed25519 signature by key 1hHmE3fo1rg5…. For a copy that never changes, cite a weekly edition. Verify either with no call to us:
node crawlcheck-verify.mjs --online traffic-index.json
The verifier: /crawlcheck-verify.mjs, zero dependencies.
Editions
One sealed edition per ISO week: the signed reading as first taken that week, never edited. Feed: Atom · all instruments: /api/v1/instruments.
Month to date, rebuilt hourly (this reading: 2026-10-16 09:01 UTC). It becomes final on 2026-11-01, after the last day of 2026-10 is in. Other months: 2026-10 · 2026-09 · 2026-08
6
reporting sites (5 edge, 1 licensed)
10,967
fetches by the 12 published agents
61.7%
verified, of requests that could be checked
3,624
forged claims, counted against the forger
Agent by agent
An agent gets a row with counts only when 5 or more reporting sites saw it this month. Verified share = verified ÷ (verified + forged).
| Agent | Sites | Fetches | Verified | Forged | Unverifiable | Verified share | Per site | Machine files |
|---|---|---|---|---|---|---|---|---|
| Googlebot Google · indexing | 6 of 6 | 3,549 | 1,397 | 2,152 | 0 | 39.4% | 591.5 | 1.6% |
| OAI-SearchBot OpenAI · indexing | 6 of 6 | 1,929 | 1,771 | 158 | 0 | 91.8% | 321.5 | 0.7% |
| meta-externalagent Meta · training | 6 of 6 | 1,138 | 0 | 0 | 1,138 | — | 189.7 | — |
| Bingbot Microsoft · indexing | 6 of 6 | 1,106 | 1,062 | 44 | 0 | 96% | 184.3 | 1.4% |
| ClaudeBot Anthropic · training | 6 of 6 | 962 | 503 | 459 | 0 | 52.3% | 160.3 | 11.2% |
| Applebot Apple · indexing | 6 of 6 | 655 | 512 | 143 | 0 | 78.2% | 109.2 | 1% |
| PerplexityBot Perplexity · indexing | 6 of 6 | 566 | 262 | 304 | 0 | 46.3% | 94.3 | 0% |
| ChatGPT-User OpenAI · live fetch | 6 of 6 | 332 | 196 | 136 | 0 | 59% | 55.3 | 0% |
| GPTBot OpenAI · training | 6 of 6 | 247 | 27 | 220 | 0 | 10.9% | 41.2 | 25.9% |
| AhrefsBot | 6 of 6 | 241 | 105 | 0 | 136 | 100% | 40.2 | 22.9% |
| Claude-User Anthropic · live fetch | 6 of 6 | 16 | 8 | 8 | 0 | 50% | 2.7 | 0% |
| Amazonbot Amazon · indexing | 5 of 6 | 226 | 0 | 0 | 226 | — | 45.2 | — |
Seen, below the floor
Named so you know they were seen; their counts stay private until 5 sites report them.
Google-Extended (4 sites), Bytespider (4 sites), SemrushBot (4 sites), YouBot (3 sites), cohere-ai (3 sites), DuckAssistBot (3 sites), CCBot (3 sites), Perplexity-User (1 site)
Use the numbers
The signed edition is /api/v1/traffic-index/2026-10 (kind crawlcheck-traffic-index). Save it and run node crawlcheck-verify.mjs --online edition.json with the public verifier: the signature and the published key are checked on your machine. Licence: CC BY 4.0.
Cite as: CrawlCheck (2026). Agent Traffic Index, 2026-10 (month to date). https://crawlcheck.io/traffic-index?month=2026-10
Add your site to the index
A site on Cloudflare reports with three lines in its Worker: the snippet in /spec/visitor-resolve checks each agent request with Visitor Resolve and sends one arrival per hit after the response is out. The site is proven by the CF-Worker header Cloudflare sets, so no key is needed, and the site is never named here. Each new site lifts more agents over the 5-site floor. Per-agent behaviour on one property is in Agent conduct.