Findings · 2026-10-15 · By VSNARY | Emmanuel Orta · 0 views
Agent conduct, October 2026: 34,676 verified crawler requests, and what they did with robots.txt
The first monthly conduct edition. Every request naming a tracked agent was checked against the address ranges its operator publishes; only the verified ones count toward the agent's record. Fourteen agents have enough verified traffic for a verdict.
From 12 August to 15 October 2026, crawlcheck.io received 34,676 requests from 14 tracked agents that passed address verification against each operator's published ranges. Nine of the twelve crawlers that robots.txt governs fetched it; three never fetched it under their own name. Four fetches of disallowed paths were counted before per-fetch records were kept, so they are reported and not judged. 842 requests (2.4%) used a tracked agent's name from outside its operator's ranges.
Disclosure: CrawlCheck publishes this edition and sells checks built on the same measurements. No site, server or company is named in this post; the per-agent and per-server readings are on the live record linked below, regenerated and signed.
A crawler that names itself in its user agent is making a claim. This edition only counts the claims that check out: a request is credited to an agent when its source address sits inside the ranges that agent's operator publishes, or passes the operator's reverse-DNS check. Everything else that used the name is counted against the impersonator, not the agent.
The figures #
| Measure | Value |
|---|---|
| Agents with a verdict (50 or more verified requests) | 14 |
| Verified requests from those agents | 34,676 |
| Requests using a tracked agent's name from outside its ranges | 842 (2.4%) |
| Crawlers governed by robots.txt that fetched it | 9 of 12 |
| Crawlers governed by robots.txt that never fetched it under their own name | 3 of 12 |
| Fetches of disallowed paths counted | 4, by 2 agents, not judged (below) |
| Agents that cannot be assessed (no published address ranges) | 12, claiming 78,534 requests |
| Busiest minute from one agent, all its addresses combined | 119 requests |
Reading robots.txt is not the same as obeying it #
Three crawlers made no request for a disallowed path and also never fetched robots.txt under their own name here. Until 15 October the record called that “respected”, which claimed more than was measured: a crawler that never read the rules cannot be credited with following them. The record now says robots_txt_not_fetched for that case. The operator may fetch robots.txt under a different name; this record only sees what each name did.
Four fetches we cannot stand behind yet #
Two agents were counted fetching paths this site's robots.txt disallows for every crawler (/r/, /backup/, /admin-old/): four fetches in all. The counter recorded that they happened but not when, which path or from which network, so none of them can be checked by hand. The record now reports them as disallowed_fetch_unrecorded and does not call either agent a rule-breaker. From 15 October every verified fetch of a disallowed path is kept with its time, path and network, and a verdict needs at least one such record.
Agents that cannot be assessed #
Twelve agents that name themselves publish no address ranges this record can read, so a request carrying their name cannot be told from an impersonator. Together they claimed 78,534 requests. They are listed, with their claimed counts, and excluded from every figure above.
What this does not show #
- One property, not the web: this is what each agent did on crawlcheck.io.
- The busiest minute is reported, not judged: this site publishes no crawl rate.
- A clean record covers the rules this site sets; it says nothing about other sites.
The per-agent record, signed: /agents/conduct · JSON: /api/v1/conduct · weekly editions (Atom): feed · every edition: /editions.
Every figure above came out of CrawlCheck.
Point it at your own domain and see the same measurements, free.
The main product
Found this on your own site? We fix it for $749.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Questions this post answers
How does CrawlCheck know a request really came from a crawler?
It checks the source address against the address ranges the crawler's operator publishes, or the operator's reverse-DNS check. Only requests that pass are credited to the crawler.
Did any crawler break robots.txt on crawlcheck.io?
Four fetches of disallowed paths were counted by two agents before 15 October 2026, but no per-fetch record was kept, so they are reported and not judged. From 15 October each such fetch is recorded with its time, path and network.
Why is a crawler that never fetched robots.txt not called compliant?
A crawler that never read the rules cannot be credited with following them. The record says robots_txt_not_fetched instead.
How often is a crawler's name forged?
842 of the requests using a tracked agent's name, 2.4%, came from outside its operator's ranges between 12 August and 15 October 2026.
Related findings
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.