Library · Crawler identity and verification
Crawler identity and verification
In one sentence
Findings are short case studies written from real measurements — a robots.txt that answered 200 with a challenge page, a site whose GPTBot traffic came from one address wearing seven crawler names, a homepage that was two percent readable text, a contact email that bounced for one published character — each with the bytes that showed it; a business found with a defect is named only after its owner has been contacted.
Which requests really came from the crawler they name, and how to check your own logs.
2026-11-02 · Research · Exclusive5 min
Eight weeks of AI crawlers on one site: who was really who
Every request that claimed a named crawler on crawlcheck.io from 13 August to 7 October 2026, checked against the ranges its operator publishes. The index crawlers were nearly always real, the browsing agents far less so, three names only ever arrived forged, and every verified agent kept to robots.txt.
Read the measurements →
2026-11-027 min
How to verify Googlebot, GPTBot and other crawlers by IP
A user-agent string is a claim anyone can type. Verification is reverse DNS with a forward confirm, or a match against the operator's published IP list, and the answer has three outcomes, not two.
Read the measurements →
2026-11-02 · Research · Exclusive6 min
Google's Googlebot check passes 198 of 200 rented servers
Google's verification rule accepts reverse DNS under googleusercontent.com, the domain every Google Cloud customer machine gets. In a 200-address sample, 198 passed, and two open-source verifiers accepted a rented server as Googlebot.
Read the measurements →
2026-11-02 · Research · Exclusive5 min
Google-Extended has no user agent. 1,582 requests used it
Google's documentation says Google-Extended, a robots.txt control token, has no HTTP user-agent string. Over seven days, 1,582 requests reached sixteen of our sites carrying it anyway.
Read the measurements →
2026-11-024 min
Verify GPTBot, ClaudeBot and PerplexityBot: tools compared
Where OpenAI, Anthropic, Perplexity and Google publish their crawler addresses, the tools that check traffic against them (CrawlCheck, Cloudflare, manual), and why unverifiable is its own answer.
ANSWER_ENGINE_REFUSED6.1%
Read the measurements →
2026-11-0210 min
AI crawler list: what each bot does and how to verify it
The crawler names a site actually sees, grouped by the job each one does, with which ones can be checked from a server log and the robots.txt lines that keep one job while refusing another.
ROBOTS_RULES_SHADOWED13%
Read the measurements →
2026-11-02 · Corrections6 min
Is your host blocking ClaudeBot? Group the 403s by IP first
A 57% block rate in our own access logs looked like a hosting provider shutting out AI crawlers. Grouped by source address, almost all of it was one machine wearing seven different crawler names.
Read the measurements →
2026-11-02 · Research5 min
Which AI crawlers can't be verified by IP range?
On 18 August, 251 of 950 requests claiming a named crawler came from operators who published no address range. Those requests are not forged; they are unfalsifiable. Two of those operators have since published ranges. Here is who is checkable and what to do about the rest.
Read the measurements →
2026-11-02 · Case studies · Exclusive4 min
Fake GPTBot and ClaudeBot requests scan for .env files
Of 874 recorded requests from clients claiming to be a named AI crawler, 69 asked for a file that holds secrets, such as cloud credentials, Terraform state and .env files, across at least 66 paths no real crawler requests.
Read the measurements →
2026-11-02 · Research · Exclusive6 min
How much AI crawler traffic is forged? We check the IPs
A user-agent string is a claim, not an identity. Every request here that names a known crawler is checked against the IP ranges its operator publishes, and the live share that fails is on this page.
Read the measurements →
2026-11-024 min
Web Bot Auth: how signed bot requests work, and who signs
IP lists stop working when agents share cloud addresses. Web Bot Auth has the agent sign each request with a key it publishes. How a site verifies one, the CDN caching trap that breaks key rotation, and which operators document signing today.
Read the measurements →
2026-11-02 · Research · Exclusive3 min
AI agents acting for people: 613 visits, one read robots.txt
The fetchers an assistant sends when a person asks about a page, counted on one site and checked against each operator’s published address ranges. They are a small share of AI traffic, the most forged names in it, and they handle robots.txt exactly as their operators document.
Read the measurements →
2026-11-02 · Research · Exclusive2 min
Agent conduct, October 2026: 34,676 verified requests
The first monthly conduct edition. Every request naming a tracked agent was checked against its operator’s published address ranges, and only verified requests count toward that agent’s record.
Read the measurements →
2026-11-023 min
How to verify Amazonbot and Amzn-SearchBot by IP address
Anyone can put Amazonbot in a user agent. Amazon publishes the addresses its crawlers really use, as a list on a web page. Here is how to check a request against it, and what we found when we started checking.
Read the measurements →