Library · Exclusive findings
Exclusive findings
In one sentence
Findings are short case studies written from real measurements — a robots.txt that answered 200 with a challenge page, a site whose GPTBot traffic came from one address wearing seven crawler names, a homepage that was two percent readable text, a contact email that bounced for one published character — each with the bytes that showed it; a business found with a defect is named only after its owner has been contacted.
Measured only on crawlcheck.io: findings never published, or never observed at this size, anywhere else.
2026-11-02 · Research · Exclusive6 min
Google's Googlebot check passes 198 of 200 rented servers
Google's verification rule accepts reverse DNS under googleusercontent.com, the domain every Google Cloud customer machine gets. In a 200-address sample, 198 passed, and two open-source verifiers accepted a rented server as Googlebot.
Read the measurements →
2026-11-02 · Research · Exclusive7 min
State of AI visibility, October 2026: 139,788 homepages
The open dataset's first full census: how many sites block AI crawlers, how many publish llms.txt, how little of a homepage is text, and what one audit cannot see.
ROBOTS_RULES_SHADOWED13%STALE_CACHE_SERVED15.4%NO_LLMS_TXT32.7%+2
Read the measurements →
2026-11-02 · Research · Exclusive3 min
Only 6% of websites block an AI crawler: the full breakdown
Which AI crawlers 139,788 sites disallow in robots.txt, how training and search agents are treated differently, and how blocking varies by kind of site.
Read the measurements →
2026-11-02 · Research · Exclusive2 min
Which AI crawlers the top 10,000 sites block most, October 2026
CCBot, Bytespider and GPTBot are closed off most often in robots.txt; AI search crawlers far less. And at the door, a fifth of top sites refuse GPTBot and ClaudeBot whatever robots.txt says.
Read the measurements →
2026-11-02 · Research · Exclusive3 min
Robots.txt allows it, the server refuses: 57 top sites
Same day, same frozen list of the most-visited sites, four AI crawler names. At 57 of 418 sites, the server refused a crawler that the site's robots.txt allows.
Read the measurements →
2026-11-02 · Research · Exclusive5 min
402 Payment Required: what top sites send AI crawlers
Three of the top 1,000 sites answer AI crawlers with HTTP 402. None quotes a price: one is a refusal, one a toll, one contradicts its own robots.txt.
Read the measurements →
2026-11-02 · Research · Exclusive5 min
MCP Registry census: what 25,983 endpoints answered
We sent the MCP handshake to every remote endpoint the official MCP Registry lists, read the tool lists that came back, and read every flagged description by hand. Most of it is clean. The exceptions matter because an agent connects to them without a person reading a word.
Read the measurements →
2026-11-02 · Research · Exclusive2 min
MCP reliability, October 2026: 39,484 handshakes measured
The first monthly reliability edition for the remote servers in the official MCP Registry. Every server that has ever answered the handshake is contacted in rotation; no tool is ever called.
Read the measurements →
2026-11-02 · Research · Exclusive4 min
MCP servers that ask for a login: what 1,366 request first
We read the login requirements of every MCP server in the official registry that asks for one, without signing in. 44% had at least one finding: most often a default request for every scope the server offers, or no metadata telling a client what to ask for.
Read the measurements →
2026-11-02 · Research · Exclusive2 min
Agent conduct, October 2026: 34,676 verified requests
The first monthly conduct edition. Every request naming a tracked agent was checked against its operator’s published address ranges, and only verified requests count toward that agent’s record.
Read the measurements →
2026-11-02 · Research · Exclusive3 min
AI agents cannot refuse what they cannot see: 27 traps
A revoked credential and an out-of-date fact look exactly like a valid one on the page. So the models walked straight through them.
Read the measurements →
2026-11-02 · Research · Exclusive4 min
Do AI agents stop before acting on a lookalike? We tested 4
27 tasks, half of them traps: lookalike domains, poisoned tool descriptions, revoked receipts and stale answers. Four models decided alone, then again with a signed answer about the domain. The traps they missed alone were the ones a page cannot reveal.
Read the measurements →
2026-11-02 · Research · Exclusive3 min
AI agents acting for people: 613 visits, one read robots.txt
The fetchers an assistant sends when a person asks about a page, counted on one site and checked against each operator’s published address ranges. They are a small share of AI traffic, the most forged names in it, and they handle robots.txt exactly as their operators document.
Read the measurements →
2026-11-02 · Research · Exclusive5 min
Eight weeks of AI crawlers on one site: who was really who
Every request that claimed a named crawler on crawlcheck.io from 13 August to 7 October 2026, checked against the ranges its operator publishes. The index crawlers were nearly always real, the browsing agents far less so, three names only ever arrived forged, and every verified agent kept to robots.txt.
Read the measurements →
2026-11-02 · Research · Exclusive5 min
Google-Extended has no user agent. 1,582 requests used it
Google's documentation says Google-Extended, a robots.txt control token, has no HTTP user-agent string. Over seven days, 1,582 requests reached sixteen of our sites carrying it anyway.
Read the measurements →
2026-11-02 · Case studies · Exclusive4 min
Fake GPTBot and ClaudeBot requests scan for .env files
Of 874 recorded requests from clients claiming to be a named AI crawler, 69 asked for a file that holds secrets, such as cloud credentials, Terraform state and .env files, across at least 66 paths no real crawler requests.
Read the measurements →
2026-11-02 · Research · Exclusive6 min
How much AI crawler traffic is forged? We check the IPs
A user-agent string is a claim, not an identity. Every request here that names a known crawler is checked against the IP ranges its operator publishes, and the live share that fails is on this page.
Read the measurements →
2026-11-02 · Research · Exclusive6 min
AI crawlers on Wix, Shopify, Squarespace, Webflow: 980 sites
We fetched 980 homepages on four website builders as GPTBot, ClaudeBot and PerplexityBot. The platform decides most of the answer: Shopify and Wix publish an llms.txt for nearly every site, Wix serves crawlers a lighter page, and Squarespace’s AI switch leaves ChatGPT search and Perplexity open.
Read the measurements →
2026-11-02 · Research · Exclusive6 min
Google's Maps provider list: 793 names, no Yelp
A recovered Geostore enumeration names every source a fact about a place can come from. The consumer directories that citation services sell by the dozen are absent. The business's own website is present, with the same standing as a purchased feed.
Read the measurements →
2026-11-02 · Research · Exclusive3 min
AI visibility vendors audited: 11 of 23 have llms.txt
The companies that grade machine readability, run through the same checks they would point at you. The reading is from a directory that re-scans itself, so today's figures are one click away.
Read the measurements →
Measured something nobody has published? Submit your finding →