Library · Web-wide census
Web-wide census
In one sentence
Findings are short case studies written from real measurements — a robots.txt that answered 200 with a challenge page, a site whose GPTBot traffic came from one address wearing seven crawler names, a homepage that was two percent readable text, a contact email that bounced for one published character — each with the bytes that showed it; a business found with a defect is named only after its owner has been contacted.
Measured across the web's homepages, not one site.
2026-11-02 · Research · Exclusive7 min
State of AI visibility, October 2026: 139,788 homepages
The open dataset's first full census: how many sites block AI crawlers, how many publish llms.txt, how little of a homepage is text, and what one audit cannot see.
ROBOTS_RULES_SHADOWED13%STALE_CACHE_SERVED15.4%NO_LLMS_TXT32.7%+2
Read the measurements →
2026-11-02 · Research · Exclusive3 min
Only 6% of websites block an AI crawler: the full breakdown
Which AI crawlers 139,788 sites disallow in robots.txt, how training and search agents are treated differently, and how blocking varies by kind of site.
Read the measurements →
2026-11-02 · Research · Exclusive2 min
Which AI crawlers the top 10,000 sites block most, October 2026
CCBot, Bytespider and GPTBot are closed off most often in robots.txt; AI search crawlers far less. And at the door, a fifth of top sites refuse GPTBot and ClaudeBot whatever robots.txt says.
Read the measurements →
2026-11-02 · Research3 min
MobileOptimized tag: on 1.65M sites, none in the top 10,000
A Microsoft tag for a browser that no longer exists sits on about 1.65 million sites and is still spreading, yet the top 10,000 have never carried it. Read by rank band, BuiltWith's adoption figures say the opposite of the raw count.
Read the measurements →
2026-11-02 · Research · Exclusive3 min
Robots.txt allows it, the server refuses: 57 top sites
Same day, same frozen list of the most-visited sites, four AI crawler names. At 57 of 418 sites, the server refused a crawler that the site's robots.txt allows.
Read the measurements →