Library
What agents are served, measured on real sites.
Every finding is one measurement of what AI agents and crawlers were actually served; every guide is how to run the same check on your own site. Organised by topic, most read first. A business found with a defect is named only after its owner has been contacted; public vendors and AI operators measured on their own sites are named.
New findings are posted on LinkedIn the day they publish, and the feed carries every one.
In one sentence
Findings are short case studies written from real measurements — a robots.txt that answered 200 with a challenge page, a site whose GPTBot traffic came from one address wearing seven crawler names, a homepage that was two percent readable text, a contact email that bounced for one published character — each with the bytes that showed it; a business found with a defect is named only after its owner has been contacted.
20 exclusive findings
Only measured here
Measured only on crawlcheck.io: findings never published, or never observed at this size, anywhere else.
2026-11-02 · Research · Exclusive5 views · 6 min
Google's Googlebot check passes 198 of 200 rented servers
Google's verification rule accepts reverse DNS under googleusercontent.com, the domain every Google Cloud customer machine gets. In a 200-address sample, 198 passed, and two open-source verifiers accepted a rented server as Googlebot.
Read the measurements →
2026-11-02 · Research · Exclusive2 views · 7 min
State of AI visibility, October 2026: 139,788 homepages
The open dataset's first full census: how many sites block AI crawlers, how many publish llms.txt, how little of a homepage is text, and what one audit cannot see.
ROBOTS_RULES_SHADOWED13%STALE_CACHE_SERVED15.1%NO_LLMS_TXT32.6%+2
Read the measurements →
2026-11-02 · Research · Exclusive1 view · 5 min
Google-Extended has no user agent. 1,582 requests used it
Google's documentation says Google-Extended, a robots.txt control token, has no HTTP user-agent string. Over seven days, 1,582 requests reached sixteen of our sites carrying it anyway.
Read the measurements →
Measured something nobody has published? Submit your finding →
11 guides · 3 findings
Crawler access and robots.txt
Whether a crawler is let in, and whether the server agrees with what robots.txt says.
2026-11-02 · Case studies5 views · 6 min
robots.txt returns 200 but no crawler can read it
Every monitor said the file was fine. Every client, from Googlebot to curl, received a 12 KB verification page instead, and an llms.txt elsewhere turned out to be a login wall. How it happens, and the two-request test that catches it.
CHALLENGE_SERVED_2000.3%UNIFORM_REFUSAL12.3%
Read the measurements →
2026-11-024 views · 7 min
How to block AI crawlers like GPTBot and ClaudeBot
Blocking is a legitimate choice and this is how to do it properly. It is also three separate decisions that usually get made as one, and the one people regret is rarely the one they intended.
AI_OPTOUT_SET2.4%ROBOTS_RULES_SHADOWED13%ANSWER_ENGINE_REFUSED6.2%+2
Read the measurements →
2026-11-023 views · 5 min
Is Cloudflare blocking AI crawlers on your site?
Cloudflare’s defaults changed on 15 September 2026 and reached hardest the sites that never opened the setting. How to tell which side of it you are on.
ANSWER_ENGINE_REFUSED6.2%AI_OPTOUT_SET2.4%CHALLENGE_SERVED_2000.3%+3
Read the measurements →
5 guides · 9 findings
Crawler identity and verification
Which requests really came from the crawler they name, and how to check your own logs.
2026-11-02 · Research · Exclusive10 views · 5 min
Eight weeks of AI crawlers on one site: who was really who
Every request that claimed a named crawler on crawlcheck.io from 13 August to 7 October 2026, checked against the ranges its operator publishes. The index crawlers were nearly always real, the browsing agents far less so, three names only ever arrived forged, and every verified agent kept to robots.txt.
Read the measurements →
2026-11-02 · Research · Exclusive5 views · 6 min
Google's Googlebot check passes 198 of 200 rented servers
Google's verification rule accepts reverse DNS under googleusercontent.com, the domain every Google Cloud customer machine gets. In a 200-address sample, 198 passed, and two open-source verifiers accepted a rented server as Googlebot.
Read the measurements →
2026-11-024 views · 7 min
How to verify Googlebot, GPTBot and other crawlers by IP
A user-agent string is a claim anyone can type. Verification is reverse DNS with a forward confirm, or a match against the operator's published IP list, and the answer has three outcomes, not two.
Read the measurements →
11 guides · 8 findings
What crawlers are actually served
Status codes, caches, JavaScript and duplicate hosts: the bytes a crawler receives next to what a browser gets.
2026-11-02 · Research · Exclusive6 views · 6 min
AI crawlers on Wix, Shopify, Squarespace, Webflow: 980 sites
We fetched 980 homepages on four website builders as GPTBot, ClaudeBot and PerplexityBot. The platform decides most of the answer: Shopify and Wix publish an llms.txt for nearly every site, Wix serves crawlers a lighter page, and Squarespace’s AI switch leaves ChatGPT search and Perplexity open.
Read the measurements →
2026-11-025 views · 5 min
Hreflang audit: fixing return-link and x-default errors
Hreflang tells a search engine which language or regional version of a page to show. It only counts when every version links back. The rules, the six errors that make Google ignore a set, and how to audit it from the outside.
HOST_DUPLICATE_2006.1%
Read the measurements →
2026-11-02 · Case studies4 views · 4 min
Why word count cannot spot a parked domain: 783 words
A live domain-sale lander carried 783 words and passed every thin-content check we ran. What actually separates a parked page from a real site, and how to check one yourself from the page source.
Read the measurements →
5 guides · 5 findings
AI agents and MCP
What an AI agent can see, use and verify on a site, and the MCP servers agents connect to.
2026-11-02 · Research · Exclusive3 views · 3 min
AI agents cannot refuse what they cannot see: 27 traps
A revoked credential and an out-of-date fact look exactly like a valid one on the page. So the models walked straight through them.
Read the measurements →
2026-11-02 · Research · Exclusive2 views · 5 min
MCP Registry census: what 25,983 endpoints answered
We sent the MCP handshake to every remote endpoint the official MCP Registry lists, read the tool lists that came back, and read every flagged description by hand. Most of it is clean. The exceptions matter because an agent connects to them without a person reading a word.
Read the measurements →
2026-11-02 · Research · Exclusive1 view · 2 min
MCP reliability, October 2026: 39,484 handshakes measured
The first monthly reliability edition for the remote servers in the official MCP Registry. Every server that has ever answered the handshake is contacted in rotation; no tool is ever called.
Read the measurements →
6 guides · 3 findings
llms.txt, sitemaps and machine files
The files written for machines, and whether anything reads them.
2026-11-028 views · 12 min
XML sitemap audit: check the file and every URL in it
A sitemap can return 200, open in a browser and still be unusable. The audit has two layers, verify the document and then verify every URL it declares, and most audits stop after the first.
NO_SITEMAP_FOUND9.8%ROBOTS_NO_SITEMAP8.8%DECLARED_SITEMAP_BROKEN0.8%+6
Read the measurements →
2026-11-026 views · 4 min
Does llms.txt work? What is known and how to test it
No model provider documents it as an input, so the honest answer is a test, not an opinion. We ran that test on our own domain, and you can run it on yours.
NO_LLMS_TXT32.6%ROBOTS_NOT_2005.2%ROBOTS_IS_HTML0.1%+5
Read the measurements →
2026-11-02 · Research4 views · 5 min
Linked from every page, our llms.txt was fetched 8 times
In two weeks, sixteen AI and search crawler names made 12,831 requests to crawlcheck.io and asked for llms.txt eight times. Per URL, that is the rate of an ordinary page, not a front door.
Read the measurements →
4 guides · 4 findings
Schema and entities
Structured data that parses, and structured data that is true.
2026-11-023 views · 5 min
Rich Results Test vs Schema Markup Validator vs CrawlCheck
Three free structured-data tools answer three questions: Google rich-result eligibility, schema.org validity, and whether the markup describes one business that matches the page.
ENTITY_COLLISION2.9%ENTITY_NO_COORDINATES6.1%NAP_UNDECLARED_LISTING4.3%+1
Read the measurements →
2026-11-022 views · 6 min
What is entity SEO? What machines must resolve to cite you
Being read is not being identified. If a system cannot tell you apart from a similar business two towns over, it names the wrong one or nobody, and both look the same from your side.
ENTITY_COLLISION2.9%ENTITY_NO_COORDINATES6.1%
Read the measurements →
2026-11-02 · Case studies1 view · 6 min
A newspaper’s schema coordinates put it in China
One missing minus sign. The address in the same schema block names a US city; the longitude below it points at northern China, and a range check calls it valid.
Read the measurements →
2 guides · 8 findings
Local business data
Where maps and AI assistants get business facts, and how to keep yours consistent.
2026-11-02 · Research5 views · 6 min
Does Data Axle feed Google? Where business data comes from
Local SEO advice says data aggregators feed Google. Google's own notice names no US supplier, no AI assistant documents Data Axle as a source, and eight of nine businesses we checked have no Data Axle record.
NAP_UNDECLARED_LISTING4.3%
Read the measurements →
2026-11-02 · Research · Exclusive2 views · 6 min
Google's Maps provider list: 793 names, no Yelp
A recovered Geostore enumeration names every source a fact about a place can come from. The consumer directories that citation services sell by the dozen are absent. The business's own website is present, with the same standing as a purchased feed.
Read the measurements →
2026-11-02 · Research2 views · 4 min
Google Places API vs Google Maps rankings: why they don't match (tested in 4 cities)
Same query, four Denver-area cities, one day. The API and Maps agreed on who competes, disagreed on the order, and the API could not find two businesses Maps ranks.
Read the measurements →
11 guides · 7 findings
Measuring AI visibility
What visibility scores and audit tools measure, compared, with CrawlCheck included.
2026-11-0212 views · 9 min
Web scraping APIs compared: the real cost per page
ScrapingBee, ZenRows, ScraperAPI, Bright Data, Firecrawl, Apify and Browse AI converted from credits into pages, plus Crawl4AI, Scrapy and Jina Reader for free, and the one question none of them answers about your own site.
ANSWER_ENGINE_REFUSED6.2%CRAWLER_SERVED_LESS0.8%CHALLENGE_SERVED_2000.3%+3
Read the measurements →
2026-11-029 views · 3 min
How to read a CrawlCheck report: grade, fixes, evidence
What each part of the report answers, which part to open first, and how to check a finding against the bytes behind it.
Read the measurements →
2026-11-029 views · 5 min
YouTube SEO case study: 81 videos rewritten, measured
A dormant channel had 81 titles and descriptions rewritten in two dated batches, with the baseline published first. Search views rose 19% against a flat year-earlier control; the flat and falling rows are printed beside it.
Read the measurements →
7 guides · 4 findings
Site health and security
Links, contrast, security headers and email: checks that break quietly.
2026-11-026 views · 11 min
How to test text color contrast on hover and keyboard focus
Static contrast checkers measure a page at rest. On our own site, hover and focus found 65 more failure patterns than the at-rest check did. Six steps to test them, and the usual causes.
Read the measurements →
2026-11-023 views · 9 min
How to move DMARC from p=none to p=quarantine without losing email
The sequence for tightening DMARC safely: inventory every sender, prove alignment from received headers, create the report mailbox first, then raise the policy in stages.
Read the measurements →
2026-11-022 views · 10 min
How to find broken internal links and dead anchor links on your site
A link audit is an inventory, a fetch, a classification and a second pass over in-page anchors. On our own site, 2,260 link targets came down to four real defects, all from shared templates.
Read the measurements →
6 findings
Web-wide census
Measured across the web's homepages, not one site.
2026-11-02 · Research · Exclusive2 views · 7 min
State of AI visibility, October 2026: 139,788 homepages
The open dataset's first full census: how many sites block AI crawlers, how many publish llms.txt, how little of a homepage is text, and what one audit cannot see.
ROBOTS_RULES_SHADOWED13%STALE_CACHE_SERVED15.1%NO_LLMS_TXT32.6%+2
Read the measurements →
2026-11-02 · Research · Exclusive1 view · 3 min
Only 6% of websites block an AI crawler: the full breakdown
Which AI crawlers 139,788 sites disallow in robots.txt, how training and search agents are treated differently, and how blocking varies by kind of site.
Read the measurements →
2026-11-02 · Research · Exclusive1 view · 2 min
Which AI crawlers the top 10,000 sites block most, October 2026
CCBot, Bytespider and GPTBot are closed off most often in robots.txt; AI search crawlers far less. And at the door, a fifth of top sites refuse GPTBot and ClaudeBot whatever robots.txt says.
Read the measurements →
36 corrections and release notesHow we fix CrawlCheck itself
Every defect found in CrawlCheck, told against ourselves, with what changed.