CrawlCheck

Library · Research

Research

In one sentence

Findings are short case studies written from real measurements — a robots.txt that answered 200 with a challenge page, a site whose GPTBot traffic came from one address wearing seven crawler names, a homepage that was two percent readable text, a contact email that bounced for one published character — each with the bytes that showed it; a business found with a defect is named only after its owner has been contacted.

Measured across the corpus or the telemetry, not one site.

107findings
66guides
12topics
9,163scans behind the live figures

2026-11-02 · Research · Exclusive10 views · 5 min

Eight weeks of AI crawlers on one site: who was really who

Every request that claimed a named crawler on crawlcheck.io from 13 August to 7 October 2026, checked against the ranges its operator publishes. The index crawlers were nearly always real, the browsing agents far less so, three names only ever arrived forged, and every verified agent kept to robots.txt.

Read the measurements →

2026-11-02 · Research · Exclusive6 views · 6 min

AI crawlers on Wix, Shopify, Squarespace, Webflow: 980 sites

We fetched 980 homepages on four website builders as GPTBot, ClaudeBot and PerplexityBot. The platform decides most of the answer: Shopify and Wix publish an llms.txt for nearly every site, Wix serves crawlers a lighter page, and Squarespace’s AI switch leaves ChatGPT search and Perplexity open.

Read the measurements →

2026-11-02 · Research · Exclusive5 views · 3 min

AI visibility vendors audited: 11 of 23 have llms.txt

The companies that grade machine readability, run through the same checks they would point at you. The reading is from a directory that re-scans itself, so today's figures are one click away.

Read the measurements →

2026-11-02 · Research5 views · 6 min

Does Data Axle feed Google? Where business data comes from

Local SEO advice says data aggregators feed Google. Google's own notice names no US supplier, no AI assistant documents Data Axle as a source, and eight of nine businesses we checked have no Data Axle record.

NAP_UNDECLARED_LISTING3.6%

Read the measurements →

2026-11-02 · Research · Exclusive5 views · 6 min

Google's Googlebot check passes 198 of 200 rented servers

Google's verification rule accepts reverse DNS under googleusercontent.com, the domain every Google Cloud customer machine gets. In a 200-address sample, 198 passed, and two open-source verifiers accepted a rented server as Googlebot.

Read the measurements →

2026-11-02 · Research4 views · 5 min

Linked from every page, our llms.txt was fetched 8 times

In two weeks, sixteen AI and search crawler names made 12,831 requests to crawlcheck.io and asked for llms.txt eight times. Per URL, that is the rate of an ordinary page, not a front door.

Read the measurements →

2026-11-02 · Research · Exclusive3 views · 3 min

AI agents cannot refuse what they cannot see: 27 traps

A revoked credential and an out-of-date fact look exactly like a valid one on the page. So the models walked straight through them.

Read the measurements →

2026-11-02 · Research · Exclusive3 views · 6 min

Google's Maps provider list: 793 names, no Yelp

A recovered Geostore enumeration names every source a fact about a place can come from. The consumer directories that citation services sell by the dozen are absent. The business's own website is present, with the same standing as a purchased feed.

Read the measurements →

2026-11-02 · Research2 views · 3 min

Missing referrer is not direct traffic: 93% of our arrivals sent none

Across five sites and 1,104 recorded arrivals, 1,026 carried no referrer at all. Every claim about where AI traffic comes from is being made on the remaining seven per cent.

Read the measurements →

2026-11-02 · Research2 views · 4 min

Google Places API vs Google Maps rankings: why they don't match (tested in 4 cities)

Same query, four Denver-area cities, one day. The API and Maps agreed on who competes, disagreed on the order, and the API could not find two businesses Maps ranks.

Read the measurements →

2026-11-02 · Research · Exclusive2 views · 7 min

State of AI visibility, October 2026: 139,788 homepages

The open dataset's first full census: how many sites block AI crawlers, how many publish llms.txt, how little of a homepage is text, and what one audit cannot see.

ROBOTS_RULES_SHADOWED14%STALE_CACHE_SERVED13.4%NO_LLMS_TXT34.2%+2

Read the measurements →

2026-11-02 · Research · Exclusive2 views · 5 min

MCP Registry census: what 25,983 endpoints answered

We sent the MCP handshake to every remote endpoint the official MCP Registry lists, read the tool lists that came back, and read every flagged description by hand. Most of it is clean. The exceptions matter because an agent connects to them without a person reading a word.

Read the measurements →

2026-11-02 · Research · Exclusive2 views · 4 min

MCP servers that ask for a login: what 1,366 request first

We read the login requirements of every MCP server in the official registry that asks for one, without signing in. 44% had at least one finding: most often a default request for every scope the server offers, or no metadata telling a client what to ask for.

Read the measurements →

2026-11-02 · Research1 view · 3 min

AI picked a local contractor 422 times: rating decided it

We changed one fact at a time about a fictional business and watched where the AI ranked it.

Read the measurements →

2026-11-02 · Research · Exclusive1 view · 5 min

Google-Extended has no user agent. 1,582 requests used it

Google's documentation says Google-Extended, a robots.txt control token, has no HTTP user-agent string. Over seven days, 1,582 requests reached sixteen of our sites carrying it anyway.

Read the measurements →

2026-11-02 · Research1 view · 6 min

What AI crawlers receive from local business sites

Three things we found by fetching real local business sites the way answer engines do, what moved the numbers on sites we run, and the commands to repeat it on yours.

PAGE_IS_MOSTLY_CODE28.4%

Read the measurements →

2026-11-02 · Research · Exclusive1 view · 3 min

Only 6% of websites block an AI crawler: the full breakdown

Which AI crawlers 139,788 sites disallow in robots.txt, how training and search agents are treated differently, and how blocking varies by kind of site.

Read the measurements →

2026-11-02 · Research · Exclusive1 view · 2 min

MCP reliability, October 2026: 39,484 handshakes measured

The first monthly reliability edition for the remote servers in the official MCP Registry. Every server that has ever answered the handshake is contacted in rotation; no tool is ever called.

Read the measurements →

2026-11-02 · Research · Exclusive1 view · 2 min

Which AI crawlers the top 10,000 sites block most, October 2026

CCBot, Bytespider and GPTBot are closed off most often in robots.txt; AI search crawlers far less. And at the door, a fifth of top sites refuse GPTBot and ClaudeBot whatever robots.txt says.

Read the measurements →

2026-11-15 · Research4 min

Top 1,000 websites: every outside tracker we logged

We opened the homepages of the 1,000 most visited websites in a real browser and logged every outside domain that loaded, set a cookie, read canvas pixels or listened to the keyboard.

Read the measurements →

2026-11-14 · Research3 min

One in four MCP hosts changed its tools in four days. Nothing told the agent.

We re-read the tool list of every public server in the MCP Registry. Most changes rewrote a description, and most kept the same version number.

Read the measurements →

2026-11-03 · Research4 min

One Cloudflare setting is behind most of the new AI training blocks on top sites

Between 3 October and 1 November 2026, about 120 of the top 10,000 sites started blocking AI training crawlers in robots.txt. Three in four did it the same way: by switching on Cloudflare’s managed robots.txt. AI search crawlers were left alone.

Read the measurements →

2026-11-03 · Research3 min

Six in ten websites already use post-quantum encryption. Whether yours does mostly depends on one thing

We tested 1,295 websites for the post-quantum key exchange that protects today’s traffic from being decrypted later. 59.6% support it. Behind Cloudflare, 99.7% do; everywhere else, 44.8%.

Read the measurements →

2026-11-02 · Research3 min

Lighthouse scores vary between runs: two runs, 225% apart

Nothing on the page changed. Blocking time came back 2,643 ms, then 812 ms, and first paint moved 69%. What two runs prove, what they do not, and how to measure your own noise floor before trusting a score.

Read the measurements →

2026-11-02 · Research3 min

AI text watermarks only work if the provider opted in

Watermarking is real and the mechanism is sound. But it covers one vendor's models, you cannot run the detector, and paraphrasing removes most of the signal.

Read the measurements →

2026-11-02 · Research6 min

Text extractors dropped our schema and still got us right

Two unrelated services that turn pages into text for language models both dropped the whole head of our page, and every JSON-LD node with it. Both still named the company correctly. Across ten of our own sites, 87% of our structured data sits where they do not look.

Read the measurements →

2026-11-02 · Research4 min

AI visibility scores measure an API, not ChatGPT

The answers these tools record are real. The label on top — “your visibility in AI search” — describes something that was never measured.

Read the measurements →

2026-11-02 · Research5 min

Which AI crawlers can't be verified by IP range?

On 18 August, 251 of 950 requests claiming a named crawler came from operators who published no address range. Those requests are not forged; they are unfalsifiable. Two of those operators have since published ranges. Here is who is checkable and what to do about the rest.

Read the measurements →

2026-11-02 · Research · Exclusive6 min

How much AI crawler traffic is forged? We check the IPs

A user-agent string is a claim, not an identity. Every request here that names a known crawler is checked against the IP ranges its operator publishes, and the live share that fails is on this page.

Read the measurements →

2026-11-02 · Research3 min

Site audits go stale: why one clean scan proves little

Measuring the same sites every day produces a number a one-off audit cannot: how many were broken at least once. The gap between “broken now” and “ever broken” is the case for a series over a snapshot.

STALE_CACHE_SERVED13.4%MACHINE_FILE_CACHE_STALE0.4%CHALLENGE_SERVED_2000.3%+1

Read the measurements →

2026-11-02 · Research3 min

MobileOptimized tag: on 1.65M sites, none in the top 10,000

A Microsoft tag for a browser that no longer exists sits on about 1.65 million sites and is still spreading, yet the top 10,000 have never carried it. Read by rank band, BuiltWith's adoption figures say the opposite of the raw count.

Read the measurements →

2026-11-02 · Research3 min

Opening hours and prices: the facts sites don’t publish

“Are they open?” and “what does it cost?” are the questions most asked about a business. Declaring that a business exists is common; declaring its hours and prices is an order of magnitude rarer.

Read the measurements →

2026-11-02 · Research · Exclusive5 min

402 Payment Required: what top sites send AI crawlers

Three of the top 1,000 sites answer AI crawlers with HTTP 402. None quotes a price: one is a refusal, one a toll, one contradicts its own robots.txt.

Read the measurements →

2026-11-02 · Research · Exclusive3 min

AI agents acting for people: 613 visits, one read robots.txt

The fetchers an assistant sends when a person asks about a page, counted on one site and checked against each operator’s published address ranges. They are a small share of AI traffic, the most forged names in it, and they handle robots.txt exactly as their operators document.

Read the measurements →

2026-11-02 · Research · Exclusive2 min

Agent conduct, October 2026: 34,676 verified requests

The first monthly conduct edition. Every request naming a tracked agent was checked against its operator’s published address ranges, and only verified requests count toward that agent’s record.

Read the measurements →

2026-11-02 · Research · Exclusive3 min

Robots.txt allows it, the server refuses: 57 top sites

Same day, same frozen list of the most-visited sites, four AI crawler names. At 57 of 418 sites, the server refused a crawler that the site's robots.txt allows.

Read the measurements →

2026-11-02 · Research6 min

Four AI models, three local businesses: who got it right

We asked Gemini with Google Search and three open-weights models for the phone number, address and hours of three home-service businesses whose true answers we know. Search fixed the phone numbers. The addresses came from directories, and one model invented whole listings.

Read the measurements →

2026-11-02 · Research · Exclusive4 min

Do AI agents stop before acting on a lookalike? We tested 4

27 tasks, half of them traps: lookalike domains, poisoned tool descriptions, revoked receipts and stale answers. Four models decided alone, then again with a signed answer about the domain. The traps they missed alone were the ones a page cannot reveal.

Read the measurements →

2026-11-02 · Research3 min

Google AI Overview sources for Denver tree removal cost

One real search on 21 October 2026, read on the results page itself. One directory, one forum thread, four local businesses. One cited contractor refuses every crawler we sent.

Read the measurements →

2026-11-02 · Research4 min

AI citations on two dates: seven matched questions

The same seven questions on Google AI Mode and Perplexity, asked on 5 and 29 October 2026: four pairs became cited, none were lost. A wider 27-question pass is alongside. Neither shows why.

Read the measurements →

Questions about this page

QAre these real sites?
Yes, every finding is a measurement taken on a live site. Domains are not named unless the author owns them.
QHow is a finding different from a guide?
A finding reports one measured case and what it showed. A guide explains how to run and read the check yourself.

Keep reading