Findings · 2026-10-15 · By VSNARY | Emmanuel Orta · 0 views

Robots.txt says yes, the server says no: 57 top sites refuse AI crawlers their own rules allow

Same day, same frozen list of the most-visited sites, four AI crawler names. At 57 of 418 sites, the server refused a crawler that the site's robots.txt allows.

Disclosure: CrawlCheck publishes this post and sells checks built on the same measurements. No site is named; the per-site records are public, signed and linked below.

A robots.txt file is a site's stated policy for crawlers. The server is what actually answers them. We compared the two on the same day, on the same frozen list of the most-visited sites, for four AI crawler names: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic) and PerplexityBot (Perplexity).

The short answer #

Of the 418 top sites that answered a browser and served a robots.txt, 79 refused at least one of the four AI crawler names while the browser was let in. At 57 of those 79, the site's own robots.txt allows that crawler. The rules say yes; the server says no.

The figures #

Census run 14, finished 15 October 2026: the top 1,000 domains of one Tranco list, frozen on 2 October so every run measures the same sites (991 after the publish filter). 540 answered a mobile browser with a 2xx page; 418 of those served a robots.txt.

Base: the 418 sites that answered a browser with 2xx and served a robots.txt. "Refuses" means the request carrying that crawler's name got a non-2xx answer where the browser got 2xx on the same run. Most refusals were plain 403 Forbidden: 30 of the 46 for GPTBot, the rest 429, 406, 503 or a 520 origin error.

The middle column is expected: robots.txt is advisory, so a server serving a crawler its rules disallow is normal, and the crawler is meant to stop itself. The first column is the one site owners rarely know about, because nothing in the robots.txt shows it.

Sites with no robots.txt #

122 of the 540 reachable sites served no robots.txt at all. Even there, around 30 refused each AI crawler name at the server: 30 for GPTBot, 29 for ClaudeBot, 28 for PerplexityBot, 27 for OAI-SearchBot. With no rules published, the edge is the only policy.

What a refusal here does and does not prove #

Each request carried the crawler's name from CrawlCheck's own network, not from the operator's published address ranges. So a refusal has two possible readings, and this census cannot tell them apart for any one site:

• The edge blocks by name. Any request calling itself GPTBot is refused, including the real one. The robots.txt allow is then dead letter, and the site is invisible to that crawler without saying so.
• The edge checks the address. The site admits crawlers only from their operator's published ranges, refuses impostors, and the real crawler gets through. This is good practice, and the robots.txt is accurate.

We re-tested six of the sites in the first column from a second, unrelated network: three refused GPTBot again while serving the browser, two served it, and one refused the browser too. The same name gets different answers from different networks, which fits edges that judge the address, not just the name, and is why the result for any one site needs checking from its own logs.

How to check your own site #

1. Read your robots.txt for the four names above, and for the User-agent: * group that applies when a crawler is not named.
2. Look at your CDN or firewall rules for anything matching crawler user agents or "AI bots": a managed bot-blocking setting can refuse AI crawlers without any line in robots.txt. How to check whether Cloudflare is blocking them.
3. In your server or CDN logs, filter requests naming GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot and check the status codes. If verified requests from the operators' ranges get 403, the block is real.
4. Run a free CrawlCheck scan: it sends each crawler name and a browser to your homepage and shows where the answers differ, side by side.

If the block is deliberate, say so in robots.txt too, so the policy you publish is the one you enforce. If it is not, the fix is in the edge, not the robots.txt.

Check the numbers #

The weekly figures are on the census page, signed, with every per-site record at /api/census/records. The verifier at /crawlcheck-verify.mjs checks a signed copy with no call to us.

Questions this post answers

Does robots.txt control whether an AI crawler can fetch my site?

No. robots.txt states your policy and well-behaved crawlers follow it, but the server or CDN decides what each request receives. On 15 October 2026, 57 of 418 top sites refused an AI crawler their robots.txt allows.

Why would a server refuse a crawler that robots.txt allows?

Usually a CDN or firewall rule that blocks crawler user agents or 'AI bots' as a category, set separately from robots.txt. Some edges instead admit crawlers only from their operator's published address ranges, which refuses impostors but lets the real crawler in.

Does a refusal in this census mean the real GPTBot is blocked?

Not necessarily. Requests carried the crawler's name from CrawlCheck's network, not OpenAI's ranges. A site that verifies addresses will refuse those requests and still admit the real GPTBot. Your own logs settle it.

How do I check what my site serves AI crawlers?

Read robots.txt for the crawler names and the * group, check CDN and firewall bot rules, filter your logs for the crawler names and their status codes, or run a free CrawlCheck scan, which compares each crawler name with a browser side by side.

Which sites were measured?

The top 1,000 registrable domains of one Tranco list, frozen on 2 October 2026 so every run measures the same sites; 991 remain after adult and gambling hosts are removed. No site is named; per-site records are public and signed.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works