CrawlCheck

Exclusive finding · Findings · 2026-10-06 · By · 1 view

Only 6% of websites block an AI crawler: the full breakdown

Which AI crawlers 139,788 sites disallow in robots.txt, how training and search agents are treated differently, and how blocking varies by kind of site.

Across 139,788 homepages measured on 3-6 October 2026, 6.3% disallow at least one named AI crawler in robots.txt. GPTBot is blocked most (3.7%), then CCBot (3.1%), Bytespider (3.0%) and ClaudeBot (2.7%). Search and user-triggered agents are blocked far less: OAI-SearchBot 0.9%, Claude-User 0.5%. Publisher-heavy sites block most; government and local-business sites block least.

0%16%139,788 homepagestop 10,000 (serving robots.txt)GPTBot3.7%14.3%CCBot3.1%15.1%Bytespider3.0%14.6%ClaudeBot2.7%13.1%Google-Extended2.0%11.5%Meta-ExternalAgent1.7%11.7%Applebot-Extended1.8%11.1%PerplexityBot1.6%9.1%ChatGPT-User1.8%8.1%OAI-SearchBot0.9%6.2%
Share of sites blocking each crawler in robots.txt. Gold: 139,788 homepages, 3–6 October 2026 (all homepages measured). Teal: the top 10,000, census finished 24 October 2026, among the 4,501 that serve a robots.txt (top-site census). The most-visited sites block every crawler four to seven times as often, and the order barely changes: training crawlers first, search and user-triggered agents last. The two bars use different bases (all homepages measured; top sites that serve a robots.txt), so read the gap as direction and size, not a precise multiple.

How many websites actually block AI crawlers? In the CrawlCheck dataset for 3-6 October 2026, the answer is 6.3%: 8,757 of 139,788 measured homepages disallow at least one named AI crawler in robots.txt. Every other site lets them all in, by choice or by default. The figures below count each domain once, at its latest reading, and name no site.

Crawler by crawler #

Crawler (user-agent)What it is forSites that block it
GPTBotCollects pages for model training3.7%
CCBotBuilds an open web crawl widely used for training3.1%
BytespiderCollects pages for model training3.0%
ClaudeBotCollects pages for model training2.7%
Google-ExtendedA control token for AI use of pages Google already crawls2.0%
ChatGPT-UserFetches a page when a user asks for it1.8%
anthropic-aiAn older token for the same purpose as ClaudeBot1.8%
Applebot-ExtendedA control token for AI use of pages Apple already crawls1.8%
meta-externalagentCollects pages for AI use1.7%
PerplexityBotIndexes pages for an answer engine1.6%
OAI-SearchBotIndexes pages for search answers0.9%
Claude-UserFetches a page when a user asks for it0.5%

The AI crawlers page lists each of these with what it does and whether its operator publishes the addresses it crawls from.

The pattern in the table is consistent: crawlers that collect pages for training are blocked two to five times as often as the agents that fetch a page to answer a question or build a search index. 3.1% of sites block GPTBot while leaving OAI-SearchBot open, which is the coherent version of an AI policy: stay out of the training data, stay in the answers. Blocking the search or user agents is what removes a site from the answers, and very few sites do it.

The control tokens are a separate case. Google-Extended and Applebot-Extended do not crawl anything themselves; they tell an existing search crawler whether its pages may be used for AI. Blocking them leaves search crawling untouched, which is why they are blocked at about the same rate as the training crawlers rather than as rarely as the search agents.

Who blocks #

LaneHomepages measuredBlock any AI crawlerBlock GPTBot
Everything else (includes publishers and media)89,7808.1%5.0%
SaaS and software19,9824.0%1.8%
US local businesses20,0422.3%0.8%
Government and education9,9842.1%1.4%

Blocking concentrates where content is the product. The everything-else lane, which holds most publishers, blocks at four times the rate of government and local-business sites. A local business has every reason to be quoted and almost no content an AI company would pay for, and its robots.txt reflects that, mostly because nobody changed the default.

Bot blocks that are not AI blocks #

Many robots.txt files block bots that have nothing to do with AI: SEO crawlers, archive crawlers and regional search engines. 15.4% of measured sites disallow at least one named crawler of any kind, more than twice the AI rate. Some of that is a platform default rather than a choice: on one large website builder, 94% of sites block the same regional search crawler and almost none block an AI crawler. Counting every bot block as an AI block would more than double the figure, which is why the 6.3% above counts only AI crawlers.

What blocking does and does not do #

A disallow in robots.txt is a request. The major AI crawlers say they honour it, and the ones that publish the addresses they crawl from can be checked; the robots.txt testing guide shows how to confirm a rule reads the way you intended. It does not remove anything already collected, and it does not stop a crawler that ignores robots.txt or claims another name. If you block a crawler for some paths rather than all of them, its group has to repeat every wildcard rule you still want applied, as the guide to user-agent groups explains.

For the rest of the October figures, from llms.txt adoption to how much of a homepage is text, see the full dataset report.

CrawlCheck’s dataset when this page loaded: 582,334 sites measured.

Check your own site

  1. What your robots.txt tells each crawler. Paste your domain into the robots.txt check: it reads the file the way each named AI crawler does, at the path you choose, and says allowed or blocked per agent. A crawler named in its own group ignores your * rules, which is the most common surprise.
  2. What the server does at the door. The file is a request; the server decides. The agent lens fetches your homepage as a browser and as ten named AI crawlers side by side, so a firewall that refuses a crawler your robots.txt allows shows up as a row that differs.
  3. Everything else the study measured. A free scan reports the rest on your own site: llms.txt, sitemap, how much of the homepage is text, HSTS, and how you sit against the sites in this study.

If the file and the door disagree, decide which one you meant and change the other. Most sites in these studies never chose; they inherited a default.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works