Agent view · Findings · 2026-10-06
Only 6% of websites block an AI crawler: the full breakdown
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/blog/only-six-percent-of-websites-block-an-ai-crawler: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/blog/only-six-percent-of-websites-block-an-ai-crawler>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 14,758 |
| words | 2,071 |
| sha-256 | c23055984c541f7a2b393fd91c3743f12bbf64deae8fcc4d38eed52c2b0bc2c9 |
| repr-digest | sha-256=:wjBVmExUH3orOT/ZHDdD8Su/ZN6uj8xNOO7VLCsLwsk=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311497;expires=1795315097;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="h88olutx46h54E9Zf36Bcg";tag="crawlcheck-page" |
| signature | sig1=:mEH2PWu+k2h6nUd6qavVQEMHkYGob+Fb6nVA6NBOGRGRnAxBZdrJZdR6BAMKimKMnobCk63RVpt/3GRdOiWrDw==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Exclusive finding · Findings · 2026-10-06 · By [VSNARY | Emmanuel Orta](/author) · 1 view # Only 6% of websites block an AI crawler: the full breakdown Which AI crawlers 139,788 sites disallow in robots.txt, how training and search agents are treated differently, and how blocking varies by kind of site. Across 139,788 homepages measured on 3-6 October 2026, 6.3% disallow at least one named AI crawler in robots.txt. GPTBot is blocked most (3.7%), then CCBot (3.1%), Bytespider (3.0%) and ClaudeBot (2.7%). Search and user-triggered agents are blocked far less: OAI-SearchBot 0.9%, Claude-User 0.5%. Publisher-heavy sites block most; government and local-business sites block least. CrawlCheck exclusive finding Sample robots.txt of 139,788 homepages, 12 named AI crawlers Measured 3 to 6 October 2026 What it established A per-crawler, per-sector block rate across the whole measured web, not a top-1,000 sample: GPTBot 3.7%, CCBot 3.1%, Bytespider 3.0%, ClaudeBot 2.7%; search and user-triggered agents under 1%. One of [23 findings measured only here](/blog?kind=exclusive). Every figure comes from CrawlCheck’s own readings, dated, with the method at the foot. Have your own? [Submit a finding](/submit-finding). Share of sites blocking each crawler in robots.txt. Gold: 139,788 homepages, 3–6 October 2026 ([all homepages measured](/blog/only-six-percent-of-websites-block-an-ai-crawler)). Teal: the top 10,000, census finished 24 October 2026, among the 4,501 that serve a robots.txt ([top-site census](/blog/which-ai-crawlers-top-sites-block-most-october-2026)). The most-visited sites block every crawler four to seven times as often, and the order barely changes: training crawlers first, search and user-triggered agents last. The two bars use different bases (all homepages measured; top sites that serve a robots.txt), so read the gap as direction and size, not a precise multiple. On this page - Crawler by crawler - Training is blocked more than search - Who blocks - Bot blocks that are not AI blocks - What blocking does and does not do How many websites actually block AI crawlers? In the CrawlCheck dataset for 3-6 October 2026, the answer is 6.3%: 8,757 of 139,788 measured homepages disallow at least one named AI crawler in [robots.txt](/glossary/robots-txt). Every other site lets them all in, by choice or by default. The figures below count each domain once, at its latest reading, and name no site. ## Crawler by crawler # | Crawler (user-agent) | What it is for | Sites that block it | | --- | --- | --- | | GPTBot | Collects pages for model training | 3.7% | | [CCBot](/glossary/ccbot) | Builds an open web crawl widely used for training | 3.1% | | [Bytespider](/glossary/bytespider) | Collects pages for model training | 3.0% | | ClaudeBot | Collects pages for model training | 2.7% | | [Google-Extended](/glossary/google-extended) | A control [token](/glossary/token) for AI use of pages Google already crawls | 2.0% | | ChatGPT-User | Fetches a page when a user asks for it | 1.8% | | anthropic-ai | An older token for the same purpose as ClaudeBot | 1.8% | | Applebot-Extended | A control token for AI use of pages Apple already crawls | 1.8% | | meta-externalagent | Collects pages for AI use | 1.7% | | PerplexityBot | Indexes pages for an [answer engine](/glossary/answer-engine) | 1.6% | | OAI-SearchBot | Indexes pages for search answers | 0.9% | | Claude-User | Fetches a page when a user asks for it | 0.5% | The [AI crawlers page](/ai-crawlers) lists each of these with what it does and whether its operator publishes the addresses it crawls from. ## Training is blocked more than search # The pattern in the table is consistent: crawlers that collect pages for training are blocked two to five times as often as the agents that fetch a page to answer a question or build a search index. 3.1% of sites block GPTBot while leaving OAI-SearchBot open, which is the coherent version of an AI policy: stay out of the [training data](/glossary/training-data), stay in the answers. Blocking the search or user agents is what removes a site from the answers, and very few sites do it. The control tokens are a separate case. Google-Extended and Applebot-Extended do not crawl anything themselves; they tell an existing search crawler whether its pages may be used for AI. Blocking them leaves search crawling untouched, which is why they are blocked at about the same rate as the training crawlers rather than as rarely as the search agents. ## Who blocks # | Lane | Homepages measured | Block any AI crawler | Block GPTBot | | --- | --- | --- | --- | | Everything else (includes publishers and media) | 89,780 | 8.1% | 5.0% | | SaaS and software | 19,982 | 4.0% | 1.8% | | US local businesses | 20,042 | 2.3% | 0.8% | | Government and education | 9,984 | 2.1% | 1.4% | Blocking concentrates where content is the product. The everything-else lane, which holds most publishers, blocks at four times the rate of government and local-business sites. A local business has every reason to be quoted and almost no content an AI company would pay for, and its robots.txt reflects that, mostly because nobody changed the default. ## Bot blocks that are not AI blocks # Many robots.txt files block bots that have nothing to do with AI: SEO crawlers, archive crawlers and regional search engines. 15.4% of measured sites disallow at least one named crawler of any kind, more than twice the AI rate. Some of that is a platform default rather than a choice: on one large website builder, 94% of sites block the same regional search crawler and almost none block an AI crawler. Counting every bot block as an AI block would more than double the figure, which is why the 6.3% above counts only AI crawlers. ## What blocking does and does not do # A disallow in robots.txt is a request. The major AI crawlers say they honour it, and the ones that publish the addresses they crawl from can be checked; [the robots.txt testing guide](/guides/how-to-test-robots-txt-for-gptbot-and-claudebot) shows how to confirm a rule reads the way you intended. It does not remove anything already collected, and it does not stop a crawler that ignores robots.txt or claims another name. If you block a crawler for some paths rather than all of them, its group has to repeat every wildcard rule you still want applied, as [the guide to user-agent groups](/guides/robots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules) explains. For the rest of the October figures, from [llms.txt](/glossary/llms-txt) adoption to how much of a homepage is text, see [the full dataset report](/blog/state-of-ai-visibility-october-2026-139788-homepages). CrawlCheck’s dataset when this page loaded: 582,334 sites measured. ## Check your own site - **What your robots.txt tells each crawler.** Paste your domain into the [robots.txt check](/tools#robots): it reads the file the way each named AI crawler does, at the path you choose, and says allowed or blocked per agent. A crawler named in its own group ignores your `*` rules, which is the most common surprise. - **What the server does at the door.** The file is a request; the server decides. The [agent lens](/tools/agent-lens) fetches your homepage as a browser and as ten named AI crawlers side by side, so a firewall that refuses a crawler your robots.txt allows shows up as a row that differs. - **Everything else the study measured.** A [free scan](/) reports the rest on your own site: llms.txt, sitemap, how much of the homepage is text, HSTS, and how you sit against the sites in this study. If the file and the door disagree, decide which one you meant and change the other. Most sites in these studies never chose; they inherited a default. Written by [Emmanuel Orta](/author) · VSNARY I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post. [Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta) Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fonly-six-percent-of-websites-block-an-ai-crawler%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fonly-six-percent-of-websites-block-an-ai-crawler%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Only%206%25%20of%20websites%20block%20an%20AI%20crawler%3A%20the%20full%20breakdown)[Email](mailto:?subject=Only%206%25%20of%20websites%20block%20an%20AI%20crawler%3A%20the%20full%20breakdown&body=https%3A%2F%2Fcrawlcheck.io%2Fblog%2Fonly-six-percent-of-websites-block-an-ai-crawler)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta) ## Related findings [2026-10-26 Which AI crawlers the top 10,000 sites block most, October 2026 Top 10,000 sites, October 2026: robots.txt block rates for 12 AI crawlers, how many refuse them at the door, and why training is b](/blog/which-ai-crawlers-top-sites-block-most-october-2026)[2026-10-26 Google-Extended and Applebot-Extended: block AI training Google-Extended and Applebot-Extended are tokens, not crawlers: they opt a site out of AI training and keep it in search. The line](/guides/google-extended-and-applebot-extended-opt-out-of-ai-training-keep-search)[2026-11-03 One Cloudflare setting is behind most of the new AI training blocks on top sites Census of the Tranco top 10,000, October to November 2026: which AI crawlers sites newly blocked, how much of it came from Cloudfl](/blog/one-cloudflare-setting-behind-most-new-ai-training-blocks-november-2026) The main product Found this on your own site? **We fix it for $100.** Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after. [Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v) [What it covers](/pricing#fix) · [scan first, it is free](/) Free browser extension Check this on any site you open, **without leaving the page.** CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account. [Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post) [What it does](/extension) Write for the library Measured something yourself? **Submit an article.** Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published. [Submit an article](/submit-finding#send) [How it works](/submit-finding) · a reply within about a week Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections). ## Comments Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*. Send for review Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan. [All findings](/blog) · [The dataset](/data) · [How the dataset works](/policy) We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io) #### How we measure, and our own machine files CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io). **Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing. **Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours. **Pages:** [/](/) · [Listing check](/listing-check) · [SDK](/sdk) · [Certificates](/certificates) | **Glossary:** [Entity-first content](/glossary/entity-first-content) · [Reduced motion](/glossary/reduced-motion) · [revocation delta](/glossary/revocation-delta) · [Sitemap directive](/glossary/sitemap-directive) | **Registry:** [beget.ru](/registry/beget.ru) · [dimmittchevrolet.com](/registry/dimmittchevrolet.com) · [infralyte.in](/registry/infralyte.in) · [ocaml.org](/registry/ocaml.org) [Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /blog/only-six-percent-of-websites-block-an-ai-crawler.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.