CrawlCheck

Exclusive finding · Findings · 2026-10-06 · By · 2 views

State of AI visibility, October 2026: 139,788 homepages

The open dataset's first full census: how many sites block AI crawlers, how many publish llms.txt, how little of a homepage is text, and what one audit cannot see.

Across 139,788 homepages measured between 3 and 6 October 2026, 6.3% block at least one named AI crawler in robots.txt and 15.4% publish an llms.txt file. The median homepage is 3.6% visible text by bytes, 61.5% are under 5%, 28.2% have no sitemap a crawler can find, and 48.4% send an HSTS header. Government and education sites trail on llms.txt, sitemaps, speed and grades.

Between 3 and 6 October 2026 the CrawlCheck dataset read 208,295 distinct domains across four lanes: SaaS and software, government and education, US local businesses, and everything else the crawl reached. 139,788 answered well enough to be measured end to end; the rest refused, timed out or never served a homepage, and they are left out of every figure below rather than counted as failures. Where a site was read more than once, only its latest reading counts. Nothing here is estimated and no site is named.

This report replaces the September edition, which worked from 1,364 deep scans. The findings from that edition that a larger, lighter census cannot measure are kept below with their own dates; its defect shares are replaced by the live rates, which keep moving.

How do the four lanes compare? #

MeasureSaaS and softwareGovernment and educationUS local businessesEverything elseAll
Homepages measured19,9829,98420,04289,780139,788
Block at least one AI crawler4.0%2.1%2.3%8.1%6.3%
Publish llms.txt28.2%5.5%22.6%12.0%15.4%
Send an HSTS header54.8%46.1%43.3%48.3%48.4%
No sitemap a crawler can find27.2%33.5%13.5%31.1%28.2%
Homepage under 5% visible text60.0%58.5%72.6%59.7%61.5%
Median time to first byte221 ms418 ms209 ms403 ms333 ms
Grade A25.5%9.9%27.2%15.9%18.5%

How many websites block AI crawlers? #

For all the attention it gets, an explicit AI-crawler block is uncommon. 6.3% of measured homepages name at least one AI crawler in robots.txt and disallow it. GPTBot is the most blocked at 3.7%, followed by CCBot (3.1%), Bytespider (3.0%) and ClaudeBot (2.7%). The breakdown by crawler, including the split between training and search agents, is in the companion post.

Government and local-business sites block least (2.1% and 2.3%); the broad everything-else lane, which includes publishers and media, blocks most at 8.1%. A small business that has never touched its robots.txt is almost certainly open to every AI crawler, whether it meant to be or not.

Are AI opt-outs chosen or copied? #

Mostly copied. Web-wide counts from BuiltWith, read on 12 August 2026, put the largest opt-out groups within 12% of each other. Independent operators do not get blocked in near-lockstep by independent decisions; most of what looks like AI policy is one copied block, one plugin default or one hosting toggle.

Crawler disallowedSites (BuiltWith, 12 August 2026)
Applebot108,452
GPTBot107,182
Common Crawl102,665
ClaudeBot101,917

The copied version that surprises owners most is an allow line added for one named crawler, which silently cancels the Disallow rules meant for everyone else. The user-agent groups guide shows how to spot it. We withdrew an earlier count of this defect after finding it overcounted sites that block a crawler outright; the corrected live rate is: 14%.

How many websites publish llms.txt? #

SaaS and software28.2%US local businesses22.6%All sites15.4%Everything else12.0%Government and education5.5%
Share of measured homepages that publish llms.txt, by lane, 3–6 October 2026. Builder defaults explain much of the local-business share.

15.4% of measured sites publish an llms.txt file. SaaS sites lead at 28.2%, and US local businesses are close behind at 22.6%, which is not because small businesses are early adopters: much of that share arrives through website builders that publish the file by default. Government and education sites trail at 5.5%. A file that exists by default says nothing about whether anyone wrote it, so presence is a weak signal on its own. What matters is whether the links in it resolve; the llms.txt guide shows how to check.

How much of a homepage is text? #

The median homepage is 3.6% visible text by bytes. 61.5% are under 5%, 30.3% are under 2%, and only 15.6% reach 10%. A crawler pays for every byte it downloads and can only quote the text, so a homepage that is 97% markup, styles and scripts gives an answer engine very little for the size of the download.

Client-side rendering makes it worse. 17.1% of homepages build their content in the browser, and their median text share is 0.3%, against 4.6% for pages rendered on the server. A crawler that does not run scripts sees almost nothing on a client-rendered page. Local-business sites are the heaviest: 72.6% are under 5% text and the median homepage weighs 151 KB. The text-to-HTML guide covers the usual causes.

Are homepages written to be quoted? #

A smaller, deeper pass over 176 homepages on 2 September 2026 read the text itself. Most homepages open with a slogan rather than a sentence that says what the site is, and almost none carry the dates and anchors that make a passage citable.

Measure (176 homepages, 2 September 2026)Median or share
Paragraphs an engine can lift whole that carry a checkable fact29%
Sentences carrying a checkable fact20%
Median sentence length15 words
Homepages that open by defining what the site is30%
Homepages carrying a visible date4%
Headings carrying an id (median share)0%

Do sites publish the facts assistants are asked for? #

Rarely. Declaring an entity is common; declaring the facts an assistant needs is an order of magnitude rarer. BuiltWith counts 354,358 sites declaring an Organization, 108,730 a PostalAddress, 91,622 a ContactPoint, 55,182 an Offer and 22,972 an OpeningHoursSpecification. These are independent detection counts, not a joint distribution, but the gap is plain: “are they open?” and “how much is it?” are among the most common questions about a business, and the fields that answer them are the ones almost nobody publishes. The two facts almost nobody publishes covers how to add them.

Do crawlers find a sitemap, and is HSTS on? #

28.2% of measured sites have no sitemap a crawler can find, neither declared in robots.txt nor at the standard locations, and 29.9% do not declare one in robots.txt at all. Government and education sites are worst at 33.5%; local businesses do best at 13.5%, most likely because the same builders that add llms.txt also generate a sitemap. 12.9% serve the same homepage on two hosts without redirecting one to the other, which splits whatever a crawler learns about the site between two addresses; on government sites it is 23%. The sitemap audit guide covers both.

95.6% of measured homepages answer over HTTPS, but only 48.4% send a Strict-Transport-Security header, so on the rest a visitor's first request can still be downgraded. It is one header line.

How fast and heavy are homepages? #

The median time to first byte is 333 ms, and 21.5% of sites take longer than a second. Government sites and the everything-else lane are about twice as slow as SaaS and local-business sites (418 ms and 403 ms against 221 ms and 209 ms). The median homepage is 120 KB, and 11.7% are over 500 KB.

Which defects are most common right now? #

The defects that keep a site out of an AI answer are mostly about delivery, not content. In the deep-scan dataset the most common is a crawler being served an older cached copy than the site now carries, so whatever was fixed last is not yet what an engine reads. The live rates for the leading defects: 13.5% (live share of scanned sites: 34.1%) (live share of scanned sites: 28.4%) (live share of scanned sites: 5.8%).

None of these is fixed by writing better copy. The caching guide shows how to check what a crawler is actually served.

How much crawler traffic is forged? #

A user-agent is a claim, not an identity. On the sites we operate, every request naming a known crawler is checked against the address ranges its operator publishes:

Citing this figure. The numbers above are recomputed from the live record when this page loads, so link the anchor rather than freezing a copy: https://crawlcheck.io/blog/how-much-ai-crawler-traffic-is-forged#the-rate. Sentence form, as of 2026-11-22: 11.6% of requests claiming a verifiable AI-crawler identity, measured by CrawlCheck on its own traffic (25509 of 220753 adjudicable claims), came from outside the operator's published IP ranges; the most-forged name was Google-Extended at 100%.

The names forged most are the ones a site is most likely to allow, so a firewall rule keyed on the user-agent string alone is keyed on a claim anyone can make. The forged-traffic study has the per-crawler detail.

Why does one audit miss most problems? #

Defect on the final day77.1%Defect on at least one day95.8%
48 sites measured every day, 14 August to 2 September 2026. A one-off audit can only see the top bar.

Across 48 sites measured every day from 14 August to 2 September 2026, 77.1% carried a defect on the final day; 95.8% carried one on at least one day. Nine sites were clean on the final day and had not been earlier. A one-off audit can only produce the first number. The second needs the same sites measured before anyone asked.

Check your own site #

Live, as you read this: the corpus now holds 582,334 domains across 9,304 scans. The figures in this piece were measured on the date above; this line is not.

Every figure above is a share of sites. Your own site is either in the majority or it is not, and a free scan answers that in under a minute. The live figures behind this report are on the open dataset page.

Share of scanned sites with each finding in this articleNo llms.txt34.1%Page is mostly code28.4%robots.txt rules overridden by a later group14%Stale cached copy served to crawlers13.5%Answer-engine crawler refused5.8%Share of scanned sites with eachfinding in this articleNo llms.txt34.1%Page is mostly code28.4%robots.txt rules overridden by a later group14%Stale cached copy served to crawlers13.5%Answer-engine crawler refused5.8%
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Check your own site

  1. What your robots.txt tells each crawler. Paste your domain into the robots.txt check: it reads the file the way each named AI crawler does, at the path you choose, and says allowed or blocked per agent. A crawler named in its own group ignores your * rules, which is the most common surprise.
  2. What the server does at the door. The file is a request; the server decides. The agent lens fetches your homepage as a browser and as ten named AI crawlers side by side, so a firewall that refuses a crawler your robots.txt allows shows up as a row that differs.
  3. Everything else the study measured. A free scan reports the rest on your own site: llms.txt, sitemap, how much of the homepage is text, HSTS, and how you sit against the sites in this study.

If the file and the door disagree, decide which one you meant and change the other. Most sites in these studies never chose; they inherited a default.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

How many websites block AI crawlers?

In the CrawlCheck census of 3 to 6 October 2026, 6.3% of 139,788 measured homepages block at least one named AI crawler in robots.txt. GPTBot is the most blocked at 3.7%.

How many websites have an llms.txt file?

15.4% of the measured homepages publish llms.txt. SaaS sites lead at 28.2%, local businesses follow at 22.6%, largely through website-builder defaults, and government and education sites trail at 5.5%.

How much of a typical homepage is readable text?

The median homepage is 3.6% visible text by bytes. 61.5% are under 5% and only 15.6% reach 10%. Pages that build their content in the browser have a median text share of 0.3%.

How many websites have no sitemap?

28.2% of measured sites have no sitemap a crawler can find, either declared in robots.txt or at the standard locations. Government and education sites are worst at 33.5%.

Why are AI opt-outs described as copied rather than chosen?

Web-wide counts put the most-blocked AI crawlers within a few per cent of each other. Independent decisions would not move in lockstep; one copied robots.txt block, a plugin default or a hosting toggle would.

How was the census data de-duplicated?

Each domain counts once, at its latest reading. 208,295 distinct domains were read; the 139,788 that could be measured end to end are the base for every percentage.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works