Exclusive finding · Findings · 2026-10-06 · By VSNARY | Emmanuel Orta · 2 views
State of AI visibility, October 2026: 139,788 homepages
The open dataset's first full census: how many sites block AI crawlers, how many publish llms.txt, how little of a homepage is text, and what one audit cannot see.
Across 139,788 homepages measured between 3 and 6 October 2026, 6.3% block at least one named AI crawler in robots.txt and 15.4% publish an llms.txt file. The median homepage is 3.6% visible text by bytes, 61.5% are under 5%, 28.2% have no sitemap a crawler can find, and 48.4% send an HSTS header. Government and education sites trail on llms.txt, sitemaps, speed and grades.
Between 3 and 6 October 2026 the CrawlCheck dataset read 208,295 distinct domains across four lanes: SaaS and software, government and education, US local businesses, and everything else the crawl reached. 139,788 answered well enough to be measured end to end; the rest refused, timed out or never served a homepage, and they are left out of every figure below rather than counted as failures. Where a site was read more than once, only its latest reading counts. Nothing here is estimated and no site is named.
This report replaces the September edition, which worked from 1,364 deep scans. The findings from that edition that a larger, lighter census cannot measure are kept below with their own dates; its defect shares are replaced by the live rates, which keep moving.
How do the four lanes compare? #
| Measure | SaaS and software | Government and education | US local businesses | Everything else | All |
|---|---|---|---|---|---|
| Homepages measured | 19,982 | 9,984 | 20,042 | 89,780 | 139,788 |
| Block at least one AI crawler | 4.0% | 2.1% | 2.3% | 8.1% | 6.3% |
| Publish llms.txt | 28.2% | 5.5% | 22.6% | 12.0% | 15.4% |
| Send an HSTS header | 54.8% | 46.1% | 43.3% | 48.3% | 48.4% |
| No sitemap a crawler can find | 27.2% | 33.5% | 13.5% | 31.1% | 28.2% |
| Homepage under 5% visible text | 60.0% | 58.5% | 72.6% | 59.7% | 61.5% |
| Median time to first byte | 221 ms | 418 ms | 209 ms | 403 ms | 333 ms |
| Grade A | 25.5% | 9.9% | 27.2% | 15.9% | 18.5% |
How many websites block AI crawlers? #
For all the attention it gets, an explicit AI-crawler block is uncommon. 6.3% of measured homepages name at least one AI crawler in robots.txt and disallow it. GPTBot is the most blocked at 3.7%, followed by CCBot (3.1%), Bytespider (3.0%) and ClaudeBot (2.7%). The breakdown by crawler, including the split between training and search agents, is in the companion post.
Government and local-business sites block least (2.1% and 2.3%); the broad everything-else lane, which includes publishers and media, blocks most at 8.1%. A small business that has never touched its robots.txt is almost certainly open to every AI crawler, whether it meant to be or not.
Are AI opt-outs chosen or copied? #
Mostly copied. Web-wide counts from BuiltWith, read on 12 August 2026, put the largest opt-out groups within 12% of each other. Independent operators do not get blocked in near-lockstep by independent decisions; most of what looks like AI policy is one copied block, one plugin default or one hosting toggle.
| Crawler disallowed | Sites (BuiltWith, 12 August 2026) |
|---|---|
| Applebot | 108,452 |
| GPTBot | 107,182 |
| Common Crawl | 102,665 |
| ClaudeBot | 101,917 |
The copied version that surprises owners most is an allow line added for one named crawler, which silently cancels the Disallow rules meant for everyone else. The user-agent groups guide shows how to spot it. We withdrew an earlier count of this defect after finding it overcounted sites that block a crawler outright; the corrected live rate is: 14%.
How many websites publish llms.txt? #
15.4% of measured sites publish an llms.txt file. SaaS sites lead at 28.2%, and US local businesses are close behind at 22.6%, which is not because small businesses are early adopters: much of that share arrives through website builders that publish the file by default. Government and education sites trail at 5.5%. A file that exists by default says nothing about whether anyone wrote it, so presence is a weak signal on its own. What matters is whether the links in it resolve; the llms.txt guide shows how to check.
How much of a homepage is text? #
The median homepage is 3.6% visible text by bytes. 61.5% are under 5%, 30.3% are under 2%, and only 15.6% reach 10%. A crawler pays for every byte it downloads and can only quote the text, so a homepage that is 97% markup, styles and scripts gives an answer engine very little for the size of the download.
Client-side rendering makes it worse. 17.1% of homepages build their content in the browser, and their median text share is 0.3%, against 4.6% for pages rendered on the server. A crawler that does not run scripts sees almost nothing on a client-rendered page. Local-business sites are the heaviest: 72.6% are under 5% text and the median homepage weighs 151 KB. The text-to-HTML guide covers the usual causes.
Are homepages written to be quoted? #
A smaller, deeper pass over 176 homepages on 2 September 2026 read the text itself. Most homepages open with a slogan rather than a sentence that says what the site is, and almost none carry the dates and anchors that make a passage citable.
| Measure (176 homepages, 2 September 2026) | Median or share |
|---|---|
| Paragraphs an engine can lift whole that carry a checkable fact | 29% |
| Sentences carrying a checkable fact | 20% |
| Median sentence length | 15 words |
| Homepages that open by defining what the site is | 30% |
| Homepages carrying a visible date | 4% |
| Headings carrying an id (median share) | 0% |
Do sites publish the facts assistants are asked for? #
Rarely. Declaring an entity is common; declaring the facts an assistant needs is an order of magnitude rarer. BuiltWith counts 354,358 sites declaring an Organization, 108,730 a PostalAddress, 91,622 a ContactPoint, 55,182 an Offer and 22,972 an OpeningHoursSpecification. These are independent detection counts, not a joint distribution, but the gap is plain: “are they open?” and “how much is it?” are among the most common questions about a business, and the fields that answer them are the ones almost nobody publishes. The two facts almost nobody publishes covers how to add them.
Do crawlers find a sitemap, and is HSTS on? #
28.2% of measured sites have no sitemap a crawler can find, neither declared in robots.txt nor at the standard locations, and 29.9% do not declare one in robots.txt at all. Government and education sites are worst at 33.5%; local businesses do best at 13.5%, most likely because the same builders that add llms.txt also generate a sitemap. 12.9% serve the same homepage on two hosts without redirecting one to the other, which splits whatever a crawler learns about the site between two addresses; on government sites it is 23%. The sitemap audit guide covers both.
95.6% of measured homepages answer over HTTPS, but only 48.4% send a Strict-Transport-Security header, so on the rest a visitor's first request can still be downgraded. It is one header line.
How fast and heavy are homepages? #
The median time to first byte is 333 ms, and 21.5% of sites take longer than a second. Government sites and the everything-else lane are about twice as slow as SaaS and local-business sites (418 ms and 403 ms against 221 ms and 209 ms). The median homepage is 120 KB, and 11.7% are over 500 KB.
Which defects are most common right now? #
The defects that keep a site out of an AI answer are mostly about delivery, not content. In the deep-scan dataset the most common is a crawler being served an older cached copy than the site now carries, so whatever was fixed last is not yet what an engine reads. The live rates for the leading defects: 13.5% (live share of scanned sites: 34.1%) (live share of scanned sites: 28.4%) (live share of scanned sites: 5.8%).
None of these is fixed by writing better copy. The caching guide shows how to check what a crawler is actually served.
How much crawler traffic is forged? #
A user-agent is a claim, not an identity. On the sites we operate, every request naming a known crawler is checked against the address ranges its operator publishes:
Citing this figure. The numbers above are recomputed from the live record when this page loads, so link the anchor rather than freezing a copy: https://crawlcheck.io/blog/how-much-ai-crawler-traffic-is-forged#the-rate. Sentence form, as of 2026-11-22: 11.6% of requests claiming a verifiable AI-crawler identity, measured by CrawlCheck on its own traffic (25509 of 220753 adjudicable claims), came from outside the operator's published IP ranges; the most-forged name was Google-Extended at 100%.
The names forged most are the ones a site is most likely to allow, so a firewall rule keyed on the user-agent string alone is keyed on a claim anyone can make. The forged-traffic study has the per-crawler detail.
Why does one audit miss most problems? #
Across 48 sites measured every day from 14 August to 2 September 2026, 77.1% carried a defect on the final day; 95.8% carried one on at least one day. Nine sites were clean on the final day and had not been earlier. A one-off audit can only produce the first number. The second needs the same sites measured before anyone asked.
Check your own site #
Live, as you read this: the corpus now holds 582,334 domains across 9,304 scans. The figures in this piece were measured on the date above; this line is not.
Every figure above is a share of sites. Your own site is either in the majority or it is not, and a free scan answers that in under a minute. The live figures behind this report are on the open dataset page.
Check your own site
- What your robots.txt tells each crawler. Paste your domain into the robots.txt check: it reads the file the way each named AI crawler does, at the path you choose, and says allowed or blocked per agent. A crawler named in its own group ignores your
*rules, which is the most common surprise. - What the server does at the door. The file is a request; the server decides. The agent lens fetches your homepage as a browser and as ten named AI crawlers side by side, so a firewall that refuses a crawler your robots.txt allows shows up as a row that differs.
- Everything else the study measured. A free scan reports the rest on your own site: llms.txt, sitemap, how much of the homepage is text, HSTS, and how you sit against the sites in this study.
If the file and the door disagree, decide which one you meant and change the other. Most sites in these studies never chose; they inherited a default.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
How many websites block AI crawlers?
In the CrawlCheck census of 3 to 6 October 2026, 6.3% of 139,788 measured homepages block at least one named AI crawler in robots.txt. GPTBot is the most blocked at 3.7%.
How many websites have an llms.txt file?
15.4% of the measured homepages publish llms.txt. SaaS sites lead at 28.2%, local businesses follow at 22.6%, largely through website-builder defaults, and government and education sites trail at 5.5%.
How much of a typical homepage is readable text?
The median homepage is 3.6% visible text by bytes. 61.5% are under 5% and only 15.6% reach 10%. Pages that build their content in the browser have a median text share of 0.3%.
How many websites have no sitemap?
28.2% of measured sites have no sitemap a crawler can find, either declared in robots.txt or at the standard locations. Government and education sites are worst at 33.5%.
Why are AI opt-outs described as copied rather than chosen?
Web-wide counts put the most-blocked AI crawlers within a few per cent of each other. Independent decisions would not move in lockstep; one copied robots.txt block, a plugin default or a hosting toggle would.
How was the census data de-duplicated?
Each domain counts once, at its latest reading. 208,295 distinct domains were read; the 139,788 that could be measured end to end are the base for every percentage.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.