Findings · 2026-09-02 · By VSNARY | Emmanuel Orta · 0 views

The state of AI visibility, September 2026: what 1,364 scans say a crawler actually receives

Seven findings from the open dataset: the most common defect is a stale cache, a quarter of sites carry a grade-capping finding, opt-outs travel by copy-paste, and a third of crawler traffic is forged.

This is what the CrawlCheck dataset says about AI visibility on the web as of 2 September 2026. Every figure below is either on the open dataset page, which updates continuously, or is a threshold printed on a report row, where it came from a pass over 176 homepages. Nothing here is estimated, and no domain is named.

The short version: the defects that keep a site out of an AI answer are mostly infrastructure, not content. A stale cache, a missing content map, a page that ships as script, an opt-out nobody remembers setting. They are cheap to fix and almost nobody has measured them.

What was measured #

1,364 scans are counted in the dataset. Each scan fetches a homepage as a browser and as each named crawler, reads the machine files on both the apex and www hosts, resolves the site's robots.txt for every named agent, and parses the structured data. Findings are recorded by code; the dataset publishes how often each code occurs and nothing about which site carried it.

Finding 1: the most common defect is a cache, not a policy #

STALE_CACHE_SERVED is the most frequent finding in the dataset, on 25.0% of scanned sites (341 of 1,364). It means a crawler was served a copy older than the origin's own freshness window, so whatever the site fixed last is not yet what an engine reads. NO_LLMS_TXT is second at 20.5% (279 sites), and PAGE_IS_MOSTLY_CODE is third at 15.5% (211 sites): under 5% of the delivered bytes on those homepages is readable text.

Taken together, the three most common defects are all about delivery. None of them is fixed by writing better copy.

Finding 2: 23.5% of sites carry a high or critical finding #

Severity and frequency are independent. The most common finding is medium; the critical ones are rare. But 23.5% of scanned sites carry at least one high or critical finding, which is the level that caps a grade at D or F regardless of everything else on the page. ANSWER_ENGINE_REFUSED, where an answer engine's crawler is turned away at the edge while a browser gets 200, is on 3.7% of sites (50). UNIFORM_REFUSAL, every identity refused, is on 2.1% (29). ROBOTS_DISALLOW_ALL is on 1.9% (26).

Finding 3: AI opt-outs are inherited, not authored #

AI_OPTOUT_SET is on 11.4% of scanned sites (156). For a reference point outside our sample, BuiltWith's web-wide counts read on 12 August 2026 put AppleBot disallowed on 108,452 sites, GPTBot on 107,182, Common Crawl on 102,665 and ClaudeBot on 101,917. The eight largest opt-out groups sit within 12% of each other. Eight independent operators do not get blocked in near-lockstep by eight independent decisions; most of what looks like AI policy is one copied block, one plugin default or one hosting toggle.

ROBOTS_RULES_SHADOWED, on 4.3% of sites (58), is the version of this that surprises owners most: an allow-list added for one named agent silently voids the Disallow rules meant for everyone else. The file reads as a policy and behaves as its opposite.

Finding 4: the facts an assistant is asked for are the ones nobody declares #

Declaring an entity is common; declaring the specific facts an assistant needs is an order of magnitude rarer. BuiltWith counts 354,358 sites declaring an Organization, 108,730 declaring a PostalAddress, 91,622 a ContactPoint, 55,182 an Offer, and 22,972 an OpeningHoursSpecification. Fewer than one site in fifteen that declares an Organization declares opening hours. "Are they open?" and "how much is it?" are among the most common things anyone asks an assistant about a business, and the fields that answer them are the ones almost nobody publishes. Those are independent detection counts, not a joint distribution; nothing here divides one row by another.

Finding 5: most homepages are not written to be quoted #

In a pass over 176 homepages run on 2 September 2026, the median share of paragraphs an engine can lift whole (15 to 70 words, carrying a checkable fact) was 29%. The median share of sentences carrying a checkable fact was 20%. The median sentence ran 15 words. Only 30% of homepages opened with a sentence that defines what the site is; the rest opened with a slogan or a mood. 4% carried a visible date. The median share of headings carrying an id, the thing that lets a passage be cited by fragment, was 0%.

Those figures are why the quotable-content bar sits just under the corpus median: a typical site passes and a slogan-first site does not.

Finding 6: a third of crawler traffic is not who it says it is #

Across 3,415 checkable requests to sites we operate, 36.8% (1,257) named a known crawler and came from an address outside the range that crawler's operator publishes. The forged rate varies by name: Googlebot 8.9% of 740 claims, ClaudeBot 17.2% of 592, Applebot 34.6% of 581, PerplexityBot 38.9% of 298, GPTBot 44% of 389, OAI-SearchBot 55.4% of 202, ChatGPT-User 65% of 246. The names people forge most are the ones a site is most likely to allow.

Meta-ExternalAgent, Amazonbot, AhrefsBot, YandexBot and SemrushBot publish no address range, so their claims are unverifiable rather than forged, and are excluded from the rate. A user-agent is a claim, not an identity; every crawler figure on the dataset page states its denominator for that reason.

Finding 7: one crawl cannot see most of the problem #

Across 48 sites measured every day from 14 August to 2 September 2026, 77.1% carried a defect on the final day of that window. 95.8% carried one on at least one day in it. The gap is 18.7 points. Nine sites were clean on the final day and had not been on at least one earlier day. These are the figures from that window and they are not restated as current: a one-off audit at any budget can only produce the first number, and the second needs the same sites measured before the question was asked.

What the sample is #

The queue is widened on purpose rather than scanning the same kind of site repeatedly. Of the sites where the dimension could be read: rendering was server-side on 572 and client-side on 158; platform was WordPress on 99, Next.js on 74, Drupal on 13, Shopify on 11, Webflow on 9, Wix on 3, Squarespace on 3, and unknown on 518; language was English on 377 and other on 258. Publishers (136), SaaS (22), e-commerce (18) and government or education (15) are the largest identifiable business types. Small local services are under-represented at 6, which matters because they are the sites an assistant is most often asked about.

The three most common defects in the dataset #

The most common defect on the web is a cache, not a policy.

Live, as you read this: the corpus now holds 139,681 domains across 4,111 scans. The figures in this piece were measured on the date above; this line is not.

What to do with this #

In order of how often the data says it is the problem: check that a crawler gets the same fresh page a browser gets; publish an llms.txt and confirm it resolves; find out what share of your homepage is readable text; read your robots.txt as each named agent would; and rewrite the first paragraph so it says what you are. The free scan measures all five, the free tools resolve the robots file and draft the llms.txt, and the dataset page is where these numbers will keep moving.

The dataset page is regenerated from every scan, so the figures in this post are dated and the live ones will differ. That is the point of publishing them with a date.

Questions this post answers

Where do the numbers in this post come from?

Every count and share is on the open dataset page at /data, which is regenerated from every scan, or is a corpus threshold printed on the quotable-content rows of every report. No figure here was estimated or extrapolated, and no domain is named.

Why is a stale cache counted as an AI visibility defect?

Because a crawler served a copy older than the origin's freshness window reads the site as it was, not as it is. Whatever was fixed last is invisible until the cache turns over, and on a quarter of scanned sites that is the state a crawler meets.

Is forged crawler traffic dangerous or just noise?

Both. Some of it is scanners wearing a crawler's name to get past rules written for that crawler; some is tools and users. What matters for a site owner is that a firewall rule keyed on the user-agent string alone is keyed on a claim anyone can make, which is why the dataset only counts a request as verified when its address sits inside the range the operator publishes.

Will these figures change?

Yes. The dataset page updates continuously and the sample widens on purpose, so the live shares will drift from the ones dated here. The post carries its date for that reason.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works