Open dataset

What crawlers receive, measured across 1,629 scans

Every scan run through the checker is counted here. No domains are published — only how often each defect occurs. Updated continuously.

Bar length is how common a finding is; bar colour is how serious — green is low severity, red is critical. The two are independent: the most common defect here is medium, and the critical ones are rare.

How that compares

A share is only meaningful against a denominator. These are population counts for the same machine-layer signals, published by BuiltWith and read on 12 August 2026. They are counts of sites where the signal was detected anywhere on the web, not a sample of ours, and they are quoted here as reference points rather than reproduced as a dataset.

The eight largest opt-out groups sit within 12% of each other. Eight independent operators do not get blocked in near-lockstep by eight independent decisions, so most of what looks like AI policy on the web is one copy-pasted block, one plugin default or one hosting toggle — inherited rather than authored. That is the claim this dataset exists to test, one scan at a time.

Which structured facts actually exist

The same source counts detections of individual schema types. Declaring an entity is common; declaring the specific facts an assistant is asked for is an order of magnitude rarer.

Fewer than one site in fifteen that declares an Organization declares opening hours. “Are they open?” and “how much is it?” are among the most common things anyone asks an assistant about a business, and the fields that answer them are the ones almost nobody publishes.

These are independent detection counts, not a joint distribution. Each row is the number of sites where that type was found anywhere; nothing here says which sites overlap, and no coverage rate should be inferred by dividing one row by another. Counts read from BuiltWith on 13 August 2026.

What a single crawl cannot see

77.1% of measured sites carry a defect today. 95.8% carried one at least once between 2026-08-14 and 2026-09-11. 9 look clean now and did not for at least one day in that window.

Both numbers describe the same sites over 29 days, 2026-08-14 to 2026-09-11. A one-off crawl at any budget can only ever produce the first row — the second needs the same sites to have been measured before the question was asked. Sites added after the window opened are excluded (208) because they have had fewer chances to be seen broken. No scanned domain is named here or anywhere else on this site.

Arrivals across the network

measured on 3 sites — too few to describe as a population, so no rate is published

Forged crawler identities

23% of the requests that named themselves as a known crawler here were not that crawler. 1369 of 5940 checkable claims came from an address outside the range the operator publishes.

A user-agent is a claim, not an identity. Verified means the source IP sits inside a range the operator publishes — OpenAI, Anthropic, Google, Microsoft, Perplexity and Apple all publish one. Unverifiable is not forgery: some operators publish no range at all, so their requests can be neither confirmed nor accused, and they are excluded from the rate rather than counted against it. This is traffic to this site only, and this site is small — it is a floor on the problem, not a survey of the web.

What is in the sample

Coverage matters as much as the counts: a defect rate measured entirely on one platform is a fact about that platform. The scanner deliberately widens the sample rather than scanning the same kind of site repeatedly, and the queue below is what it has not reached yet.

Crawlers we have actually seen

Everything else on this page comes from probes — we send a request wearing a crawler’s name and record what comes back. This table is the opposite: visits nobody asked for, on 4 sites we operate, over 30 days. Verified means Cloudflare matched the request to the address ranges that crawler’s operator publishes.

How to read this, and how not to. 22601 visits across a handful of sites is a sample, not a census — it says what reached these sites, not what the web receives. And unverified is not proof of an impostor: plenty of legitimate traffic arrives from ranges nobody publishes, and our own testing appears in these counts as unverified because it is. What the column does show is that the name in a user-agent string is a claim, and it is checkable.

Platform, rendering, size and business type are read from what each site discloses in its own response. Most sites now answer from behind a CDN, which replaces the origin identity in the headers, so unknown here means the site did not disclose one — not that we did not look. We would rather publish a large unknown than a confident guess.

How to read these numbers

Each row counts sites, not pages, and a site is counted once per finding however many times it was scanned. A finding is recorded only when the check actually returned an answer: lookups that failed on our side are excluded rather than counted as clean, which is why the totals here can be smaller than the scan count. Domains are never named, in aggregate or individually, and any operator can exclude a domain from this dataset in one line of robots.txt — see the policy page.

Every figure on this page is recomputed when the page is requested, so a number quoted elsewhere is a snapshot of the moment it was read. The same counts are served as JSON at /api/public/counts with an at timestamp — no key, CORS open — so anyone quoting us can check the current value instead of taking ours. A record is only worth more than a single reading if the reading can be repeated.

Run a scan