The state of AI visibility, month by month.

What sites serve AI crawlers across the open dataset and the weekly census of the most-visited sites: refusals, machine files, what a crawler actually receives. Each month’s issue is written from the open dataset at /data and the weekly census at /data/census/weekly; every count links back to them.

Issues

• 139,788 homepages in four days: what AI crawlers get 2026-10-06
  139,788 homepages measured 3-6 October 2026: 6.3% block an AI crawler, 15.4% publish llms.txt, and the median homepage is 3.6% text.
• The state of AI visibility, September 2026: what 1,364 scans say a crawler actually receives 2026-09-02
  Seven findings from the CrawlCheck dataset as of 2 September 2026, every one sourced to the open data page or a published corpus threshold: which defects are most common, how often they cap a grade, why AI opt-outs move in lockstep, how few homepages are written to be quoted, and how much crawler traffic is not who it claims to be.

The data behind it

• The open dataset: every scan, regenerated, CC BY 4.0.
• The weekly census: the most-visited sites, this week against last.
• Datasets: the same measurements as lists.

Feed (Atom): /data/state-of-ai-visibility/feed.xml · every edition: /editions.