CrawlCheck

Exclusive finding · Findings · 2026-08-19 · By · 5 views

AI visibility vendors audited: 11 of 23 have llms.txt

The companies that grade machine readability, run through the same checks they would point at you. The reading is from a directory that re-scans itself, so today's figures are one click away.

On 9 September 2026, the CrawlCheck directory listed 25 SEO and AI-visibility tools and graded 23 of them. Eleven of the 23 published an llms.txt and none published an entity map. Scores ran from 44 to 84, with a median of 78, and two vendors whose edge refused the scan are listed as refused rather than graded. An earlier one-off run of 33 vendor sites in August found the same pattern: roughly a third had an llms.txt, none had an entity map.

Every tool in this list sells a judgement about other people's websites. So we pointed CrawlCheck at them — the same crawler-identity, machine-file, cache and render checks a customer gets — and kept the results in the directory, which re-scans each site on a rolling basis and keeps a dated ledger of what moved. The figures below are its reading on 9 September 2026; open the directory for today's.

Disclosure: CrawlCheck publishes this piece and sells a tool in the same category. crawlcheck.io is not in its own corpus, because a scanner cannot grade its own host from the outside view it uses for everyone else; check our files from any client you like.

llms.txtEntity map11 of 230 of 23bar width = share of the 23 graded vendor sites
Machine files on the sites of SEO and AI-visibility vendors, directory reading of 9 September 2026.
Directory, 2026-09-09First run, 2026-08-19
Vendors25 listed, 23 graded33 scanned
Publish an llms.txt11 of 23roughly a third
Publish an entity map0 of 230
Highest score84 (Sitebulb)92
Lowest completed scan4441
Median78not published
Refused CrawlCheck2, listed as refused and not graded1

The two columns are different populations. The directory is curated to tools that sell a judgement about other sites; the August run was every vendor domain the crawl touched. Read them as two readings of a similar group, not one group changing over time.

Live, as you read this: the corpus now holds 582,334 domains across 9,304 scans. The figures in this piece were measured on the date above; this line is not.

What has not moved? #

The entity-map count. Zero of twenty-three, weeks after the first run, on the sites of companies whose product includes telling you to publish structured data about yourself. That is not a lag. If you are weighing whether these files matter, see does llms.txt work and what entity SEO is.

The refusals are now a category rather than an anecdote. A vendor whose edge challenges datacentre traffic is listed as refused CrawlCheck — not graded, with no number, because a number would be our vantage point dressed up as their defect. The first time we met one of these, we graded the firewall instead of the website.

How close are the grades? #

Close. Twenty of the twenty-three graded sites landed between 62 and 84, a few points apart, on a test each of them would put a customer through. The outliers say more than the middle: a site crawler at 44 with nine findings, and an enterprise SEO platform with zero findings and a 65, which is what a clean scan of a page that says almost nothing looks like.

Open door, little to quote #

Access and substance do not move together, and where they split is the useful part. Two AI-visibility monitors let every crawler in and have almost nothing on the page an answer engine could repeat: no stated number, no dated claim, no answered question. A site crawler lets crawlers in, but most of what arrives is markup and script rather than words. A single blended score hides both, which is why each directory card shows the evidence behind the number.

What the August run found #

The first run covered 33 vendor sites on 19 August 2026. The best scored 92 of 100 and the worst that completed a scan scored 41. The median site delivered under one visible word in twenty bytes of payload; the rest was markup and script, from companies that sell payload advice. Several served a cached machine file that no longer matched what their own server answered, so a crawler and a human could be told two different things. One site answered every path identically, including one that cannot exist: a firewall speaking, not a website.

How to read these figures fairly #

Each is a snapshot of the day the scan ran. A vendor who fixes their machine layer tomorrow deserves the better number, and the directory ledger records the change with its date. That is the point of keeping it as a page rather than a spreadsheet: a stale accusation is unfair to the vendor who fixed the thing.

The industry that grades machine readability is, on its own machine layer, about average. The average is low.

CrawlCheck’s dataset when this page loaded: 582,334 sites measured.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

Do AI-visibility vendors follow their own advice?

Mostly not on entity maps, and only about half on llms.txt. In the CrawlCheck directory reading, no graded vendor site published an entity map, and fewer than half published an llms.txt.

Is it a fair comparison?

Each vendor gets the same checks from the same client, no vendor is told in advance, and a site whose edge refuses the scan is listed as refused rather than given a score. The figures are dated snapshots and the directory re-scans them.

Why is a vendor listed as refused instead of graded?

Because when a firewall challenges datacentre traffic, the scan measures the firewall, not the website. Scoring that would present CrawlCheck's vantage point as the vendor's defect.

Is CrawlCheck itself in the directory?

No. A scanner cannot grade its own host from the same outside view it uses for other sites, so crawlcheck.io is left out and its machine files are open for anyone to check directly.

What does the result say about the checks?

That they are not specialist requirements only an expert can meet. They are cheap to implement, and the companies that sell the advice have mostly not implemented all of it.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All findings · The dataset · How the dataset works