CrawlCheck

Tested 2 November 2026

The best AI audit tool on the web? We planted ten faults and counted.

In one sentence

This is CrawlCheck's blind test of free AI audit tools: a fake business site with ten planted faults in how AI crawlers see it (robots.txt, refusals, challenge pages, stripped pages, JavaScript-only content, llms.txt, JSON-LD, security.txt, sitemap), twelve tools run against it with default settings, and every result scored against a public answer key.

Most “top AI audit tools” lists compare feature pages. We built a fake business website with ten known problems in how AI crawlers see it, ran every machine-layer checker we could run for free without an account, and counted what each one found.

Read this first. CrawlCheck publishes this page, built the test site and chose the ten faults, so the test is not neutral by construction: it rewards fetching a page as several crawlers, which is how CrawlCheck works. To make it checkable, the site stays up at lab.crawlcheck.io and the answer key is public, so anyone can run any tool against it. CrawlCheck is scored on its first run. We fixed our own misses afterwards; that second run is reported separately and does not count.

Result

Faults caught, out of ten

A full catch counts 1 and a partial catch counts 0.5. “Wrong” means the tool told the user the opposite of the truth, which is worse than silence, so it is shown next to the score instead of hidden in it.

CrawlCheckfirst run, before any fix7.5
GEO Toolboxthree free checkers6
aicrawlercheck.comAI crawler access checker5.5
Swetrix AI Crawlability Checkerfree check3
Hostinger AI Visibility Checkerfree check32 wrong
Writesonic AI Crawl Checkerfree check2.53 wrong
SEOptimerfree one-page audit22 wrong
Google Rich Results Testsingle-purpose1
Schema.org Validatorsingle-purpose1
Seobilityfree SEO check1
isitagentready.comagent-readiness checker1
Knowatoa AI Search Consolefree check11 wrong

Fault by fault

What each tool saw

Caught Partly Missed✕Said the opposite–Outside its scope
ToolF1F2F3F4F5F6F7F8F9F10Score
CrawlCheckfirst run, before any fixCaughtCaughtCaughtCaughtCaughtPartlyMissedCaughtMissedCaught7.5/10
GEO Toolboxthree free checkersCaughtCaughtCaughtMissedCaughtMissedCaughtMissedMissedCaught6/10
aicrawlercheck.comAI crawler access checkerCaughtCaughtCaughtMissedPartlyMissedMissedCaughtMissedCaught5.5/10
Swetrix AI Crawlability Checkerfree checkCaughtCaughtMissedMissedMissedMissedMissedMissedMissedCaught3/10
Hostinger AI Visibility Checkerfree checkMissedPartlyWrong: said the oppositeMissedMissedWrong: said the oppositeCaughtPartlyMissedCaught3/10
Writesonic AI Crawl Checkerfree checkCaughtCaughtWrong: said the oppositeWrong: said the oppositeMissedWrong: said the oppositeMissedPartlyMissedMissed2.5/10
SEOptimerfree one-page auditWrong: said the oppositeCaughtMissedMissedMissedMissedMissedWrong: said the oppositeMissedCaught2/10
Google Rich Results Testsingle-purposeOutside this tool's scopeCaughtOutside this tool's scopeOutside this tool's scopeOutside this tool's scopeOutside this tool's scopeOutside this tool's scopeMissedOutside this tool's scopeOutside this tool's scope1/2
Schema.org Validatorsingle-purposeOutside this tool's scopeOutside this tool's scopeOutside this tool's scopeOutside this tool's scopeOutside this tool's scopeOutside this tool's scopeOutside this tool's scopeCaughtOutside this tool's scopeOutside this tool's scope1/1
Seobilityfree SEO checkMissedCaughtMissedMissedMissedMissedMissedMissedMissedMissed1/10
isitagentready.comagent-readiness checkerMissedMissedMissedMissedMissedMissedMissedMissedMissedCaught1/10
Knowatoa AI Search Consolefree checkMissedMissedCaughtWrong: said the oppositeMissedMissedMissedMissedMissedMissed1/10
  1. F1 · ClaudeBot is fully allowed. robots.txt blocks everyone with User-agent: *, but ClaudeBot has its own group holding only a Crawl-delay. Under RFC 9309 the * rules no longer apply to it, so ClaudeBot may crawl everything.
  2. F2 · Every unnamed crawler is blocked. The same Disallow: / shuts out ChatGPT-User, Claude-SearchBot, Googlebot, Bingbot and every crawler without its own group.
  3. F3 · PerplexityBot is refused at the server. robots.txt allows PerplexityBot, but the server answers its user agent with HTTP 403.
  4. F4 · Applebot gets a challenge page. robots.txt allows Applebot. The server answers 200, but the body is a “Just a moment…” browser check, not the page.
  5. F5 · OAI-SearchBot gets a stripped page. OAI-SearchBot receives HTTP 200 with about 60% of the text removed. Status codes alone look perfect.
  6. F6 · Prices exist only after JavaScript. The HTML says “Loading prices…”; an inline script writes the real prices. A crawler that does not run scripts never sees a price.
  7. F7 · llms.txt is malformed. The file exists but has no H1 title and no markdown links, so tools that parse the format skip it.
  8. F8 · The JSON-LD does not parse. One trailing comma makes the LocalBusiness block invalid JSON, so no parser reads it.
  9. F9 · security.txt has expired. Its Expires date is 1 January 2025. RFC 9116 says an expired file should not be trusted.
  10. F10 · The sitemap in robots.txt is missing. robots.txt points to /sitemap.xml, which returns 404.

The runners-up

Two tools came close, for different reasons

GEO Toolbox scored 6. Its free scanner does what most tools here skip: it fetches the page as six AI crawlers and compares what they get. That is how it caught the 403 and the stripped page (it reported OAI-SearchBot receiving only 39 words). Its separate llms.txt checker caught the malformed file. It missed what sits outside its six crawlers and its file list: the Applebot challenge page, the JavaScript-only prices, the broken JSON-LD and the expired security.txt.

aicrawlercheck.com scored 5.5. Its robots.txt engine is the broadest here: it evaluates 265 named bot tokens, where CrawlCheck evaluates 13 crawler identities, and its fixes are copy-paste ready. It fetches as 16 bots, but checks each response on its own instead of against a browser, so the stripped page passed all six of its checks. It never fetched as Applebot, and its file checks stop at “the file exists”.

Both are good tools. The difference is not how many things a tool looks at but whether each answer is right. The score counts only right answers.

Tool by tool

How each one checks, and where it slipped

CrawlCheck 7.5 of 10

How it checks. Fetches the page as 13 named crawlers plus a browser and a mobile control, compares what each one is served, applies robots.txt per crawler, then reads the machine files.

What it missed. Missed the malformed llms.txt and the expired security.txt (it checked only that each file existed). Flagged F6 only partly.

GEO Toolbox 6 of 10

How it checks. Its free Agent Readiness Scanner fetches the page as six AI crawlers, renders it in a browser and reads robots.txt for 34 bots. We also ran its free llms.txt Checker, as we combined aicrawlercheck.com's free checks.

What it missed. Applebot was not among the six crawlers it fetches as, so the challenge page passed. Its browser render did not report the JavaScript-only prices, its structured-data score did not say the JSON-LD fails to parse, and security.txt is not checked.

aicrawlercheck.com 5.5 of 10

How it checks. Evaluates robots.txt for 265 named bot tokens, then fetches as 16 of them through six gates: robots, directives, status, redirects, challenge page, text without JavaScript.

What it missed. Applebot was never fetched, so the challenge page passed. OAI-SearchBot's stripped page passed all six gates; the text count (198 against 832 characters) was shown but blamed on JavaScript. llms.txt passed as “published (9 words)”. For ClaudeBot it gave the right verdict but quoted another group's line (line 9, GPTBot's Allow).

Swetrix AI Crawlability Checker 3 of 10

How it checks. Reads robots.txt for 8 AI crawlers, checks that llms.txt and a sitemap exist, and counts structured data blocks.

What it missed. No crawler fetches, so every server-side fault is outside its view. llms.txt was reported as “Found” and the broken JSON-LD counted as one structured data block. For ClaudeBot the verdict was right but the matched rule shown was another group’s Allow line.

Hostinger AI Visibility Checker 3 of 10

How it checks. About 25 weighted checks in seven groups: crawl permission, sitemap, readability without JavaScript, structured data, contact and price legibility, llms.txt, agent protocols.

What it missed. Per-crawler results are not shown, so the ClaudeBot rule never surfaces. It reported the JSON-LD as missing rather than broken. Its llms.txt check did mark the file as badly formed.

Where it said the opposite of the truth:

  • F3: “Not blocked by bot detection”, full marks (PerplexityBot gets 403)
  • F6: “Prices are reachable without JavaScript or login”, full marks
Writesonic AI Crawl Checker 2.5 of 10

How it checks. Fetches once as GPTBot and once in a real browser, compares word counts, and gives a robots.txt verdict for 16 bots.

What it missed. The per-bot “can read” answers come from robots.txt alone, so the 403 and the challenge page both read as “yes”. JSON-LD was reported as absent rather than broken.

Where it said the opposite of the truth:

  • F3: said PerplexityBot can read the page (it gets 403)
  • F4: said Applebot can read the page (it gets a challenge)
  • F6: “nothing is waiting on JavaScript”
SEOptimer 2 of 10

How it checks. One-page SEO audit with a GEO section: robots.txt, llms.txt presence, schema presence, rendered-content share.

What it missed. It treats any mention of a crawler under the * block as blocked, ignoring named groups.

Where it said the opposite of the truth:

  • F1: listed ClaudeBot as blocked (and GPTBot, OAI-SearchBot and PerplexityBot, which robots.txt explicitly allows)
  • F8: “You are using JSON-LD Schema” and “LocalBusiness identified”, on JSON that does not parse
Google Rich Results Test 1 of 2

How it checks. Fetches one URL as Google-InspectionTool and tests its structured data for rich results.

What it missed. It reported “blocked by robots.txt” and could not test the markup, so the broken JSON-LD (its own subject) went unchecked. Pasting the code instead of the URL would have tested it.

Schema.org Validator 1 of 1

How it checks. Parses structured data on one page.

What it missed. Only F8 is in its scope, and it caught it exactly (JSON parse error at the trailing comma).

Seobility 1 of 10

How it checks. A general on-page SEO checker. Its crawler obeys robots.txt.

What it missed. It stopped at “The website is blocked by robots.txt” and audited nothing else. That is correct behaviour for a polite crawler, and it is also all a site owner learns.

isitagentready.com 1 of 10

How it checks. Checks for agent standards: MCP server card, API catalog, OAuth discovery, A2A, Content Signals, Markdown negotiation, sitemap.

What it missed. Its robots.txt check passes when AI rules exist, without evaluating them. It is built for agent standards, which this test barely covers. Cloudflare Radar’s URL Scanner runs the same agent-readiness checks; we ran it too, got the same result and did not count it twice.

Knowatoa AI Search Console 1 of 10

How it checks. Requests the page as 25 AI user agents and reports the HTTP status for each.

What it missed. Status codes only: robots.txt is not read, and a 200 challenge page counts as success.

Where it said the opposite of the truth:

  • F4: Applebot “success, 200 OK” on a challenge page

Where others are better

What CrawlCheck does not do as well

  • Breadth of bot names. aicrawlercheck.com evaluates robots.txt for 265 tokens and GEO Toolbox for 34; CrawlCheck for 13 crawler identities.
  • Real browser rendering. GEO Toolbox, Writesonic and Screaming Frog render the page in a browser. CrawlCheck reads the bytes the server sends and looks for script-filled placeholders; it does not run the scripts.
  • Real traffic. Cloudflare AI Crawl Control sees the crawlers that actually reach your site. CrawlCheck tests what a crawler would get.
  • Backlinks and keywords. Ahrefs, Semrush and Moz pair their crawls with their own link and keyword indexes. CrawlCheck crawls the whole site, but it has no backlink or keyword data.
  • Agent standards. isitagentready.com covers OAuth discovery, A2A and payment protocols that CrawlCheck does not.

Our own misses

CrawlCheck’s first run, and what we changed

First run: 7 caught, F6 partly, F7 and F9 missed. CrawlCheck saw that llms.txt and security.txt existed and stopped there. We changed three things on 2 November and ran it again:

  • llms.txt is now read for the format (an H1 title first, then markdown links).
  • security.txt is now read for its Contact and Expires fields.
  • An element delivered empty or as “Loading…” that an inline script fills is now reported.

Second run: 10 of 10. It is not a blind result, because we knew the faults when we made the changes, and it is not used in the ranking.

Not tested

Tools we could not run without an account

These are not ranked. The last column is how many of 14 machine-layer questions each tool’s own documentation answers, which says what a tool claims, not what it catches. CrawlCheck answers 14 by the same count; aicrawlercheck.com answers 7.5. Sorted from fewest to most. (LLM Pulse’s free robots.txt checker is left out: it only sends its report by email.)

ToolWhy not runBest atDocumented
Moz Pro Site CrawlFrom $99/moAccount requiredWeekly crawl hygiene next to Moz's link metrics.1.5/14
SE Ranking Website AuditFrom $129/moAccount required for the audit; our run of its free robots tester did not finishBroad technical audits at a low price, with an API and MCP included.4/14
Cloudflare AI Crawl ControlFree on CloudflareWatches real traffic on a domain proxied by Cloudflare; it does not test a site you nameShowing which AI crawlers actually hit your site and whether they obey robots.txt.4.5/14
ScrunchFrom $250/moPaid platformEnterprise AI-agent traffic analytics through CDN integrations.4.5/14
SitebulbFrom $18/mo14-day trial with registrationExplaining and prioritising issues, especially JavaScript rendering differences.5/14
Ahrefs Site AuditFrom $129/moAccount requiredPairing a site audit with Ahrefs' own backlink and keyword index, recrawled continuously.5/14
██████████Custom; self-serve tiers removed in September 2026Account requiredMeasuring real AI-crawler traffic from CDN and server logs, verifying each visit, alongside brand tracking across answer engines.5/14
LumarQuote onlyDemo onlyVery large crawls with monitoring and pre-release QA for big organisations.6/14
Conductor Website MonitoringQuote onlyTrial with a verified siteReal-time change tracking and alerts on large sites.6/14
Otterly.aiFrom $29/moResults held behind an email formCheap, high-volume single-page AI-readiness audits bundled with prompt tracking.6/14
Screaming Frog SEO Spider£199/yrDesktop download; the free version disables JavaScript rendering and structured dataDeep, configurable desktop crawling with exact control over requests and extraction.7/14
Semrush Site AuditFrom $139/moAccount requiredThe widest documented AI-search checks among the big suites: llms.txt plus robots.txt for 8 named bots.7/14

The withheld name

One of the leading companies in AI visibility got our own site wrong.

████████████Name withheld · vendor not yet contacted

This is not a small tool. It is one of the best-known names in AI search visibility, and companies use its numbers to decide where their money goes. On 1 October 2026 we typed crawlcheck.io into its free AI-visibility report. The report argued with itself.

0%What it reported after renaming us to a .com domain we do not own: cited in 0% of AI responses.
9%Where its own list of cited websites put crawlcheck.io on the same page: first, at 9%.

One page. Two numbers for the same brand. Only one of them measured the site we asked about.

A company this size should hear it from us before anyone else does. We are contacting them first. Until then the bar stays black, and the finding stays on this page.

A different kind of tool

AI visibility checkers answer another question

LeadSnap’s AI Visibility Checker and HubSpot’s AI Search Grader ask what AI engines say about a business, by prompting the engines. They do not inspect the website, so they are not scored here. If the question is “does ChatGPT recommend us in our city”, start with them; if it is “can the crawlers that feed ChatGPT read our site”, start with the tools above.

Run it yourself

How to check our work

Point any tool at https://lab.crawlcheck.io. The site is a fake gutter company and says so on the page. Every fault is listed above; the answer key is at /answer-key.json on the same site. Each tool here was run once, with its default settings, on the evening of 2 November 2026 (Mountain Time). Midway through, the site moved to this address from a temporary one; its content and faults did not change. If a tool has changed since, or we misread a result, send it to our corrections page and we will re-run it and date the change.

Your turn

You have the data and our opinion. What do you think?

Two questions, one vote each. Results appear as soon as you vote.

Which tool would you trust to check how AI crawlers see your site?
Was this test fair?

One vote per question per visitor. We publish the totals as they stand; we do not edit them.

Questions about this page

QWhat is the best AI audit tool?
In this test CrawlCheck caught the most faults (7.5 of 10 on its first run), then GEO Toolbox (6) and aicrawlercheck.com (5.5). CrawlCheck built the test, so the site and its answer key are public for anyone to re-run with any tool.
QHow were the tools tested?
Each free tool that works without an account was run once with its default settings against the same test site on 2 November 2026. A full catch scored 1 and a partial catch 0.5; answers that said the opposite of the truth are listed separately.
QWhy are some tools not ranked?
Screaming Frog, Sitebulb, Ahrefs, Semrush, Lumar, Conductor, SE Ranking, Moz, Cloudflare AI Crawl Control, Otterly and Scrunch need an account, a trial or a download, so they are listed by what their documentation covers and not ranked.
QWhy is one vendor's name blacked out?
We found a self-contradicting result in a leading AI-visibility vendor's free report and are contacting the company before naming it. The finding stays on the page.