CrawlCheck

Guides · 2026-10-03 · By · 3 views

Screaming Frog vs Sitebulb vs CrawlCheck: AI crawler audits

Screaming Frog, Sitebulb, Semrush, Ahrefs and Lighthouse compared on the question that decides what each can find: what request it sends, and as whom. With where CrawlCheck falls short.

Screaming Frog, Sitebulb, Semrush and Ahrefs crawl your whole site for technical SEO issues, each as one identity: its own user-agent, from your IP or the vendor's cloud. Lighthouse and PageSpeed Insights render one page as a browser. None compares what a browser, GPTBot, ClaudeBot and PerplexityBot are each served, so a site that serves people and refuses AI crawlers passes all of them. CrawlCheck makes that comparison in a free scan but reads far fewer pages. Use a desktop crawler or suite for site-wide SEO and CrawlCheck for AI crawler access.

Check your own site free

About a minute. First scan free with your email, nothing installed.

Every audit tool produces a list of issues, and the lists overlap enough that comparisons usually come down to interface and price. The question that decides what a tool can find is narrower: what request does it send, and as whom? Screaming Frog, Sitebulb, Semrush and Ahrefs crawl as one identity. Lighthouse renders as a browser. None of them fetches the same page as a browser, GPTBot, ClaudeBot and PerplexityBot and compares the answers, and that comparison is how you find a site that serves people and refuses AI crawlers.

Disclosure: CrawlCheck publishes this comparison and is one of the products in it. Every price and feature claim about another company comes from that company’s own pricing or documentation page, read in October 2026; prices change, so the vendor page is the source of truth. Where CrawlCheck does less than a competitor, the tables say so.

What does each audit tool actually fetch? #

ToolWhat it requestsAs whomBlind toStarting price
Screaming Frog SEO SpiderEvery URL it can discover, from your desktop; renders JavaScript in headless Chrome; can run custom AI prompts during a crawlOne user-agent per crawl (presets for Googlebot, Bingbot and browsers, or a custom string), from your IPResponses that differ by crawler identity; edge rules that trust your IP£199 a year; free up to 500 URLs
SitebulbWhole site from desktop (Lite 10,000 URLs per audit; Pro 500,000, up to 2 million) or cloud; MCP access into Claude and ChatGPTOne user-agent per crawlResponses that differ by crawler identityCloud from £95/month; 14-day free trial
Semrush Site AuditWhole site from Semrush’s cloud, 140+ checks, with an AI Search Site Audit viewSemrush’s own crawlerWhat a named AI crawler is served$139/month (Pro); 7-day trial
Ahrefs Site AuditWhole site from Ahrefs’ cloud, 170+ issues; Bot Analytics shows bots from your logs after the factAhrefs’ own crawlerWhat a named AI crawler is servedIncluded in Ahrefs Free; paid plans from $29/month
Lighthouse / PageSpeed Insights (Google)One page, rendered, plus field data from real Chrome usersA browserEverything a non-browser client is servedFree
CrawlCheckHomepage and machine files (robots.txt, sitemap, llms.txt, entity data), fetched from outside; deep crawls to 500 pages on the API, 5,000 on GrowthA browser and several named AI and search crawlers in one scan, compared against each otherRanking, traffic, backlinks and site-wide on-page issuesFirst scan free, then $3 a scan; Watch $29/month for 3 domains; API from $99/month

Sources: Screaming Frog pricing and configuration guide; Sitebulb pricing; Semrush’s SEO Toolkit pricing page; Ahrefs’ pricing and Site Audit pages; About PageSpeed Insights; CrawlCheck pricing.

One identity per crawlSeveral identities, comparedAudit toolone user-agent200, full pagenothing to compareReport: site looks fineBrowser200, full pageGPTBot200, full pageClaudeBot403Difference = the findingSame URL, fetched close together in time. Only the comparison showsthat one crawler was refused while the others were served.
Why a deep single-identity crawl can pass a site that refuses an AI crawler. The 403 is an illustration, not a measured site.

What do all of them miss? #

Access failures are differences: the same URL answered one way for a browser and another way for GPTBot. A crawl that uses a single identity sees one side and has nothing to compare it with. On one site we scanned, GPTBot and ClaudeBot received 502 while Googlebot and PerplexityBot were served normally, with robots.txt allowing all four. A desktop crawl, a cloud audit and a Lighthouse run would each have reported that site as healthy. These are the problems that depend on the comparison, with how often each turns up across CrawlCheck’s scans:

ProblemWhy a single-identity audit misses itShare of scanned sites (live)
An answer engine’s crawler refused while a browser was servedThe audit is the browser, or a crawler the rule lets through5.8%
A named crawler served materially less text than a browserOnly one response size is ever seen1.1%
A challenge page served with status 200A 200 with HTML reads as success0.2%
A stale copy served from the edge cacheTools add cache-busting parameters or hit a warm, correct copy13.5%
Named AI groups in robots.txt drop the * rulesThe file is read as one crawler, so one group14%
Content that only exists after JavaScript runsBrowser-based tools render it, so the page looks complete2.1%
Almost nothing a machine receives is readable textNot measured as a crawler receives it28.4%
Machine files that cost more to fetch than the pageMachine files are not followed as a crawler follows them22.7%
An AI opt-out signal already setRarely reported per crawler1.8%

The robots.txt row is the one site owners most often cause themselves. A file can allow Googlebot everything and, through a group added for GPTBot further down, drop every Disallow rule for that crawler. See why naming a crawler drops your Disallow rules. And a challenge page returned with status 200 and HTML is parsed as no directives, everything allowed, which most uptime monitors and validators will call healthy.

Can I just set Screaming Frog’s user-agent to GPTBot? #

You can, and it is a useful spot check. A crawl with GPTBot’s string shows what your server returns to that name. Three things limit what it proves. It tests one identity per crawl, so comparing GPTBot with a browser means two crawls and a manual diff. It runs from your own IP address, which your CDN may trust differently from a data centre. And many bot-management rules key on more than the user-agent string. It does not replace a side-by-side comparison of every crawler that matters to you.

How do I check what an AI crawler receives myself? #

Fetch the page as a browser and as GPTBot, and compare the status and size:

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/128 Safari/537.36" https://example.com/

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" https://example.com/

A 403 or 429 on the second line, or a response a fraction of the size of the first, means the crawler is treated differently. Then turn JavaScript off in your browser and reload: whatever disappears is what most AI crawlers never see (do AI crawlers render JavaScript). Two caveats: a user-agent string is not the real crawler, so a firewall that checks published IP ranges may treat your test and the real GPTBot differently; and one fetch is one moment, because challenges and rate limits come and go.

Do the AI features in these tools check AI crawler access? #

No. “AI website audit” is sold as one thing and means two. In Semrush, Ahrefs, Screaming Frog and Sitebulb, the AI sits inside the audit: Semrush’s AI Search Site Audit view flags issues it associates with visibility in LLMs, Screaming Frog runs custom prompts through OpenAI, Gemini, Ollama or Anthropic as it crawls, Sitebulb puts its audits in front of Claude and ChatGPT over MCP, and Ahrefs’ AI Content Helper is a separate writing tool. All useful; none changes who the crawl arrives as. A firewall rule that challenges AI crawlers and lets SEO crawlers through never shows up in those reports. An audit of what AI receives has to fetch the site as the AI crawlers do.

Is a perfect Lighthouse score enough? #

It measures a different visitor. Lighthouse runs the page in Chrome, so a page whose text only appears after JavaScript scores normally while most AI crawlers receive the empty shell. Core Web Vitals do not apply to crawlers at all: a crawler does not paint, scroll or click, and it does not count toward Chrome field data. Lab scores also move between runs: we measured a 225% spread on two runs of the same unchanged page, so use field data where a page has the traffic and a median of several runs where it does not.

Lighthouse saysCrawlCheck saysWhat it usually means
Fast, 90+Crawler refusedA firewall or bot rule treats crawlers differently from browsers; speed is irrelevant until access is fixed
Fast, 90+Page is mostly codeThe text is built by JavaScript; server-render the main content
SlowCleanA people problem, not a machine one: fix images, scripts and server time for visitors
SlowCrawler served a challengeOften the same cause: a heavy security layer in front of the site

How do I turn one finding into a site-wide fix list? #

The tools work best in sequence. CrawlCheck finds the problem from the outside, on the pages an AI crawler reaches first; the desktop crawler then finds every page that shares the cause. The most common case is content that only exists after JavaScript runs:

  1. The CrawlCheck report flags the homepage: the HTML a crawler receives carries little of the text a browser shows.
  2. In Screaming Frog, enable JavaScript rendering and check the Contains JavaScript Content issue, which lists pages whose rendered text differs from the original HTML. In Sitebulb, the Response vs Render report makes the same comparison.
  3. Group the affected pages by template; the fix is usually one change to the template or the rendering setup, not page-by-page edits.
  4. Rescan with CrawlCheck to confirm AI crawlers now receive the text.

Why is a one-off audit not enough? #

Every tool above is sold as an audit: a report on a day. A site changes with every deploy, cache purge and plugin update, and measured continuously the gap is not small:

Citing this figure. The numbers above are recomputed from the live record when this page loads, so link the anchor rather than freezing a copy: https://crawlcheck.io/blog/your-site-was-clean-the-day-you-audited-it#the-gap. Sentence form, as of 2026-11-22: Among the 48 sites CrawlCheck measured daily between 2026-08-14 and 2026-11-22, 87.5% carried a machine-layer defect on the latest reading, while 100% carried one at least once — a gap of 12.5 points invisible to any single-day audit.

A one-off audit at any budget can only show the first figure; the second needs the same sites watched before the question was asked. The full measurement is in site audits go stale. CrawlCheck Watch re-measures automatically for $29 a month across three domains.

Which should I use? #

You areUseWhy
A small business with one siteA free CrawlCheck scan, plus Ahrefs Free or Screaming Frog’s free version for on-page issuesBoth cost nothing; one answers whether AI crawlers are served, the other lists what to fix on the page
An e-commerce or large siteScreaming Frog or Sitebulb, plus CrawlCheckThousands of product pages need a full crawl; the AI access check is per site and cheap to add
An agency with many clientsSemrush or Ahrefs for reporting; the CrawlCheck API for AI access across clientsThe suites report and white-label; the API runs the access check at scale
Anyone chasing speedPageSpeed InsightsField Core Web Vitals for real visitors, free

Where CrawlCheck is weaker #

  • No full-site crawl: link graphs, redirect chains across thousands of URLs, duplicate titles, custom extraction and visualisations are the desktop crawlers’ job. The free scan reads one homepage and the machine files around it; deep crawls top out at 500 or 5,000 pages by plan.
  • No keyword, backlink or rank data.
  • No field Core Web Vitals, accessibility audits at Lighthouse’s depth, or a performance waterfall.
  • It reads what the server sends before JavaScript, by design, and it sends each crawler’s published user-agent; it cannot send from the AI companies’ own IP addresses.

If you need a 100,000-URL audit, buy a desktop crawler or a suite. If you need to know whether ChatGPT, Claude and Perplexity are being served your site, that is the question none of them asks; run a free scan.

Share of scanned sites with each finding in this articlePage is mostly code28.4%Machine files behind a long redirect chain22.7%robots.txt rules overridden by a later group14%Stale cached copy served to crawlers13.5%Answer-engine crawler refused5.8%Content needs JavaScript to appear2.1%AI training opt-out set1.8%Crawler served less than a browser1.1%Share of scanned sites with eachfinding in this articlePage is mostly code28.4%Machine files behind a long redirect chain22.7%robots.txt rules overridden by a later group14%Stale cached copy served to crawlers13.5%Answer-engine crawler refused5.8%Content needs JavaScript to appear2.1%AI training opt-out set1.8%Crawler served less than a browser1.1%
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

Can Screaming Frog check if AI crawlers can read my site?

Partly. Screaming Frog lets you set a custom user-agent such as GPTBot's and crawl as that name, one identity per crawl, from your own IP address. It does not compare several AI crawler identities against a browser in one pass or read robots.txt separately for each AI crawler, so differences between what each crawler is served stay hidden.

Does Semrush or Ahrefs Site Audit check AI crawlers?

Not as the crawlers themselves. Both audits crawl from the vendor's cloud as the vendor's own bot. Semrush adds an AI Search Site Audit view and Ahrefs shows bots from your logs after the fact, but neither fetches your site as GPTBot or ClaudeBot, so a rule that refuses only AI crawlers is outside what they can see.

Does PageSpeed Insights or Lighthouse check AI crawlers?

No. PageSpeed Insights runs Lighthouse in a browser and shows field data from real Chrome users. It measures performance, accessibility, best practices and basic SEO for people. Its crawlability check looks for noindex tags and headers that block all crawlers, not robots.txt rules for individual crawlers such as GPTBot.

Can a site score 100 in Lighthouse and still be invisible to ChatGPT?

Yes. A firewall can refuse AI crawlers while serving browsers, or the content can exist only after JavaScript runs, which Lighthouse executes and most AI crawlers do not. Neither problem shows in the score.

Screaming Frog vs Sitebulb: which is better?

Both are mature technical SEO crawlers that render JavaScript and crawl a whole site deeply. Screaming Frog is a desktop licence with a free version for small sites; Sitebulb offers desktop and cloud plans with prioritised hints. The better choice depends on workflow, and neither is built to compare what different AI crawlers are served.

Do I need both a desktop crawler and CrawlCheck?

For most sites, yes. A desktop crawler or SEO suite covers site-wide technical SEO at scale; CrawlCheck covers whether AI crawlers are allowed, served and able to quote the site. They answer different questions, and CrawlCheck's first scan is free.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works