Guides · 2026-10-03 · By VSNARY | Emmanuel Orta · 3 views
Screaming Frog vs Sitebulb vs CrawlCheck: AI crawler audits
Screaming Frog, Sitebulb, Semrush, Ahrefs and Lighthouse compared on the question that decides what each can find: what request it sends, and as whom. With where CrawlCheck falls short.
Screaming Frog, Sitebulb, Semrush and Ahrefs crawl your whole site for technical SEO issues, each as one identity: its own user-agent, from your IP or the vendor's cloud. Lighthouse and PageSpeed Insights render one page as a browser. None compares what a browser, GPTBot, ClaudeBot and PerplexityBot are each served, so a site that serves people and refuses AI crawlers passes all of them. CrawlCheck makes that comparison in a free scan but reads far fewer pages. Use a desktop crawler or suite for site-wide SEO and CrawlCheck for AI crawler access.
Every audit tool produces a list of issues, and the lists overlap enough that comparisons usually come down to interface and price. The question that decides what a tool can find is narrower: what request does it send, and as whom? Screaming Frog, Sitebulb, Semrush and Ahrefs crawl as one identity. Lighthouse renders as a browser. None of them fetches the same page as a browser, GPTBot, ClaudeBot and PerplexityBot and compares the answers, and that comparison is how you find a site that serves people and refuses AI crawlers.
Disclosure: CrawlCheck publishes this comparison and is one of the products in it. Every price and feature claim about another company comes from that company’s own pricing or documentation page, read in October 2026; prices change, so the vendor page is the source of truth. Where CrawlCheck does less than a competitor, the tables say so.
What does each audit tool actually fetch? #
| Tool | What it requests | As whom | Blind to | Starting price |
|---|---|---|---|---|
| Screaming Frog SEO Spider | Every URL it can discover, from your desktop; renders JavaScript in headless Chrome; can run custom AI prompts during a crawl | One user-agent per crawl (presets for Googlebot, Bingbot and browsers, or a custom string), from your IP | Responses that differ by crawler identity; edge rules that trust your IP | £199 a year; free up to 500 URLs |
| Sitebulb | Whole site from desktop (Lite 10,000 URLs per audit; Pro 500,000, up to 2 million) or cloud; MCP access into Claude and ChatGPT | One user-agent per crawl | Responses that differ by crawler identity | Cloud from £95/month; 14-day free trial |
| Semrush Site Audit | Whole site from Semrush’s cloud, 140+ checks, with an AI Search Site Audit view | Semrush’s own crawler | What a named AI crawler is served | $139/month (Pro); 7-day trial |
| Ahrefs Site Audit | Whole site from Ahrefs’ cloud, 170+ issues; Bot Analytics shows bots from your logs after the fact | Ahrefs’ own crawler | What a named AI crawler is served | Included in Ahrefs Free; paid plans from $29/month |
| Lighthouse / PageSpeed Insights (Google) | One page, rendered, plus field data from real Chrome users | A browser | Everything a non-browser client is served | Free |
| CrawlCheck | Homepage and machine files (robots.txt, sitemap, llms.txt, entity data), fetched from outside; deep crawls to 500 pages on the API, 5,000 on Growth | A browser and several named AI and search crawlers in one scan, compared against each other | Ranking, traffic, backlinks and site-wide on-page issues | First scan free, then $3 a scan; Watch $29/month for 3 domains; API from $99/month |
Sources: Screaming Frog pricing and configuration guide; Sitebulb pricing; Semrush’s SEO Toolkit pricing page; Ahrefs’ pricing and Site Audit pages; About PageSpeed Insights; CrawlCheck pricing.
What do all of them miss? #
Access failures are differences: the same URL answered one way for a browser and another way for GPTBot. A crawl that uses a single identity sees one side and has nothing to compare it with. On one site we scanned, GPTBot and ClaudeBot received 502 while Googlebot and PerplexityBot were served normally, with robots.txt allowing all four. A desktop crawl, a cloud audit and a Lighthouse run would each have reported that site as healthy. These are the problems that depend on the comparison, with how often each turns up across CrawlCheck’s scans:
| Problem | Why a single-identity audit misses it | Share of scanned sites (live) |
|---|---|---|
| An answer engine’s crawler refused while a browser was served | The audit is the browser, or a crawler the rule lets through | 5.8% |
| A named crawler served materially less text than a browser | Only one response size is ever seen | 1.1% |
| A challenge page served with status 200 | A 200 with HTML reads as success | 0.2% |
| A stale copy served from the edge cache | Tools add cache-busting parameters or hit a warm, correct copy | 13.5% |
| Named AI groups in robots.txt drop the * rules | The file is read as one crawler, so one group | 14% |
| Content that only exists after JavaScript runs | Browser-based tools render it, so the page looks complete | 2.1% |
| Almost nothing a machine receives is readable text | Not measured as a crawler receives it | 28.4% |
| Machine files that cost more to fetch than the page | Machine files are not followed as a crawler follows them | 22.7% |
| An AI opt-out signal already set | Rarely reported per crawler | 1.8% |
The robots.txt row is the one site owners most often cause themselves. A file can allow Googlebot everything and, through a group added for GPTBot further down, drop every Disallow rule for that crawler. See why naming a crawler drops your Disallow rules. And a challenge page returned with status 200 and HTML is parsed as no directives, everything allowed, which most uptime monitors and validators will call healthy.
Can I just set Screaming Frog’s user-agent to GPTBot? #
You can, and it is a useful spot check. A crawl with GPTBot’s string shows what your server returns to that name. Three things limit what it proves. It tests one identity per crawl, so comparing GPTBot with a browser means two crawls and a manual diff. It runs from your own IP address, which your CDN may trust differently from a data centre. And many bot-management rules key on more than the user-agent string. It does not replace a side-by-side comparison of every crawler that matters to you.
How do I check what an AI crawler receives myself? #
Fetch the page as a browser and as GPTBot, and compare the status and size:
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
-A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/128 Safari/537.36" https://example.com/
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
-A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" https://example.com/A 403 or 429 on the second line, or a response a fraction of the size of the first, means the crawler is treated differently. Then turn JavaScript off in your browser and reload: whatever disappears is what most AI crawlers never see (do AI crawlers render JavaScript). Two caveats: a user-agent string is not the real crawler, so a firewall that checks published IP ranges may treat your test and the real GPTBot differently; and one fetch is one moment, because challenges and rate limits come and go.
Do the AI features in these tools check AI crawler access? #
No. “AI website audit” is sold as one thing and means two. In Semrush, Ahrefs, Screaming Frog and Sitebulb, the AI sits inside the audit: Semrush’s AI Search Site Audit view flags issues it associates with visibility in LLMs, Screaming Frog runs custom prompts through OpenAI, Gemini, Ollama or Anthropic as it crawls, Sitebulb puts its audits in front of Claude and ChatGPT over MCP, and Ahrefs’ AI Content Helper is a separate writing tool. All useful; none changes who the crawl arrives as. A firewall rule that challenges AI crawlers and lets SEO crawlers through never shows up in those reports. An audit of what AI receives has to fetch the site as the AI crawlers do.
Is a perfect Lighthouse score enough? #
It measures a different visitor. Lighthouse runs the page in Chrome, so a page whose text only appears after JavaScript scores normally while most AI crawlers receive the empty shell. Core Web Vitals do not apply to crawlers at all: a crawler does not paint, scroll or click, and it does not count toward Chrome field data. Lab scores also move between runs: we measured a 225% spread on two runs of the same unchanged page, so use field data where a page has the traffic and a median of several runs where it does not.
| Lighthouse says | CrawlCheck says | What it usually means |
|---|---|---|
| Fast, 90+ | Crawler refused | A firewall or bot rule treats crawlers differently from browsers; speed is irrelevant until access is fixed |
| Fast, 90+ | Page is mostly code | The text is built by JavaScript; server-render the main content |
| Slow | Clean | A people problem, not a machine one: fix images, scripts and server time for visitors |
| Slow | Crawler served a challenge | Often the same cause: a heavy security layer in front of the site |
How do I turn one finding into a site-wide fix list? #
The tools work best in sequence. CrawlCheck finds the problem from the outside, on the pages an AI crawler reaches first; the desktop crawler then finds every page that shares the cause. The most common case is content that only exists after JavaScript runs:
- The CrawlCheck report flags the homepage: the HTML a crawler receives carries little of the text a browser shows.
- In Screaming Frog, enable JavaScript rendering and check the Contains JavaScript Content issue, which lists pages whose rendered text differs from the original HTML. In Sitebulb, the Response vs Render report makes the same comparison.
- Group the affected pages by template; the fix is usually one change to the template or the rendering setup, not page-by-page edits.
- Rescan with CrawlCheck to confirm AI crawlers now receive the text.
Why is a one-off audit not enough? #
Every tool above is sold as an audit: a report on a day. A site changes with every deploy, cache purge and plugin update, and measured continuously the gap is not small:
Citing this figure. The numbers above are recomputed from the live record when this page loads, so link the anchor rather than freezing a copy: https://crawlcheck.io/blog/your-site-was-clean-the-day-you-audited-it#the-gap. Sentence form, as of 2026-11-22: Among the 48 sites CrawlCheck measured daily between 2026-08-14 and 2026-11-22, 87.5% carried a machine-layer defect on the latest reading, while 100% carried one at least once — a gap of 12.5 points invisible to any single-day audit.
A one-off audit at any budget can only show the first figure; the second needs the same sites watched before the question was asked. The full measurement is in site audits go stale. CrawlCheck Watch re-measures automatically for $29 a month across three domains.
Which should I use? #
| You are | Use | Why |
|---|---|---|
| A small business with one site | A free CrawlCheck scan, plus Ahrefs Free or Screaming Frog’s free version for on-page issues | Both cost nothing; one answers whether AI crawlers are served, the other lists what to fix on the page |
| An e-commerce or large site | Screaming Frog or Sitebulb, plus CrawlCheck | Thousands of product pages need a full crawl; the AI access check is per site and cheap to add |
| An agency with many clients | Semrush or Ahrefs for reporting; the CrawlCheck API for AI access across clients | The suites report and white-label; the API runs the access check at scale |
| Anyone chasing speed | PageSpeed Insights | Field Core Web Vitals for real visitors, free |
Where CrawlCheck is weaker #
- No full-site crawl: link graphs, redirect chains across thousands of URLs, duplicate titles, custom extraction and visualisations are the desktop crawlers’ job. The free scan reads one homepage and the machine files around it; deep crawls top out at 500 or 5,000 pages by plan.
- No keyword, backlink or rank data.
- No field Core Web Vitals, accessibility audits at Lighthouse’s depth, or a performance waterfall.
- It reads what the server sends before JavaScript, by design, and it sends each crawler’s published user-agent; it cannot send from the AI companies’ own IP addresses.
If you need a 100,000-URL audit, buy a desktop crawler or a suite. If you need to know whether ChatGPT, Claude and Perplexity are being served your site, that is the question none of them asks; run a free scan.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
Can Screaming Frog check if AI crawlers can read my site?
Partly. Screaming Frog lets you set a custom user-agent such as GPTBot's and crawl as that name, one identity per crawl, from your own IP address. It does not compare several AI crawler identities against a browser in one pass or read robots.txt separately for each AI crawler, so differences between what each crawler is served stay hidden.
Does Semrush or Ahrefs Site Audit check AI crawlers?
Not as the crawlers themselves. Both audits crawl from the vendor's cloud as the vendor's own bot. Semrush adds an AI Search Site Audit view and Ahrefs shows bots from your logs after the fact, but neither fetches your site as GPTBot or ClaudeBot, so a rule that refuses only AI crawlers is outside what they can see.
Does PageSpeed Insights or Lighthouse check AI crawlers?
No. PageSpeed Insights runs Lighthouse in a browser and shows field data from real Chrome users. It measures performance, accessibility, best practices and basic SEO for people. Its crawlability check looks for noindex tags and headers that block all crawlers, not robots.txt rules for individual crawlers such as GPTBot.
Can a site score 100 in Lighthouse and still be invisible to ChatGPT?
Yes. A firewall can refuse AI crawlers while serving browsers, or the content can exist only after JavaScript runs, which Lighthouse executes and most AI crawlers do not. Neither problem shows in the score.
Screaming Frog vs Sitebulb: which is better?
Both are mature technical SEO crawlers that render JavaScript and crawl a whole site deeply. Screaming Frog is a desktop licence with a free version for small sites; Sitebulb offers desktop and cloud plans with prioritised hints. The better choice depends on workflow, and neither is built to compare what different AI crawlers are served.
Do I need both a desktop crawler and CrawlCheck?
For most sites, yes. A desktop crawler or SEO suite covers site-wide technical SEO at scale; CrawlCheck covers whether AI crawlers are allowed, served and able to quote the site. They answer different questions, and CrawlCheck's first scan is free.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.