CrawlCheck

Agent view · Guides · 2026-10-03

Screaming Frog vs Sitebulb vs CrawlCheck: AI crawler audits

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/screaming-frog-vs-sitebulb-vs-crawlcheck-ai-crawler-audit: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

status200
content-typetext/markdown; charset=utf-8
varyAccept
link<https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/screaming-frog-vs-sitebulb-vs-crawlcheck-ai-crawler-audit>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory"
cache-controlpublic, max-age=900, stale-while-revalidate=2400
bytes23,682
words3,541
sha-2562e19a5ee9d18407a73e20e4d439ab1b5c34ab1a832ab7f8caa19f6cc87730a9b
repr-digestsha-256=:Lhml7p0YQHpz4g5NQ5qxtcNKsagyq3+Mqhn2zIdzCps=: ← matches the body shown
signature-inputsig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795311496;expires=1795315096;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="CJ0byIgvSQX0imCa5hrjHQ";tag="crawlcheck-page"
signaturesig1=:B8IktAfUK/lby3vsvjN9I9xb0gI+YFd86g0MYjfpxwjZaaM+5DFGVA0X/8MtStgG9zKbknw80dAlmO8ybJMgCg==:
verifiedyes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba
Guides · 2026-10-03 · By [VSNARY | Emmanuel Orta](/author) · 3 views

# Screaming Frog vs Sitebulb vs CrawlCheck: AI crawler audits

Screaming Frog, Sitebulb, Semrush, Ahrefs and Lighthouse compared on the question that decides what each can find: what request it sends, and as whom. With where CrawlCheck falls short.

Screaming Frog, Sitebulb, Semrush and Ahrefs crawl your whole site for technical SEO issues, each as one identity: its own user-agent, from your IP or the vendor's cloud. Lighthouse and PageSpeed Insights render one page as a browser. None compares what a browser, GPTBot, ClaudeBot and PerplexityBot are each served, so a site that serves people and refuses AI crawlers passes all of them. CrawlCheck makes that comparison in a free scan but reads far fewer pages. Use a desktop crawler or suite for site-wide SEO and CrawlCheck for AI crawler access.

Check your own site free

Check my site

About a minute. First scan free with your email, nothing installed.

On this page

- What does each audit tool actually fetch?
- What do all of them miss?
- Can I just set Screaming Frog’s user-agent to GPTBot?
- How do I check what an AI crawler receives myself?
- Do the AI features in these tools check AI crawler access?
- Is a perfect Lighthouse score enough?
- How do I turn one finding into a site-wide fix list?
- Why is a one-off audit not enough?
- Which should I use?
- Where CrawlCheck is weaker

Every audit tool produces a list of issues, and the lists overlap enough that comparisons usually come down to interface and price. The question that decides what a tool can find is narrower: **what request does it send, and as whom?** Screaming Frog, Sitebulb, Semrush and Ahrefs crawl as one identity. [Lighthouse](/glossary/lighthouse) renders as a browser. None of them fetches the same page as a browser, GPTBot, ClaudeBot and PerplexityBot and compares the answers, and that comparison is how you find a site that serves people and refuses AI crawlers.

**Disclosure:** CrawlCheck publishes this comparison and is one of the products in it. Every price and feature claim about another company comes from that company’s own pricing or documentation page, read in October 2026; prices change, so the vendor page is the source of truth. Where CrawlCheck does less than a competitor, the tables say so.

## What does each audit tool actually fetch? #

| Tool | What it requests | As whom | Blind to | Starting price |
| --- | --- | --- | --- | --- |
| Screaming Frog SEO Spider | Every [URL](/glossary/url) it can discover, from your desktop; renders JavaScript in headless Chrome; can run custom AI prompts during a crawl | One user-agent per crawl (presets for Googlebot, Bingbot and browsers, or a custom string), from your IP | Responses that differ by crawler identity; edge rules that trust your IP | £199 a year; free up to 500 URLs |
| Sitebulb | Whole site from desktop (Lite 10,000 URLs per audit; Pro 500,000, up to 2 million) or cloud; [MCP](/glossary/mcp) access into Claude and ChatGPT | One user-agent per crawl | Responses that differ by crawler identity | Cloud from £95/month; 14-day free trial |
| Semrush Site Audit | Whole site from Semrush’s cloud, 140+ checks, with an AI Search Site Audit view | Semrush’s own crawler | What a named AI crawler is served | $139/month (Pro); 7-day trial |
| Ahrefs Site Audit | Whole site from Ahrefs’ cloud, 170+ issues; Bot Analytics shows bots from your logs after the fact | Ahrefs’ own crawler | What a named AI crawler is served | Included in Ahrefs Free; paid plans from $29/month |
| Lighthouse / PageSpeed Insights (Google) | One page, rendered, plus [field data](/glossary/field-data) from real Chrome users | A browser | Everything a non-browser client is served | Free |
| CrawlCheck | Homepage and machine files ([robots.txt](/glossary/robots-txt), sitemap, [llms.txt](/glossary/llms-txt), entity data), fetched from outside; deep crawls to 500 pages on the API, 5,000 on Growth | A browser and several named AI and search crawlers in one scan, compared against each other | Ranking, traffic, backlinks and site-wide on-page issues | First scan free, then $3 a scan; Watch $29/month for 3 domains; API from $99/month |

Sources: [Screaming Frog pricing](https://www.screamingfrog.co.uk/seo-spider/pricing/) and [configuration guide](https://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/); [Sitebulb pricing](https://sitebulb.com/pricing/); Semrush’s SEO Toolkit pricing page; Ahrefs’ pricing and Site Audit pages; [About PageSpeed Insights](https://developers.google.com/speed/docs/insights/v5/about); CrawlCheck [pricing](/pricing).
Why a deep single-identity crawl can pass a site that refuses an AI crawler. The 403 is an illustration, not a measured site.

## What do all of them miss? #

Access failures are differences: the same URL answered one way for a browser and another way for GPTBot. A crawl that uses a single identity sees one side and has nothing to compare it with. On one site we scanned, GPTBot and ClaudeBot received 502 while Googlebot and PerplexityBot were served normally, with robots.txt allowing all four. A desktop crawl, a cloud audit and a Lighthouse run would each have reported that site as healthy. These are the problems that depend on the comparison, with how often each turns up across CrawlCheck’s scans:

| Problem | Why a single-identity audit misses it | Share of scanned sites (live) |
| --- | --- | --- |
| An [answer engine](/glossary/answer-engine)’s crawler refused while a browser was served | The audit is the browser, or a crawler the rule lets through | 5.8% |
| A named crawler served materially less text than a browser | Only one response size is ever seen | 1.1% |
| A challenge page served with status 200 | A 200 with HTML reads as success | 0.2% |
| A stale copy served from the [edge cache](/glossary/edge-cache) | Tools add cache-busting parameters or hit a warm, correct copy | 13.5% |
| Named AI groups in robots.txt drop the * rules | The file is read as one crawler, so one group | 14% |
| Content that only exists after JavaScript runs | Browser-based tools render it, so the page looks complete | 2.1% |
| Almost nothing a machine receives is readable text | Not measured as a crawler receives it | 28.4% |
| Machine files that cost more to fetch than the page | Machine files are not followed as a crawler follows them | 22.7% |
| An [AI opt-out](/glossary/ai-opt-out) signal already set | Rarely reported per crawler | 1.8% |

The robots.txt row is the one site owners most often cause themselves. A file can allow Googlebot everything and, through a group added for GPTBot further down, drop every Disallow rule for that crawler. See [why naming a crawler drops your Disallow rules](/guides/robots-txt-user-agent-groups-why-naming-a-crawler-drops-your-disallow-rules). And a challenge page returned with status 200 and HTML is parsed as no directives, everything allowed, which most uptime monitors and validators will call healthy.

## Can I just set Screaming Frog’s user-agent to GPTBot? #

You can, and it is a useful spot check. A crawl with GPTBot’s string shows what your server returns to that name. Three things limit what it proves. It tests one identity per crawl, so comparing GPTBot with a browser means two crawls and a manual diff. It runs from your own IP address, which your [CDN](/glossary/cdn) may trust differently from a data centre. And many bot-management rules key on more than the user-agent string. It does not replace a side-by-side comparison of every crawler that matters to you.

## How do I check what an AI crawler receives myself? #

Fetch the page as a browser and as GPTBot, and compare the status and size:

```
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
 -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/128 Safari/537.36" https://example.com/

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
 -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" https://example.com/
```

A 403 or 429 on the second line, or a response a fraction of the size of the first, means the crawler is treated differently. Then turn JavaScript off in your browser and reload: whatever disappears is what most AI crawlers never see ([do AI crawlers render JavaScript](/guides/do-ai-crawlers-render-javascript)). Two caveats: a user-agent string is not the real crawler, so a firewall that checks published IP ranges may treat your test and the real GPTBot differently; and one fetch is one moment, because challenges and rate limits come and go.

## Do the AI features in these tools check AI crawler access? #

No. “AI website audit” is sold as one thing and means two. In Semrush, Ahrefs, Screaming Frog and Sitebulb, the AI sits inside the audit: Semrush’s AI Search Site Audit view flags issues it associates with visibility in LLMs, Screaming Frog runs custom prompts through OpenAI, Gemini, Ollama or Anthropic as it crawls, Sitebulb puts its audits in front of Claude and ChatGPT over MCP, and Ahrefs’ AI Content Helper is a separate writing tool. All useful; none changes who the crawl arrives as. A firewall rule that challenges AI crawlers and lets SEO crawlers through never shows up in those reports. An audit of what AI receives has to fetch the site as the AI crawlers do.

## Is a perfect Lighthouse score enough? #

It measures a different visitor. Lighthouse runs the page in Chrome, so a page whose text only appears after JavaScript scores normally while most AI crawlers receive the empty shell. [Core Web Vitals](/glossary/core-web-vitals) do not apply to crawlers at all: a crawler does not paint, scroll or click, and it does not count toward Chrome field data. Lab scores also move between runs: we measured [a 225% spread on two runs of the same unchanged page](/blog/one-lighthouse-run-cannot-support-a-finding-225-percent-variance), so use field data where a page has the traffic and a median of several runs where it does not.

| Lighthouse says | CrawlCheck says | What it usually means |
| --- | --- | --- |
| Fast, 90+ | Crawler refused | A firewall or bot rule treats crawlers differently from browsers; speed is irrelevant until access is fixed |
| Fast, 90+ | Page is mostly code | The text is built by JavaScript; server-render the main content |
| Slow | Clean | A people problem, not a machine one: fix images, scripts and server time for visitors |
| Slow | Crawler served a challenge | Often the same cause: a heavy security layer in front of the site |

## How do I turn one finding into a site-wide fix list? #

The tools work best in sequence. CrawlCheck finds the problem from the outside, on the pages an AI crawler reaches first; the desktop crawler then finds every page that shares the cause. The most common case is content that only exists after JavaScript runs:

- The CrawlCheck report flags the homepage: the HTML a crawler receives carries little of the text a browser shows.
- In Screaming Frog, enable JavaScript rendering and check the [Contains JavaScript Content](https://www.screamingfrog.co.uk/seo-spider/issues/javascript/contains-javascript-content/) issue, which lists pages whose rendered text differs from the original HTML. In Sitebulb, the [Response vs Render](https://sitebulb.com/product/response-vs-render) report makes the same comparison.
- Group the affected pages by template; the fix is usually one change to the template or the rendering setup, not page-by-page edits.
- Rescan with CrawlCheck to confirm AI crawlers now receive the text.

## Why is a one-off audit not enough? #

Every tool above is sold as an audit: a report on a day. A site changes with every deploy, cache purge and plugin update, and measured continuously the gap is not small:

**Citing this figure.** The numbers above are recomputed from the live record when this page loads, so link the anchor rather than freezing a copy: `https://crawlcheck.io/blog/your-site-was-clean-the-day-you-audited-it#the-gap`. Sentence form, as of 2026-11-22: *Among the 48 sites CrawlCheck measured daily between 2026-08-14 and 2026-11-22, 87.5% carried a machine-layer defect on the latest reading, while 100% carried one at least once — a gap of 12.5 points invisible to any single-day audit.*

A one-off audit at any budget can only show the first figure; the second needs the same sites watched before the question was asked. The full measurement is in [site audits go stale](/blog/your-site-was-clean-the-day-you-audited-it). CrawlCheck Watch re-measures automatically for $29 a month across three domains.

## Which should I use? #

| You are | Use | Why |
| --- | --- | --- |
| A small business with one site | A free CrawlCheck scan, plus Ahrefs Free or Screaming Frog’s free version for on-page issues | Both cost nothing; one answers whether AI crawlers are served, the other lists what to fix on the page |
| An e-commerce or large site | Screaming Frog or Sitebulb, plus CrawlCheck | Thousands of product pages need a full crawl; the AI access check is per site and cheap to add |
| An agency with many clients | Semrush or Ahrefs for reporting; the CrawlCheck API for AI access across clients | The suites report and white-label; the API runs the access check at scale |
| Anyone chasing speed | PageSpeed Insights | Field Core Web Vitals for real visitors, free |

## Where CrawlCheck is weaker #

- No full-site crawl: link graphs, redirect chains across thousands of URLs, duplicate titles, custom extraction and visualisations are the desktop crawlers’ job. The free scan reads one homepage and the machine files around it; deep crawls top out at 500 or 5,000 pages by plan.
- No keyword, backlink or rank data.
- No field Core Web Vitals, accessibility audits at Lighthouse’s depth, or a performance waterfall.
- It reads what the server sends before JavaScript, by design, and it sends each crawler’s published user-agent; it cannot send from the AI companies’ own IP addresses.

If you need a 100,000-URL audit, buy a desktop crawler or a suite. If you need to know whether ChatGPT, Claude and Perplexity are being served your site, that is the question none of them asks; [run a free scan](/).
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fscreaming-frog-vs-sitebulb-vs-crawlcheck-ai-crawler-audit%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fscreaming-frog-vs-sitebulb-vs-crawlcheck-ai-crawler-audit%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=Screaming%20Frog%20vs%20Sitebulb%20vs%20CrawlCheck%3A%20AI%20crawler%20audits)[Email](mailto:?subject=Screaming%20Frog%20vs%20Sitebulb%20vs%20CrawlCheck%3A%20AI%20crawler%20audits&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fscreaming-frog-vs-sitebulb-vs-crawlcheck-ai-crawler-audit)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Questions this post answers

### Can Screaming Frog check if AI crawlers can read my site?

Partly. Screaming Frog lets you set a custom user-agent such as GPTBot's and crawl as that name, one identity per crawl, from your own IP address. It does not compare several AI crawler identities against a browser in one pass or read robots.txt separately for each AI crawler, so differences between what each crawler is served stay hidden.

### Does Semrush or Ahrefs Site Audit check AI crawlers?

Not as the crawlers themselves. Both audits crawl from the vendor's cloud as the vendor's own bot. Semrush adds an AI Search Site Audit view and Ahrefs shows bots from your logs after the fact, but neither fetches your site as GPTBot or ClaudeBot, so a rule that refuses only AI crawlers is outside what they can see.

### Does PageSpeed Insights or Lighthouse check AI crawlers?

No. PageSpeed Insights runs Lighthouse in a browser and shows field data from real Chrome users. It measures performance, accessibility, best practices and basic SEO for people. Its crawlability check looks for noindex tags and headers that block all crawlers, not robots.txt rules for individual crawlers such as GPTBot.

### Can a site score 100 in Lighthouse and still be invisible to ChatGPT?

Yes. A firewall can refuse AI crawlers while serving browsers, or the content can exist only after JavaScript runs, which Lighthouse executes and most AI crawlers do not. Neither problem shows in the score.

### Screaming Frog vs Sitebulb: which is better?

Both are mature technical SEO crawlers that render JavaScript and crawl a whole site deeply. Screaming Frog is a desktop licence with a free version for small sites; Sitebulb offers desktop and cloud plans with prioritised hints. The better choice depends on workflow, and neither is built to compare what different AI crawlers are served.

### Do I need both a desktop crawler and CrawlCheck?

For most sites, yes. A desktop crawler or SEO suite covers site-wide technical SEO at scale; CrawlCheck covers whether AI crawlers are allowed, served and able to quote the site. They answer different questions, and CrawlCheck's first scan is free.

## Related findings

[2026-10-03 Agent readiness checkers: Is It Agent Ready? vs CrawlCheck Is It Agent Ready? vs CrawlCheck: agent standards checked, declared vs working capabilities, AI crawler access, scores that skip w](/guides/agent-readiness-checkers-compared-isitagentready-vs-crawlcheck)[2026-10-03 AI crawler monitoring vs uptime monitors: what each one catches UptimeRobot, Visualping and CrawlCheck Watch compared: what each monitors, how often, price, and the AI crawler failures uptime ch](/guides/monitoring-ai-crawler-access-vs-uptime-and-change-monitors)[2026-11-13 Claude-User uses two published address lists. We checked one CrawlCheck checked Claude-User against one of two Anthropic address lists and counted real fetches as failed. Fixed 13 November 20](/blog/claude-user-comes-from-two-published-address-lists-we-checked-one)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Registry:** [MCP: Operator](/registry/mcp/operator) · [MCP](/registry/mcp) · [AI policy](/registry/aipolicy) · [amicited.com](/registry/amicited.com) · [crestviewbuickgmc.com](/registry/crestviewbuickgmc.com) · [gosuslugi.ru](/registry/gosuslugi.ru) | **Pages:** [IP ranges](/data/ip-ranges) | **Glossary:** [Approximate nearest neighbor](/glossary/approximate-nearest-neighbor) · [Cache key](/glossary/cache-key) · [Content refresh](/glossary/content-refresh) · [MCP preflight](/glossary/mcp-preflight) · [frame-ancestors](/glossary/frame-ancestors)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)

Same body as a file: /guides/screaming-frog-vs-sitebulb-vs-crawlcheck-ai-crawler-audit.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.