CrawlCheck

Guides · 2026-09-30 · By · 2 views

AI crawlers refused, redirected or served less: how to check

Fetch the same page as each named crawler and as an ordinary browser, at the same moment, and compare. Four outcomes are possible, and only a difference from the browser counts.

Fetch your homepage as each named search and answer crawler (GPTBot, ClaudeBot, PerplexityBot, Googlebot and others) and once as an ordinary browser, from the same address at the same time, then compare status, redirect target and word count. A crawler refused with a non-200, redirected somewhere the browser was not sent, or served a markedly thinner page is being treated differently. Two results mean there is nothing to compare: a homepage that answers 202 (a verification step, not a page) and an identical wall on every path, including one that cannot exist.

Check your own site free

About a minute. First scan free with your email, nothing installed.

The useful question is not whether a page loads. It is whether the page a search or answer crawler receives is the page a person receives. Sites split the two constantly and almost never on purpose: a bot-management rule, a cache keyed on user-agent, a redirect written for one client, a template that serves a lighter page to anything without JavaScript. The test is simple to state: fetch the same URL as each named crawler and as an ordinary browser, at the same time, and compare.

Same URL, same moment: crawler vs browserDid anyone get a real page?No: 202, or the samewall on every pathyesStatus differscrawler non-200, browser 200Redirect differsonly the crawler is movedWords differcrawler page much thinnerRefusedSent somewhere elseServed lessAnything everyone gets, browser included, is a site decision, not a split
The four outcomes of a crawler-versus-browser comparison. Only a difference from the browser counts.

How do I run the comparison myself? #

Send one request per identity from the same machine, one after another, and read four things from each: status, final address after redirects, size and word count. Always compare each crawler against the browser request, never one crawler against another, because the browser is what a person sees.

BROWSER='paste your own browser user-agent string here'
for ua in "$BROWSER" "GPTBot" "ClaudeBot" "PerplexityBot" "Googlebot"; do
  printf '%s  ' "$ua"
  curl -sL -A "$ua" -o page.html -w '%{http_code} %{url_effective} %{size_download} bytes  ' https://example.com/
  sed 's/<[^>]*>/ /g' page.html | wc -w
done

The first line is the browser control: paste the user-agent your own browser sends, from its developer tools. Any crawler line whose status, final address or word count departs clearly from it is a split worth investigating.

Outcome one: is the crawler refused? #

A named crawler gets a non-200 the browser did not get. This is the loud case: the site is refusing the crawler by name, or by the address range it comes from. The Cloudflare guide covers the most common source. Share of CrawlCheck scans where at least one answer crawler is refused: 5.8%.

Outcome two: is the crawler sent somewhere else? #

A named crawler is redirected to an address the browser was not sent to, such as a bot-check page. A redirect that everyone follows, browser included, is a site decision and is not a split. Share of scans where a crawler is redirected away from the page a person gets: 0.2%.

Outcome three: is the crawler served less? #

A named crawler gets a 200 and a page, but the page is much thinner than the browser's. This is the quiet case, and it matters most, because the thinner copy is the one an engine quotes from. Small differences are noise; a non-200 belongs to outcome one, not here. Share of scans where a crawler receives a markedly thinner page: 1.1%.

Outcome four: did anybody get a page? #

Two results make the comparison meaningless. If the homepage answers HTTP 202, a verification or queue step is answering in place of the site: 202 is an acknowledgement, not a page. The useful fact then is which crawlers, if any, still received a real page. Share of scanned sites whose homepage is such an interstitial: 0.5%.

If the homepage and a path that cannot exist return the same wall, every path is a wall, and nothing measured through it describes the site. Request something like /this-page-cannot-exist alongside the homepage to tell the two apart. Share of scanned sites that answer every path with the same refusal: 16%.

What does each result look like? #

Named crawlerBrowserWhat it means
403200Refused
301 to /bot-check200 on /Sent somewhere else
301 to /home301 to /homeNothing: a site decision
200, 210 words200, 1,400 wordsServed less
200, 1,100 words200, 1,400 wordsNothing: difference too small
202202No page to compare: an interstitial
Same tiny answer on / and an impossible pathSameEvery path is a wall: one finding, nothing else

Why compare with a browser and not another crawler? #

Comparing GPTBot with Googlebot tells you the two are treated differently and nothing about which one is right. Comparing each with a browser tells you which one is getting something other than the page a person gets. That is the only comparison with a consequence: an engine quoting the thinner copy is quoting a page no visitor has seen.

What can this test not see? #

It reads what the server sends and does not run JavaScript, so a page that builds its content in the browser looks thin to every identity equally; that is a different problem, covered in the rendering guide. It also tests how the site treats a crawler's name, not its addresses. A site that filters by IP range will pass and still refuse the real crawler; checking your logs against published ranges covers that side, as in the verification guide. The reverse problem, requests that use a crawler's name without being that crawler, is in how much AI crawler traffic is forged. Live rates for every outcome above are on the dataset page, and the free scan runs the whole comparison for one URL.

Share of scanned sites with each finding in this articleEvery crawler refused alike16%Answer-engine crawler refused5.8%Crawler served less than a browser1.1%Homepage is an interstitial page0.5%Crawler redirected away0.2%Share of scanned sites with eachfinding in this articleEvery crawler refused alike16%Answer-engine crawler refused5.8%Crawler served less than a browser1.1%Homepage is an interstitial page0.5%Crawler redirected away0.2%
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Questions this post answers

How do I check whether AI crawlers get a different page from visitors?

Fetch the same URL as each named crawler user-agent and once as an ordinary browser, from the same machine at the same time, and compare status, final address after redirects and word count. Any crawler that clearly departs from the browser result is being treated differently.

Why is a redirect only a problem when the browser was not redirected?

A redirect that every client follows is a site decision, such as a move to a new homepage. Only a redirect that sends a named crawler somewhere a person is not sent is a split between crawler and visitor.

What does a 202 homepage mean for crawlers?

202 Accepted is an acknowledgement, not a page. A verification or queue step is answering in place of the site, so there is no page to compare. The useful question is which crawlers, if any, still received the real page.

What is a uniform refusal?

The homepage and a path that cannot exist get the same wall. Every path is a wall, so nothing measured through it describes the site, and the only finding is the wall itself.

Does passing this test prove the real crawler gets the page?

No. It tests how the site treats the crawler's name from one address. A site that filters by address range can pass and still refuse the real crawler; checking access logs against the crawler's published ranges covers that side.

Related findings

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Write for the library

Measured something yourself? Submit an article.

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.

Submit an article

How it works · a reply within about a week

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works