Guides · 2026-09-30 · By VSNARY | Emmanuel Orta · 2 views
AI crawlers refused, redirected or served less: how to check
Fetch the same page as each named crawler and as an ordinary browser, at the same moment, and compare. Four outcomes are possible, and only a difference from the browser counts.
Fetch your homepage as each named search and answer crawler (GPTBot, ClaudeBot, PerplexityBot, Googlebot and others) and once as an ordinary browser, from the same address at the same time, then compare status, redirect target and word count. A crawler refused with a non-200, redirected somewhere the browser was not sent, or served a markedly thinner page is being treated differently. Two results mean there is nothing to compare: a homepage that answers 202 (a verification step, not a page) and an identical wall on every path, including one that cannot exist.
The useful question is not whether a page loads. It is whether the page a search or answer crawler receives is the page a person receives. Sites split the two constantly and almost never on purpose: a bot-management rule, a cache keyed on user-agent, a redirect written for one client, a template that serves a lighter page to anything without JavaScript. The test is simple to state: fetch the same URL as each named crawler and as an ordinary browser, at the same time, and compare.
How do I run the comparison myself? #
Send one request per identity from the same machine, one after another, and read four things from each: status, final address after redirects, size and word count. Always compare each crawler against the browser request, never one crawler against another, because the browser is what a person sees.
BROWSER='paste your own browser user-agent string here'
for ua in "$BROWSER" "GPTBot" "ClaudeBot" "PerplexityBot" "Googlebot"; do
printf '%s ' "$ua"
curl -sL -A "$ua" -o page.html -w '%{http_code} %{url_effective} %{size_download} bytes ' https://example.com/
sed 's/<[^>]*>/ /g' page.html | wc -w
doneThe first line is the browser control: paste the user-agent your own browser sends, from its developer tools. Any crawler line whose status, final address or word count departs clearly from it is a split worth investigating.
Outcome one: is the crawler refused? #
A named crawler gets a non-200 the browser did not get. This is the loud case: the site is refusing the crawler by name, or by the address range it comes from. The Cloudflare guide covers the most common source. Share of CrawlCheck scans where at least one answer crawler is refused: 5.8%.
Outcome two: is the crawler sent somewhere else? #
A named crawler is redirected to an address the browser was not sent to, such as a bot-check page. A redirect that everyone follows, browser included, is a site decision and is not a split. Share of scans where a crawler is redirected away from the page a person gets: 0.2%.
Outcome three: is the crawler served less? #
A named crawler gets a 200 and a page, but the page is much thinner than the browser's. This is the quiet case, and it matters most, because the thinner copy is the one an engine quotes from. Small differences are noise; a non-200 belongs to outcome one, not here. Share of scans where a crawler receives a markedly thinner page: 1.1%.
Outcome four: did anybody get a page? #
Two results make the comparison meaningless. If the homepage answers HTTP 202, a verification or queue step is answering in place of the site: 202 is an acknowledgement, not a page. The useful fact then is which crawlers, if any, still received a real page. Share of scanned sites whose homepage is such an interstitial: 0.5%.
If the homepage and a path that cannot exist return the same wall, every path is a wall, and nothing measured through it describes the site. Request something like /this-page-cannot-exist alongside the homepage to tell the two apart. Share of scanned sites that answer every path with the same refusal: 16%.
What does each result look like? #
| Named crawler | Browser | What it means |
|---|---|---|
| 403 | 200 | Refused |
| 301 to /bot-check | 200 on / | Sent somewhere else |
| 301 to /home | 301 to /home | Nothing: a site decision |
| 200, 210 words | 200, 1,400 words | Served less |
| 200, 1,100 words | 200, 1,400 words | Nothing: difference too small |
| 202 | 202 | No page to compare: an interstitial |
| Same tiny answer on / and an impossible path | Same | Every path is a wall: one finding, nothing else |
Why compare with a browser and not another crawler? #
Comparing GPTBot with Googlebot tells you the two are treated differently and nothing about which one is right. Comparing each with a browser tells you which one is getting something other than the page a person gets. That is the only comparison with a consequence: an engine quoting the thinner copy is quoting a page no visitor has seen.
What can this test not see? #
It reads what the server sends and does not run JavaScript, so a page that builds its content in the browser looks thin to every identity equally; that is a different problem, covered in the rendering guide. It also tests how the site treats a crawler's name, not its addresses. A site that filters by IP range will pass and still refuse the real crawler; checking your logs against published ranges covers that side, as in the verification guide. The reverse problem, requests that use a crawler's name without being that crawler, is in how much AI crawler traffic is forged. Live rates for every outcome above are on the dataset page, and the free scan runs the whole comparison for one URL.
Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn
Questions this post answers
How do I check whether AI crawlers get a different page from visitors?
Fetch the same URL as each named crawler user-agent and once as an ordinary browser, from the same machine at the same time, and compare status, final address after redirects and word count. Any crawler that clearly departs from the browser result is being treated differently.
Why is a redirect only a problem when the browser was not redirected?
A redirect that every client follows is a site decision, such as a move to a new homepage. Only a redirect that sends a named crawler somewhere a person is not sent is a split between crawler and visitor.
What does a 202 homepage mean for crawlers?
202 Accepted is an acknowledgement, not a page. A verification or queue step is answering in place of the site, so there is no page to compare. The useful question is which crawlers, if any, still received the real page.
What is a uniform refusal?
The homepage and a path that cannot exist get the same wall. Every path is a wall, so nothing measured through it describes the site, and the only finding is the wall itself.
Does passing this test prove the real crawler gets the page?
No. It tests how the site treats the crawler's name from one address. A site that filters by address range can pass and still refuse the real crawler; checking access logs against the crawler's published ranges covers that side.
Related findings
The main product
Found this on your own site? We fix it for $100.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Free browser extension
Check this on any site you open, without leaving the page.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
Write for the library
Measured something yourself? Submit an article.
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
How it works · a reply within about a week
Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.