Agent view · Guides · 2026-09-30
AI crawlers refused, redirected or served less: how to check
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/crawler-delivery-comparison-refused-redirected-or-served-less: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/crawler-delivery-comparison-refused-redirected-or-served-less>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 15,097 |
| words | 2,185 |
| sha-256 | 6ff52c67385ac48ffe766467a35c07d4a52c81d72846d0df4785dd5e5a150d16 |
| repr-digest | sha-256=:b/UsZzhaxI/+dmRno1wH1KUsgdcoRtDfR4XdXloVDRY=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795182058;expires=1795185658;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="M1tSO446MDSpRwagCKJoDw";tag="crawlcheck-page" |
| signature | sig1=:c03bN7Ma+Tqr67b+/OspcJZtFuodotp9ODJ70ZxhbtwzlLu1L4NzRORg+rFjm5UpUgF5hKau3GoABWG2/za8Cg==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Guides · 2026-09-30 · By [VSNARY | Emmanuel Orta](/author) · 2 views
# AI crawlers refused, redirected or served less: how to check
Fetch the same page as each named crawler and as an ordinary browser, at the same moment, and compare. Four outcomes are possible, and only a difference from the browser counts.
Fetch your homepage as each named search and answer crawler (GPTBot, ClaudeBot, PerplexityBot, Googlebot and others) and once as an ordinary browser, from the same address at the same time, then compare status, redirect target and word count. A crawler refused with a non-200, redirected somewhere the browser was not sent, or served a markedly thinner page is being treated differently. Two results mean there is nothing to compare: a homepage that answers 202 (a verification step, not a page) and an identical wall on every path, including one that cannot exist.
Check your own site free
Check my site
About a minute. First scan free with your email, nothing installed.
On this page
- How do I run the comparison myself?
- Outcome one: is the crawler refused?
- Outcome two: is the crawler sent somewhere else?
- Outcome three: is the crawler served less?
- Outcome four: did anybody get a page?
- What does each result look like?
- Why compare with a browser and not another crawler?
- What can this test not see?
The useful question is not whether a page loads. It is whether the page a search or answer crawler receives is the page a person receives. Sites split the two constantly and almost never on purpose: a bot-management rule, a cache keyed on user-agent, a redirect written for one client, a template that serves a lighter page to anything without JavaScript. The test is simple to state: fetch the same [URL](/glossary/url) as each named crawler and as an ordinary browser, at the same time, and compare.
The four outcomes of a crawler-versus-browser comparison. Only a difference from the browser counts.
## How do I run the comparison myself? #
Send one request per identity from the same machine, one after another, and read four things from each: status, final address after redirects, size and word count. Always compare each crawler against the browser request, never one crawler against another, because the browser is what a person sees.
```
BROWSER='paste your own browser user-agent string here'
for ua in "$BROWSER" "GPTBot" "ClaudeBot" "PerplexityBot" "Googlebot"; do
printf '%s ' "$ua"
curl -sL -A "$ua" -o page.html -w '%{http_code} %{url_effective} %{size_download} bytes ' https://example.com/
sed 's/<[^>]*>/ /g' page.html | wc -w
done
```
The first line is the browser control: paste the user-agent your own browser sends, from its developer tools. Any crawler line whose status, final address or word count departs clearly from it is a split worth investigating.
## Outcome one: is the crawler refused? #
A named crawler gets a non-200 the browser did not get. This is the loud case: the site is refusing the crawler by name, or by the address range it comes from. The [Cloudflare guide](/guides/is-cloudflare-blocking-ai-crawlers-on-your-site) covers the most common source. Share of CrawlCheck scans where at least one answer crawler is refused: **5.8%**.
## Outcome two: is the crawler sent somewhere else? #
A named crawler is redirected to an address the browser was not sent to, such as a bot-check page. A redirect that everyone follows, browser included, is a site decision and is not a split. Share of scans where a crawler is redirected away from the page a person gets: **0.2%**.
## Outcome three: is the crawler served less? #
A named crawler gets a 200 and a page, but the page is much thinner than the browser's. This is the quiet case, and it matters most, because the thinner copy is the one an engine quotes from. Small differences are noise; a non-200 belongs to outcome one, not here. Share of scans where a crawler receives a markedly thinner page: **1.1%**.
## Outcome four: did anybody get a page? #
Two results make the comparison meaningless. If the homepage answers HTTP 202, a verification or queue step is answering in place of the site: 202 is an acknowledgement, not a page. The useful fact then is which crawlers, if any, still received a real page. Share of scanned sites whose homepage is such an interstitial: **0.5%**.
If the homepage and a path that cannot exist return the same wall, every path is a wall, and nothing measured through it describes the site. Request something like `/this-page-cannot-exist` alongside the homepage to tell the two apart. Share of scanned sites that answer every path with the same [refusal](/glossary/refusal): **16%**.
## What does each result look like? #
| Named crawler | Browser | What it means |
| --- | --- | --- |
| 403 | 200 | Refused |
| 301 to /bot-check | 200 on / | Sent somewhere else |
| 301 to /home | 301 to /home | Nothing: a site decision |
| 200, 210 words | 200, 1,400 words | Served less |
| 200, 1,100 words | 200, 1,400 words | Nothing: difference too small |
| 202 | 202 | No page to compare: an interstitial |
| Same tiny answer on / and an impossible path | Same | Every path is a wall: one finding, nothing else |
## Why compare with a browser and not another crawler? #
Comparing GPTBot with Googlebot tells you the two are treated differently and nothing about which one is right. Comparing each with a browser tells you which one is getting something other than the page a person gets. That is the only comparison with a consequence: an engine quoting the thinner copy is quoting a page no visitor has seen.
## What can this test not see? #
It reads what the server sends and does not run JavaScript, so a page that builds its content in the browser looks thin to every identity equally; that is a different problem, covered in [the rendering guide](/guides/do-ai-crawlers-render-javascript). It also tests how the site treats a crawler's name, not its addresses. A site that filters by IP range will pass and still refuse the real crawler; checking your logs against published ranges covers that side, as in [the verification guide](/guides/how-to-verify-googlebot-bingbot-and-ai-crawlers). The reverse problem, requests that use a crawler's name without being that crawler, is in [how much AI crawler traffic is forged](/blog/how-much-ai-crawler-traffic-is-forged). Live rates for every outcome above are on the [dataset page](/data), and the [free scan](/) runs the whole comparison for one URL.
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.
Written by
[Emmanuel Orta](/author) · VSNARY
I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.
[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)
Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fcrawler-delivery-comparison-refused-redirected-or-served-less%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fcrawler-delivery-comparison-refused-redirected-or-served-less%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=AI%20crawlers%20refused%2C%20redirected%20or%20served%20less%3A%20how%20to%20check)[Email](mailto:?subject=AI%20crawlers%20refused%2C%20redirected%20or%20served%20less%3A%20how%20to%20check&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fcrawler-delivery-comparison-refused-redirected-or-served-less)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)
## Questions this post answers
### How do I check whether AI crawlers get a different page from visitors?
Fetch the same URL as each named crawler user-agent and once as an ordinary browser, from the same machine at the same time, and compare status, final address after redirects and word count. Any crawler that clearly departs from the browser result is being treated differently.
### Why is a redirect only a problem when the browser was not redirected?
A redirect that every client follows is a site decision, such as a move to a new homepage. Only a redirect that sends a named crawler somewhere a person is not sent is a split between crawler and visitor.
### What does a 202 homepage mean for crawlers?
202 Accepted is an acknowledgement, not a page. A verification or queue step is answering in place of the site, so there is no page to compare. The useful question is which crawlers, if any, still received the real page.
### What is a uniform refusal?
The homepage and a path that cannot exist get the same wall. Every path is a wall, so nothing measured through it describes the site, and the only finding is the wall itself.
### Does passing this test prove the real crawler gets the page?
No. It tests how the site treats the crawler's name from one address. A site that filters by address range can pass and still refuse the real crawler; checking access logs against the crawler's published ranges covers that side.
## Related findings
[2026-10-15 Robots.txt allows it, the server refuses: 57 top sites At 57 of 418 top sites, the server refused GPTBot, ClaudeBot, OAI-SearchBot or PerplexityBot even though robots.txt allows it. How](/blog/robots-txt-allows-server-refuses-ai-crawlers-october-2026)[2026-10-02 How to verify Googlebot, GPTBot and other crawlers by IP Verify Googlebot, Bingbot, GPTBot and ClaudeBot by reverse DNS or the operator's IP list. The file for each crawler, the traps, an](/guides/how-to-verify-googlebot-bingbot-and-ai-crawlers)[2026-08-28 How to check if AI crawlers can read your site Five curl checks that show whether GPTBot, ClaudeBot and PerplexityBot can read your site: content type, cache, crawler vs browser](/guides/how-to-check-if-ai-crawlers-can-read-your-site)
The main product
Found this on your own site? **We fix it for $100.**
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)
Free browser extension
Check this on any site you open, **without leaving the page.**
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.
[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)
Write for the library
Measured something yourself? **Submit an article.**
Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published.
[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week
Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).
## Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.
Send for review
Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.
[All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy)
We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)
#### How we measure, and our own machine files
CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).
**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.
**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.
**Registry:** [MCP: Operator](/registry/mcp/operator) · [MCP](/registry/mcp) · [AI policy](/registry/aipolicy) · [congress.gov](/registry/congress.gov) · [flippio.pl](/registry/flippio.pl) · [lhh.com](/registry/lhh.com) · [severadao.ai](/registry/severadao.ai) · [Place: Crestview](/registry/place/fl/crestview) · [app.neilpatel.com](/registry/capabilities/app.neilpatel.com) | **Pages:** [IP ranges](/data/ip-ranges) | **Datasets:** [Technologies: Zendesk](/datasets/technologies/zendesk) · [Problems: ARD legacy path](/datasets/problems/ard-legacy-path)
[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /guides/crawler-delivery-comparison-refused-redirected-or-served-less.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.