Guides · 2026-09-16 · By VSNARY | Emmanuel Orta

HTTP status codes for crawlers: what a 200 can hide

Challenge pages ship as 200, walls answer every path the same, and a crawler can get half the words a browser got. How to read the rest of the response, with the commands to test it.

Almost every crawlability check stops at the status line: 200 reachable, 404 missing, 403 blocked, 301 moved. Each reading is right about the number and can be wrong about the page, because the body, the headers and the client that asked all change what the number means. This guide covers what a crawler actually receives once the first line has been read, and how to test it from a command line without trusting the number.

What does each status class actually tell a crawler? #

Two rows need more than the table. A 429 is a rate answer, not a content answer, and reporting it as "blocked" overstates it. A 5xx served from cache is worse than a live one: the failure was stored and is replayed to every client until the object expires. If you only ever saw one 503, read how to tell a refusal from a hiccup before acting on it.

When is a 200 not the page? #

Read every response body, not just the status. These are the five ways a 200 turns out to be something else, with the live share of scanned sites where each one was found:

The first row is the one most tools miss. A challenge page is built to satisfy a browser, so it is served as 200 with HTML, and a checker that reads the status line reports a clean site. The body's size, content type and markup settle it from the response itself, with no need to guess which vendor produced it.

How do you tell a firewall from a website? #

Fetch the homepage. Then fetch a long, random path under the same host that cannot exist. A working website answers the second request differently: a 404 page, a redirect, a soft 404 with different bytes. A site behind a wall answers both the same way, because the wall never consulted the site. That is why a 403 on the homepage means little on its own: a 403 that changes on the impossible path is a page-level rule; a 403 that is identical on both is the whole host refusing this client.

The same control separates a refusal of your address from a refusal of crawlers. A datacenter address being turned away is not evidence about what Googlebot or GPTBot receive. Repeat the test from a second network before concluding anything about them.

Does every client get the same answer? #

Test as a browser and as named crawlers: search indexers, training crawlers and the user-triggered fetchers answer engines send when someone asks a question. A browser served while an answer engine's crawler is refused was found on 6.1% of scanned sites. A structural wall that challenges every unverified request is a different problem from a policy aimed at one crawler, and the fix has a different owner.

The most common per-client failure is quieter than a 4xx. The crawler gets a 200 and a page with a fraction of the browser's words, because the server rendered a lighter template for a client it did not recognise. The status line does not change. The crawler delivery comparison guide covers refused, redirected and served-less in detail.

Why is robots.txt different? #

Crawlers apply special rules to robots.txt status codes: an HTML body is read as a file with no rules, a 5xx is treated by Google as a temporary full disallow, and a 4xx is treated as absent, which allows everything. A restriction served as a 403 to the crawler it targets has therefore published a permission. Each code is covered in the robots.txt status codes guide; the same two-layer logic for sitemaps is in the sitemap audit guide.

Redirects: which hop matters? #

Follow the chain to its end and look at two things. The final status: a chain that ends on a 200 challenge, a 404 or a 5xx is a failure that a redirect count will not show. The final host: a homepage that ends on a different hostname makes the scanned domain an alias, and the entity, schema and links all belong to the destination. A www-to-apex or http-to-https hop stays on the host and is housekeeping.

Can a cache keep refusing after the rule is gone? #

Yes. A cached answer carries the status of the moment it was stored: a 5xx captured during an outage, a challenge captured during a bot rule, a stale 200 from before a redeploy. All of them keep answering after the cause is gone. The CDN's cache-status header and the object's age expose it, and the fix is a purge, not a rule change. One of our own purges returned 200 and evicted nothing, so check the headers again after purging.

Test it yourself #

All you need is curl. Set ua to your own browser's user-agent string first (search "what is my user agent" and paste it). Most firewall rules match the crawler's name, so GPTBot is enough for a first pass; use the full string from the crawler's documentation to be thorough, and repeat with the names of the other crawlers you care about, such as Googlebot.

# Browser vs named crawler: compare status, type and size
curl -s -o browser.html -w '%{http_code} %{content_type} %{size_download}\n' -A "$ua" https://example.com/
curl -s -o crawler.html -w '%{http_code} %{content_type} %{size_download}\n' -A "GPTBot" https://example.com/
sed -e 's/<[^>]*>//g' browser.html | wc -w
sed -e 's/<[^>]*>//g' crawler.html | wc -w

# The impossible-path control
curl -s -o /dev/null -w '%{http_code} %{size_download}\n' https://example.com/
curl -s -o /dev/null -w '%{http_code} %{size_download}\n' https://example.com/zz-no-such-page-qxv

# robots.txt: status and content type before the rules
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://example.com/robots.txt

# Redirect chain: every hop and the final host
curl -sIL https://example.com/ | grep -iE '^(HTTP|location)'

# Cache headers on any answer that looked wrong
curl -sI -A "GPTBot" https://example.com/ | grep -iE '^(age|cache-control|cf-cache-status|x-cache)'

What to compare: the crawler's status, type and word count against the browser's; the impossible path's status and size against the homepage's; robots.txt's content type (it should be text/plain, not text/html); the last status line and location of the chain; and a cache HIT on any response that looked wrong. Run the whole set from two networks before you believe a refusal is about the crawler. The free scan runs the same comparison as each named crawler and keeps the response each one received.

What to fix first #

1. A wall on every path, because nothing else can be measured behind it.
2. Challenges served as 200, and challenges pinned in cache: bot-management configuration and a purge, not code.
3. robots.txt status and content type, because a wrong answer there rewrites every other permission.
4. Per-client differences, starting with answer-engine crawlers refused while browsers are served.
5. Off-host redirects and stale caches: real, often deliberate, and the only two a site owner may decide to keep.

Questions this post answers

Does a 200 status code mean a crawler can read the page?

No. A 200 means the server produced a body it considers a success. Challenge pages, block templates and consent walls are all served as 200 with HTML. The body has to be read to know whether it is the site's content.

How do I tell a firewall from a website?

Request the homepage and a random URL that cannot exist under the same host. A website answers them differently. A wall answers both with the same status and the same bytes, because the site behind it was never consulted.

Why is a 403 on the homepage not always a block on crawlers?

A 403 that changes on an impossible path is a page-level rule. A 403 identical on every path is the host refusing that client. And a 403 served to one datacenter address says nothing about what search or answer-engine crawlers receive, so test from a second network.

What happens when robots.txt returns a status other than 200?

An HTML body is read as a file with no rules, a 5xx is treated by Google as a temporary disallow of the whole site, and a 4xx is treated as absent, which allows everything. A restriction served as a 403 to the crawler it targets has published a permission.

Can a cached response keep refusing crawlers after the rule is removed?

Yes. A challenge stored by the CDN keeps answering with the status and body it was stored with until it expires or is purged. A challenge arriving with a cache HIT header needs a purge, not a configuration change.

Does a redirect to another hostname matter to a crawler?

Yes. A homepage that ends on a different host makes the scanned domain an alias, and the entity, schema and links belong to the destination. A hop that stays on the same host, such as http to https or www to apex, is housekeeping.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works