Guides · 2026-09-16 · By VSNARY | Emmanuel Orta · 0 views

HTTP status codes for crawlers: what a 200 can hide, when a 403 is not a block, and the one control that tells them apart

The status line is the smallest part of a response. Challenge pages ship as 200, walls answer every path the same, and a crawler can get a page with half the words the browser got. Here is how to read the second half of the answer.

An HTTP status code is one integer at the top of a response, and almost every crawlability check stops at it. A 200 is read as reachable, a 404 as missing, a 403 as blocked, a 301 as moved. Each of those readings is right about the status line and can be wrong about the page, because the body, the headers and the client that asked all change what the number means. This guide is about the second half: what a crawler, an answer engine or a scanner actually receives once the first line has been read, and how to test it without trusting the number.

The short answer #

Read the status code as a claim, not a verdict. Then check three things the code does not carry: what the body is (a page, a challenge, a block template, an HTML file where a machine file should be), who asked (the same URL can answer differently to a browser, a named crawler and an answer engine fetching on a user's behalf), and whether the answer is the same for a path that cannot exist. The last one is the control that tells a firewall from a website: a site that answers an impossible URL with the same status and the same bytes as its homepage is not serving pages at all.

What each class actually tells a crawler #

2xx. The server produced a body it considers a success. Nothing in the code says the body is the page. Bot-management products, maintenance screens and consent walls all return 200 with a document that contains none of the site's content. A crawler that indexes on the status alone indexes the wall.

3xx. The answer lives at another URL. What matters is where the chain ends and on which host. A homepage that 301s to a different hostname has moved the whole entity, and every signal the crawler collected under the old host now points at an alias. A redirect to the same host with a trailing slash or a scheme change is housekeeping. A redirect that lands on a 200 challenge is a refusal with an extra hop.

4xx. The server refuses or cannot find. 404 and 410 are the honest ones: the resource is absent and a crawler should drop it. 401 and 403 are policy: the resource may exist and this client may not have it. 429 is a rate answer, not a content answer, and a scanner that reports it as "blocked" is overstating. The useful question for any 4xx is whether it is per client (a named crawler is refused while a browser is served) or per site (every client gets it), because those are different problems with different owners.

5xx. The origin failed. The two that matter for crawlers are 503, which most search engines treat as temporary and retry, and 500 or 502 from an edge that could not reach the origin, which looks identical to a hard block from outside. A 5xx served from cache is worse than either: the failure has been stored and is now being replayed to every client until the object expires.

Five ways a 200 is not the page #

The scanner sends fifteen client identities to a site's homepage from one address inside one second, and then reads every response body, not just the status. That is how these classes are told apart:

The first row is the one most tools miss. A challenge page is built to satisfy a browser, so it is served as 200 with HTML, and a checker that reads the status line reports a clean site. The scanner matches the body against challenge signatures and records the content type and size alongside the status so the finding can be reproduced from the response, not from a guess about the vendor.

The impossible-path control #

Fetch the homepage. Then fetch a path that cannot exist on any site, something long and random under the same host. Compare status and byte count. A working website answers the second request differently: a 404 page, a redirect to a search page, a soft 404 with different bytes. A site behind a wall answers both the same way, because the wall never consulted the site. That single comparison is what the UNIFORM_REFUSAL finding records, and it is why a 403 on the homepage is not reported as a refusal until the control has been run: a 403 with a different answer on the impossible path is a page-level rule, a 403 with the same answer is the whole host refusing this client.

The same control separates a refusal of this scanner from a refusal of crawlers. When the homepage and robots.txt both refuse the scanner's address, the report says so as ORIGIN_REFUSED_SCANNER and leaves the homepage sections unmeasured rather than scored, because a datacenter address being refused is not evidence about what Googlebot or GPTBot receive. Check from a second network before concluding anything about them.

Per-client answers #

The identities sent include a plain browser, a mobile browser and the named crawlers: search indexers, training crawlers and the user-triggered fetchers answer engines use when someone asks a question. When the browser control is served and an answer-engine identity is refused, the report records ANSWER_ENGINE_REFUSED with the status each identity received. That finding is only confirmed after the refusal reproduces on a later scan; a refusal that is structural, such as an identity-verification wall that challenges every unverified request, is marked unconfirmed rather than counted as a block, because it reproduces every day by design and that is not the same thing as a policy against the crawler.

The most common per-client failure is quieter than a 4xx. The crawler gets a 200 and a page, and the page has a fraction of the words the browser got, because the server rendered a lighter template for a client it did not recognise. Nothing in the status line changes. CRAWLER_SERVED_LESS compares the word count each identity received against the control and names the identities that came up short.

Machine files have their own status rules #

robots.txt is the file where a wrong status does the most damage, and the rules crawlers apply to it are not the rules they apply to pages. A robots.txt that answers 200 with an HTML document is read as a file with no rules. One that answers 5xx is, for Google, treated as a temporary full disallow until it recovers. One that answers 4xx is treated as absent, which means everything is allowed. A site that intends to restrict a crawler and serves its robots.txt as a 403 to that crawler has published a permission, not a restriction. The scanner reports the status it received as ROBOTS_NOT_200 and reads the body separately so an HTML answer is not mistaken for a policy. The same logic is applied to sitemaps in the sitemap audit guide, where the file and every URL it declares are verified in two separate layers.

Redirects: read the last hop and the host #

Follow the chain to its end and record every hop's status, then look at two things. First, the final status: a chain that ends on a 200 challenge, a 404 or a 5xx is a failure that a redirect count will not show. Second, the host: a homepage that ends on a different hostname is recorded as HOMEPAGE_REDIRECTS_OFF_HOST because the scanned domain is, from a crawler's point of view, an alias, and the entity, the schema and the links all belong to the destination. A www-to-apex or http-to-https hop stays on the host and is not a finding.

Cached failures #

Every class above can be served from a cache, and a cached answer carries the status of the moment it was stored. A 5xx captured during an outage, a challenge captured during a bot-fight rule, a stale 200 captured before a redeploy: all of them keep answering after the cause is gone. The headers that expose this are the cache-status header the CDN adds and the age of the object, read against the validators the origin sent. A challenge arriving with a cache HIT is recorded on its own as CHALLENGE_PINNED_AT_EDGE, because it means the site is refusing clients it no longer intends to refuse, and the fix is a purge, not a rule change. One of our own purges returned 200 and evicted nothing, which is the same lesson from the other side.

Test it yourself #

The whole test needs a command line and no tool. Request the homepage as a browser and again as a named crawler, with the header that names it, and compare status, content type, byte count and word count. Request an impossible path under the same host and compare it with the homepage answer. Request robots.txt and look at the content type before the rules. Follow the homepage redirect chain with the hop statuses printed and note the final host. Then read the cache and age headers on any answer that looked wrong. Do it from two networks before you believe a refusal is about the crawler rather than about your address.

The free scan runs exactly this sequence across fifteen identities and records the response each one received, so the findings above can be checked against the bytes rather than argued from the status line. How to read a report explains how a refused or unmeasured section is shown, and why an unmeasured section is never scored zero.

Fix order #

Fix a uniform refusal first, because nothing else can be measured behind it. Then any challenge served as 200 and any challenge pinned at the edge, both of which are bot-management configuration, not code. Then robots.txt status and content type, because a wrong answer there rewrites every other permission. Then per-client differences, starting with answer-engine identities that are refused while the browser is served. Off-host redirects and stale caches come last: they are real, they are usually deliberate, and they are the only two on this list a site owner may decide to keep.

Questions this post answers

Does a 200 status code mean a crawler can read the page?

No. A 200 means the server produced a body it considers a success. Challenge pages, block templates and consent walls are all served as 200 with HTML. The body has to be read to know whether the page is the site's content, which is why the scanner records content type, size and a signature match beside the status.

How do I tell a firewall from a website?

Request the homepage and a URL that cannot exist under the same host. A website answers them differently. A wall answers both with the same status and the same bytes, because the site was never consulted. The same-answer case is recorded as a uniform refusal.

Why is a 403 on the homepage not always reported as a block?

A 403 that changes on an impossible path is a page-level rule. A 403 that is identical on every path is the host refusing this client. And a 403 served only to the scanner's datacenter address says nothing about what search or answer-engine crawlers receive, so the report marks those sections unmeasured rather than scored.

What happens when robots.txt returns a status other than 200?

The rules crawlers apply to robots.txt differ from pages. An HTML answer is read as a file with no rules, a 5xx is treated by Google as a temporary disallow, and a 4xx is treated as absent, which allows everything. A restriction served as a 403 to the crawler it targets has published a permission.

Can a cached response keep refusing crawlers after the rule is removed?

Yes. A challenge stored by the CDN keeps answering with the status and body it was stored with until it expires or is purged. A challenge arriving with a cache hit header is recorded separately because the fix is a purge, not a configuration change.

Does a redirect to another hostname matter to a crawler?

Yes. A homepage that ends on a different host makes the scanned domain an alias; the entity, schema and links belong to the destination. A hop that stays on the same host, such as http to https or www to apex, is housekeeping and is not reported.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works