Guides · 2026-09-23 · By VSNARY | Emmanuel Orta · 0 views

robots.txt 404, 403, 429 and 503: what each tells a crawler

The status code on robots.txt decides how a crawler treats the whole site before it reads a single rule. A 404 means crawl everything. A 503 means crawl nothing. A 403 is where crawlers disagree.

Every crawler that respects robots.txt fetches it before anything else, and the first thing it reads is not a rule. It is the status code. The standard that governs the file, RFC 9309, says what each class of status means, and the meanings are not the ones most site owners expect. A missing file does not mean no crawling. A server error does not mean try again as normal. This guide goes through them in the order a crawler meets them.

The short version #

robots.txt returns 200: the only answer that publishes rules #

A 200 with a plain-text body is the file doing its job. The crawler finds the group that names it, or the * group if none does, and obeys only that group. That last part catches many sites: a named group does not inherit the rules under *, which is explained in the guide to whether AI crawlers respect robots.txt.

The standard asks crawlers to parse at least the first 500 kibibytes of the file. A larger file is not an error, but rules past that point may never be read. Keep the file short enough that the limit never matters.

A 200 that is not a robots.txt #

The most expensive answer is a 200 that carries an HTML page. It happens when a framework or content system answers every unknown path with a page, and robots.txt was never created as a real file. The status says success; the body contains no directives; a crawler parses it, finds no rules, and treats the whole site as allowed. Whatever the owner believes the file says is not in force.

CrawlCheck separates two versions of this. When robots.txt returns the same catch-all page a nonsense path returns, the report says there is no robots.txt at all, ROBOTS_IS_CATCHALL, found on 0.3% of scans. When it returns some other HTML, the report says the file is an HTML page, ROBOTS_IS_HTML, found on 0.3%. A plain-text body with the wrong content type, ROBOTS_CTYPE, is milder: most crawlers parse it anyway, and it shows up on 0.1%.

robots.txt redirects: followed, within limits #

RFC 9309 asks crawlers to follow at least five consecutive redirects for robots.txt. Google follows at least five and then stops, treating the file as a 404, which means no restrictions. A redirect from the bare host to www, or from http to https, is normal and harmless. What matters is the answer at the end of the chain, and whether that answer belongs to the host being crawled. A robots.txt that redirects to another domain's file is being read as that domain's rules.

robots.txt returns 404 or 410: everything is allowed #

A 404 or 410 on robots.txt means, under the standard, that the file is unavailable, and a crawler may access any resource on the server. That is the right outcome for a site that has no restrictions. It is the wrong outcome for a site that believes it has some: a file deleted in a migration, or placed at the wrong path, silently turns every restriction off.

CrawlCheck reports these as ROBOTS_NOT_200, found on 3.9% of scans. The fix is to serve the file you mean with a 200, or to confirm that allowing everything is what you intended.

robots.txt returns 403 or 401: rules ignored by Google #

401 and 403 are 4xx codes, so by the letter of RFC 9309 they mean unavailable, and a crawler may fetch everything. Google documents the same treatment: its crawlers treat every 4xx except 429 as if no valid robots.txt existed, so a 403 and a 404 are equivalent and every rule in the file is ignored. Some crawlers and crawler libraries instead treat an authorisation refusal as a sign the site does not want to be crawled, and stop. A site that serves robots.txt as a 403 to a particular crawler, intending to keep it out, has under the standard published a permission, not a restriction.

Because the outcome depends on who is reading, CrawlCheck reports a 401 or 403 on robots.txt as ROBOTS_BLOCKED, the same code as a server error, found on 0.5% of scans. Either way, the rules you wrote are not what the crawler read. This most often comes from a firewall or bot-management rule that challenges every client it does not recognise, robots.txt included.

robots.txt returns 429: treated as a server error #

A 429 means too many requests. It is a 4xx code, so the standard would allow a crawler to treat it as no file. Google does not: it treats a 429 on robots.txt as a server error, the same as a 5xx, and slows or stops crawling rather than assuming everything is allowed. A rate limit that catches robots.txt therefore has the opposite effect on Google from the one a 403 has. CrawlCheck reports a 429 on robots.txt as ROBOTS_NOT_200, found on 3.9% of scans across every non-200 answer it covers. Exempt robots.txt from rate limits.

robots.txt returns 500 or 503: crawling stops #

A 500, 502 or 503 on robots.txt means the file is unreachable, and the standard says a crawler must assume complete disallow. A site that is otherwise working but returns 503 on robots.txt, for example because a maintenance rule or a rate limit catches that path, is telling every compliant crawler to stay away.

The standard allows a crawler to fall back to a copy it cached earlier, and after an outage lasting a long time, on the order of 30 days, to treat the file as unavailable instead. Google documents its own sequence: for the first 12 hours it stops crawling the site while it keeps retrying robots.txt, then for up to 30 days it uses the last version it fetched successfully, retrying a 503 fairly often. Neither helps a new site, and neither is something to rely on. Crawlers should also not keep a cached copy for more than 24 hours in normal operation, so a fix is picked up within about a day.

How to check yours #

Request /robots.txt on the bare URL, from outside your own network, with no cookies. Record four things: the status, the content type, the first line of the body, and the final host after any redirects. Then request it with a crawler's user-agent, because firewalls often answer crawlers differently from browsers. The file tools on the tools page run these reads and compare them.

If the answers differ between the browser and the crawler identity, the crawler's answer is the one that matters. The per-client differences, and why a 200 to your browser proves little, are covered in the guide to what status codes a crawler actually gets.

What to fix first #

A 5xx comes first, because it is the only answer that turns crawling off entirely. Then a 200 that returns HTML, because it silently turns your rules off. Then a 403 or 401 aimed at crawlers, because its outcome depends on the reader. Then a 404, which is harmless if intended and costly if not. Content type comes last. The live share of each is on the dataset page.

Questions this post answers

What happens if robots.txt returns 404?

Under RFC 9309 a 404 means the file is unavailable, and a crawler may fetch every page on the site. Any restriction you meant to publish is not in force.

What happens if robots.txt returns 503?

A 5xx means the file is unreachable, and the standard says a crawler must assume the whole site is disallowed. Google stops crawling for the first 12 hours, then uses the last good copy for up to 30 days.

Does a 403 on robots.txt block crawlers?

No, not for Google. Google treats 403 like 404: as if no robots.txt existed, so every rule in the file is ignored and everything may be crawled. Some other crawlers read a 403 as a refusal.

What happens if robots.txt returns 429?

Google treats a 429 on robots.txt as a server error, the same as a 5xx, and slows or stops crawling. Under the standard alone a 429 is a 4xx, but Google is explicit that 429 is the exception to its 4xx rule.

Why is a robots.txt that returns an HTML page a problem?

A crawler parses the HTML, finds no directives and treats every path as allowed. The status says success while none of your rules are in force.

How long does a crawler keep an old robots.txt?

RFC 9309 says a crawler should not use a cached copy for more than 24 hours unless the file cannot be fetched.

How many redirects will a crawler follow for robots.txt?

The standard asks crawlers to follow at least five consecutive redirects. The answer at the end of the chain is the one that counts.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works