Guides · 2026-09-23 · By VSNARY | Emmanuel Orta

robots.txt status codes: what a 404, 403, 429 or 503 tells Google and AI crawlers

The status code on robots.txt decides how a crawler treats the whole site before it reads a single rule. A 404 means crawl everything. A 503 means crawl nothing. A 403 is where crawlers disagree.

Every crawler that respects robots.txt fetches it before anything else, and the first thing it reads is not a rule. It is the status code. The standard that governs the file, RFC 9309, says what each class of status means, and the meanings are not the ones most site owners expect. A missing file does not mean no crawling. A server error does not mean try again as normal. This guide goes through them in the order a crawler meets them.

The short version #

robots.txt returns 200: the only answer that publishes rules #

A 200 with a plain-text body is the file doing its job. The crawler finds the group that names it, or the * group if none does, and obeys only that group. That last part catches many sites: a named group does not inherit the rules under *, which is explained in the guide to whether AI crawlers respect robots.txt.

The standard asks crawlers to parse at least the first 500 kibibytes of the file. A larger file is not an error, but rules past that point may never be read. Keep the file short enough that the limit never matters.

A 200 that is not a robots.txt #

The most expensive answer is a 200 that carries an HTML page. It happens when a framework or content system answers every unknown path with a page, and robots.txt was never created as a real file. The status says success; the body contains no directives; a crawler parses it, finds no rules, and treats the whole site as allowed. Whatever the owner believes the file says is not in force.

There are two versions of this. When robots.txt returns the site’s catch-all page, there is effectively no robots.txt at all; that shows up on 0.5% of scanned sites. When it returns some other HTML page, the same thing happens; that shows up on 0.1%. A plain-text body with the wrong content type is milder: most crawlers parse it anyway, and it shows up on 0.1%.

robots.txt redirects: followed, within limits #

RFC 9309 asks crawlers to follow at least five consecutive redirects for robots.txt. Google follows at least five and then stops, treating the file as a 404, which means no restrictions. A redirect from the bare host to www, or from http to https, is normal and harmless. What matters is the answer at the end of the chain, and whether that answer belongs to the host being crawled. A robots.txt that redirects to another domain's file is being read as that domain's rules.

robots.txt returns 404 or 410: everything is allowed #

A 404 or 410 on robots.txt means, under the standard, that the file is unavailable, and a crawler may access any resource on the server. That is the right outcome for a site that has no restrictions. It is the wrong outcome for a site that believes it has some: a file deleted in a migration, or placed at the wrong path, silently turns every restriction off.

A robots.txt that does not answer 200 shows up on 5.3% of scanned sites. The fix is to serve the file you mean with a 200, or to confirm that allowing everything is what you intended.

robots.txt returns 403 or 401: rules ignored by Google #

401 and 403 are 4xx codes, so by the letter of RFC 9309 they mean unavailable, and a crawler may fetch everything. Google documents the same treatment: its crawlers treat every 4xx except 429 as if no valid robots.txt existed, so a 403 and a 404 are equivalent and every rule in the file is ignored. Some crawlers and crawler libraries instead treat an authorisation refusal as a sign the site does not want to be crawled, and stop. A site that serves robots.txt as a 403 to a particular crawler, intending to keep it out, has under the standard published a permission, not a restriction.

Because the outcome depends on who is reading, treat a 401 or 403 on robots.txt as seriously as a server error. It shows up on 0.5% of scanned sites. Either way, the rules you wrote are not what the crawler read. This most often comes from a firewall or bot-management rule that challenges every client it does not recognise, robots.txt included.

robots.txt returns 429: treated as a server error #

A 429 means too many requests. It is a 4xx code, so the standard would allow a crawler to treat it as no file. Google does not: it treats a 429 on robots.txt as a server error, the same as a 5xx, and slows or stops crawling rather than assuming everything is allowed. A rate limit that catches robots.txt therefore has the opposite effect on Google from the one a 403 has. Exempt robots.txt from rate limits.

robots.txt returns 500 or 503: crawling stops #

A 500, 502 or 503 on robots.txt means the file is unreachable, and the standard says a crawler must assume complete disallow. A site that is otherwise working but returns 503 on robots.txt, for example because a maintenance rule or a rate limit catches that path, is telling every compliant crawler to stay away.

The standard allows a crawler to fall back to a copy it cached earlier, and after an outage lasting a long time, on the order of 30 days, to treat the file as unavailable instead. Google documents its own sequence: for the first 12 hours it stops crawling the site while it keeps retrying robots.txt, then for up to 30 days it uses the last version it fetched successfully, retrying a 503 fairly often. Neither helps a new site, and neither is something to rely on. Crawlers should also not keep a cached copy for more than 24 hours in normal operation, so a fix is picked up within about a day.

How to check yours #

Request /robots.txt on the bare URL, from outside your own network, with no cookies. Record four things: the status, the content type, the first line of the body, and the final host after any redirects. Then request it with a crawler's user-agent, because firewalls often answer crawlers differently from browsers. The file tools on the tools page run these reads and compare them.

If the answers differ between the browser and the crawler identity, the crawler's answer is the one that matters. The per-client differences, and why a 200 to your browser proves little, are covered in the guide to what status codes a crawler actually gets.

What to fix first #

A 5xx comes first, because it is the only answer that turns crawling off entirely. Then a 200 that returns HTML, because it silently turns your rules off. Then a 403 or 401 aimed at crawlers, because its outcome depends on the reader. Then a 404, which is harmless if intended and costly if not. Content type comes last. The live share of each is on the dataset page.

Share this finding:LinkedInXEmailFollow Emmanuel on LinkedIn

Related findings

Spotted a mistake or a bug on this page? Report an error. Confirmed errors are fixed and logged on the corrections page.

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain (how). Those appear at once, marked verified.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works