Agent view · Guides · 2026-09-23

robots.txt status codes: what a 404, 403, 429 or 503 tells Google and AI crawlers

This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/robots-txt-status-codes-what-404-403-and-503-tell-a-crawler: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.

Guides · 2026-09-23 · By [VSNARY | Emmanuel Orta](/author)

# robots.txt status codes: what a 404, 403, 429 or 503 tells Google and AI crawlers

The status code on robots.txt decides how a crawler treats the whole site before it reads a single rule. A 404 means crawl everything. A 503 means crawl nothing. A 403 is where crawlers disagree.

Under RFC 9309, a robots.txt that answers 2xx is obeyed. A 404, 410 or other 4xx means no file, so crawlers may fetch everything and your rules are not in force; Google treats 401 and 403 the same way. Google treats a 429 like a server error. A 5xx means crawlers must assume everything is disallowed: Google stops crawling for 12 hours, then uses its last good copy for up to 30 days. A 200 that returns HTML is read as a file with no rules.

Check your own site free

Check my site

About a minute. No account, no card, nothing installed.

On this page

- The short version
- robots.txt returns 200: the only answer that publishes rules
- A 200 that is not a robots.txt
- robots.txt redirects: followed, within limits
- robots.txt returns 404 or 410: everything is allowed
- robots.txt returns 403 or 401: rules ignored by Google
- robots.txt returns 429: treated as a server error
- robots.txt returns 500 or 503: crawling stops
- How to check yours
- What to fix first

Every crawler that respects [robots.txt](/glossary/robots-txt) fetches it before anything else, and the first thing it reads is not a rule. It is the status code. The standard that governs the file, RFC 9309, says what each class of status means, and the meanings are not the ones most site owners expect. A missing file does not mean no crawling. A server error does not mean try again as normal. This guide goes through them in the order a crawler meets them.

## The short version #

| robots.txt answers | RFC 9309 calls it | A standards-following crawler | What a scan reports |
| --- | --- | --- | --- |
| 200, text/plain, rules | successful access | obeys the rules for its own group | rules parsed |
| 200, HTML page | parse finds no rules | treats every path as allowed | robots.txt is an HTML page, or the site’s catch-all page |
| 200, other content type | successful access | usually parses it | wrong content type |
| 3xx | redirect | follows at least five hops | final answer judged |
| 404, 410, most other 4xx | unavailable | may fetch everything | robots.txt does not answer 200 |
| 429 | unavailable (4xx) | standard: may fetch everything; Google: treated as a server error | robots.txt does not answer 200 |
| 401, 403 | unavailable (4xx) | standard: may fetch everything; some treat as [refusal](/glossary/refusal) | robots.txt refused |
| 500, 502, 503 | unreachable | must assume everything is disallowed | robots.txt refused or erroring |

## robots.txt returns 200: the only answer that publishes rules #

A 200 with a plain-text body is the file doing its job. The crawler finds the group that names it, or the `*` group if none does, and obeys only that group. That last part catches many sites: a named group does not inherit the rules under `*`, which is explained in [the guide to whether AI crawlers respect robots.txt](/guides/do-ai-crawlers-respect-robots-txt).

The standard asks crawlers to parse at least the first 500 kibibytes of the file. A larger file is not an error, but rules past that point may never be read. Keep the file short enough that the limit never matters.

## A 200 that is not a robots.txt #

The most expensive answer is a 200 that carries an HTML page. It happens when a framework or content system answers every unknown path with a page, and robots.txt was never created as a real file. The status says success; the body contains no directives; a crawler parses it, finds no rules, and treats the whole site as allowed. Whatever the owner believes the file says is not in force.

There are two versions of this. When robots.txt returns the site’s catch-all page, there is effectively no robots.txt at all; that shows up on 0.5% of scanned sites. When it returns some other HTML page, the same thing happens; that shows up on 0.1% . A plain-text body with the wrong content type is milder: most crawlers parse it anyway, and it shows up on 0.1% .

## robots.txt redirects: followed, within limits #

RFC 9309 asks crawlers to follow at least five consecutive redirects for robots.txt. Google follows at least five and then stops, treating the file as a 404, which means no restrictions. A redirect from the bare host to `www`, or from http to https, is normal and harmless. What matters is the answer at the end of the chain, and whether that answer belongs to the host being crawled. A robots.txt that redirects to another domain's file is being read as that domain's rules.

## robots.txt returns 404 or 410: everything is allowed #

A 404 or 410 on robots.txt means, under the standard, that the file is unavailable, and a crawler may access any resource on the server. That is the right outcome for a site that has no restrictions. It is the wrong outcome for a site that believes it has some: a file deleted in a migration, or placed at the wrong path, silently turns every restriction off.

A robots.txt that does not answer 200 shows up on 5.3% of scanned sites. The fix is to serve the file you mean with a 200, or to confirm that allowing everything is what you intended.

## robots.txt returns 403 or 401: rules ignored by Google #

401 and 403 are 4xx codes, so by the letter of RFC 9309 they mean unavailable, and a crawler may fetch everything. Google documents the same treatment: its crawlers treat every 4xx except 429 as if no valid robots.txt existed, so a 403 and a 404 are equivalent and every rule in the file is ignored. Some crawlers and crawler libraries instead treat an authorisation refusal as a sign the site does not want to be crawled, and stop. A site that serves robots.txt as a 403 to a particular crawler, intending to keep it out, has under the standard published a permission, not a restriction.

Because the outcome depends on who is reading, treat a 401 or 403 on robots.txt as seriously as a server error. It shows up on 0.5% of scanned sites. Either way, the rules you wrote are not what the crawler read. This most often comes from a firewall or bot-management rule that challenges every client it does not recognise, robots.txt included.

## robots.txt returns 429: treated as a server error #

A 429 means too many requests. It is a 4xx code, so the standard would allow a crawler to treat it as no file. Google does not: it treats a 429 on robots.txt as a server error, the same as a 5xx, and slows or stops crawling rather than assuming everything is allowed. A [rate limit](/glossary/rate-limit) that catches robots.txt therefore has the opposite effect on Google from the one a 403 has. Exempt robots.txt from rate limits.

## robots.txt returns 500 or 503: crawling stops #

A 500, 502 or 503 on robots.txt means the file is unreachable, and the standard says a crawler must assume complete disallow. A site that is otherwise working but returns 503 on robots.txt, for example because a maintenance rule or a rate limit catches that path, is telling every compliant crawler to stay away.

The standard allows a crawler to fall back to a copy it cached earlier, and after an outage lasting a long time, on the order of 30 days, to treat the file as unavailable instead. Google documents its own sequence: for the first 12 hours it stops crawling the site while it keeps retrying robots.txt, then for up to 30 days it uses the last version it fetched successfully, retrying a 503 fairly often. Neither helps a new site, and neither is something to rely on. Crawlers should also not keep a cached copy for more than 24 hours in normal operation, so a fix is picked up within about a day.

## How to check yours #

Request `/robots.txt` on the bare [URL](/glossary/url), from outside your own network, with no cookies. Record four things: the status, the content type, the first line of the body, and the final host after any redirects. Then request it with a crawler's user-agent, because firewalls often answer crawlers differently from browsers. The file tools on [the tools page](/tools) run these reads and compare them.

If the answers differ between the browser and the crawler identity, the crawler's answer is the one that matters. The per-client differences, and why a 200 to your browser proves little, are covered in [the guide to what status codes a crawler actually gets](/guides/http-status-codes-what-a-crawler-actually-gets).

## What to fix first #

A 5xx comes first, because it is the only answer that turns crawling off entirely. Then a 200 that returns HTML, because it silently turns your rules off. Then a 403 or 401 aimed at crawlers, because its outcome depends on the reader. Then a 404, which is harmless if intended and costly if not. [Content type](/glossary/content-type) comes last. The live share of each is on [the dataset page](/data).
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

Written by

[Emmanuel Orta](/author) · VSNARY

I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post.

[Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta)

Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Frobots-txt-status-codes-what-404-403-and-503-tell-a-crawler%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Frobots-txt-status-codes-what-404-403-and-503-tell-a-crawler%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=robots.txt%20status%20codes%3A%20what%20a%20404%2C%20403%2C%20429%20or%20503%20tells%20Google%20and%20AI%20crawlers)[Email](mailto:?subject=robots.txt%20status%20codes%3A%20what%20a%20404%2C%20403%2C%20429%20or%20503%20tells%20Google%20and%20AI%20crawlers&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Frobots-txt-status-codes-what-404-403-and-503-tell-a-crawler)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta)

## Related findings

[2026-10-22 One 503 is not a blocked site: refusal or hiccup? Why one 503 on robots.txt or a sitemap is not proof a site blocks crawlers, and a three-request test that tells a refusal from a h](/blog/one-503-is-not-a-blocked-site-how-to-tell-a-refusal-from-a-hiccup)[2026-10-26 Google-Extended and Applebot-Extended: block AI training Google-Extended and Applebot-Extended are tokens, not crawlers: they opt a site out of AI training and keep it in search. The line](/guides/google-extended-and-applebot-extended-opt-out-of-ai-training-keep-search)[2026-10-30 Unblocking AI crawlers: what an access repair proves Twelve robots.txt parser cases prove which rules apply to a named crawler. They do not prove access, citations or traffic. What to](/blog/unblocking-ai-crawlers-access-is-not-a-citation-result)

The main product

Found this on your own site? **We fix it for $100.**

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

[Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v)
[What it covers](/pricing#fix) · [scan first, it is free](/)

Free browser extension

Check this on any site you open, **without leaving the page.**

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

[Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post)
[What it does](/extension)

Write for the library

Measured something yourself? **Submit an article.**

Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit, and placement is never for sale.

[Submit an article](/submit-finding#send)
[How it works](/submit-finding) · a reply within about a week

Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections).

## Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*.

Send for review

Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan.

[All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy)

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [Keys](/keys) · [Listing check](/listing-check) · [Entity map validator](/entity-map-validator) | **Registry:** [MCP](/registry/mcp) · [patronus.ai](/registry/patronus.ai) · [selectel.ru](/registry/selectel.ru) · [trigger.dev](/registry/trigger.dev) · [decouvrant.fr](/registry/capabilities/decouvrant.fr) | **Certificates:** [crawlcheck.io](/certificate/crawlcheck.io) · [mostly.ai](/certificate/mostly.ai) · [turso.tech](/certificate/turso.tech) | **Datasets:** [Technologies: Microsoft clarity](/datasets/technologies/microsoft-clarity)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Report an error](/report-error)

Same body as a file: /guides/robots-txt-status-codes-what-404-403-and-503-tell-a-crawler.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.