Findings · 2026-10-22 · By VSNARY | Emmanuel Orta
One 503 is not a blocked site: refusal or hiccup?
A test scan read one 503 from a site's machine files and called the origin blocked. The same files answered 200 to six requests straight after. What a single refusal means, what a repeated one means, and how to tell them apart from your own desk.
A single 503 or 403 on a site's robots.txt, sitemap or llms.txt is not proof that the site refuses crawlers. Origins answer one request badly for ordinary reasons, such as a cold start, a deploy or a brief overload. A refusal that repeats when the same file is asked for again after a short pause is a real block; one that does not repeat is intermittent. To check by hand, request the file three times about a minute apart with a cache-busting query string and compare the status codes and response times.
One of CrawlCheck's findings says that a site's own server, behind its CDN, refuses the files crawlers read first: robots.txt, the sitemap, llms.txt. It is a serious finding, because a crawler that cannot read robots.txt often treats the whole site as off limits. It is also an accusation, and accusations need more than one reading.
What went wrong #
During a test scan of one of the sites we monitor, one request for one machine file came back 503. The finding fired. When we checked by hand, the same file answered 200 to six cache-busted requests in a row. The site was not blocking anything. Its server had answered one request badly, which servers do.
This was a test scan, not a published report, but the rule that produced it was live. A single bad answer was enough to accuse a site of refusing crawlers.
Why one bad answer happens #
A 503 means “try again later”. A server sends it during a deploy, while a cache is being rebuilt, when a worker process is starting up, or when it is briefly overloaded. A 403 can come from a security layer that challenged one request and let the next one through. None of these is a decision to keep crawlers out, and all of them clear on their own in seconds.
Our own hand-check showed the other common pattern. The same site's robots.txt and llms.txt took about three seconds on the first cache-busted request and about a third of a second on each one after. That is a cold origin waking up, not a refusal. A crawler with a short timeout could still give up on the first read, which is worth knowing, but it is a speed problem with a different fix.
What we changed #
When CrawlCheck's direct check of a machine file at the origin now comes back with anything other than a 200 or a 404, it waits a moment and asks again. If the second answer is a 200, the report records the first refusal as intermittent and raises no finding. If the refusal repeats, the finding stands, with both readings attached. A site is only called blocked when it said no twice.
What a crawler does with a refused robots.txt #
The robots.txt standard, RFC 9309, treats the two families of error differently. A 4xx answer such as 404 means the file is unavailable, and a crawler may proceed as if there were no rules. A 5xx answer such as 503 means the server could not be reached properly, and a crawler should assume it is not allowed to crawl anything until it can read the file again; a crawler holding an earlier copy may keep using that for a while. So one 503 at the wrong moment can pause crawling of the whole site, even though nothing about the site's policy changed.
That is why a hiccup still deserves attention, and why it deserves a different response from a block. A block is a setting to change. A hiccup is a reliability problem: the fix is to keep the file cached at the edge, so the crawler gets an answer even while the server restarts.
The three-request test you can run yourself #
You do not need our scanner to tell a refusal from a hiccup. Request the file three times, about a minute apart, adding a different query string each time (such as ?check=1, ?check=2, ?check=3) so that the CDN cannot answer from its cache and the request reaches your server. Note the status code and how long each request took. Most browsers' developer tools show both, and so does any command-line HTTP client.
| What you see | What it means | What to do |
|---|---|---|
| Refused all three times | a real block, at the origin or a security layer | find the rule that refuses it; what each status code tells a crawler explains the consequences |
| Refused once, then 200 | intermittent; deploys, restarts or brief overload | check the server's error log for that minute; repeat the test later |
| 200 every time, first one slow | a cold origin waking up | keep the file cached at the edge or warm, so a crawler with a short timeout is not the one that wakes it |
| 200 to a browser, refused to a crawler name | the server treats crawlers differently | compare by user agent; the delivery comparison guide shows how |
| 200 every time, fast | the file is fine | nothing to do |
Read the cache header too #
Most CDNs add a header saying whether the response came from their cache or from your server. The standard name is Cache-Status, and many providers use their own name for the same thing. A refusal that only appears when the request reaches your server, and never when the CDN answers from cache, points at the origin. A refusal served from the cache means the CDN stored a bad answer and keeps handing it out, which has a different fix: purge the file, and stop the CDN from caching error responses. Machine-file caching covers that case in detail.
Why this matters beyond one finding #
A scanner reads a site once. A crawler reads it many times. If CrawlCheck treats one bad answer as the site's policy, it will tell some owners they are blocking crawlers that are in fact reading them fine, and those owners will change settings that were working. A finding that accuses a site of something should be true the next time you look, and the simplest way to make sure of that is to look twice.
We hold accusing findings to a stricter standard than the rest of a report for that reason; our accuracy page lists which findings have been confirmed against live sites by hand and which are still marked advisory.
The main product
Found this on your own site? We fix it for $500.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Questions this post answers
Does a 503 on robots.txt mean my site blocks crawlers?
Not on its own. A 503 means “try again later” and often comes from a deploy, a restart or brief overload. If the same file answers 200 when requested again a minute later, the refusal was intermittent, not a policy.
How can I check whether my robots.txt is really refused?
Request it three times about a minute apart, adding a different query string each time so the CDN passes the request to your server. Refused all three times is a real block; refused once and then 200 is intermittent.
Why add a query string when testing?
A CDN can answer from its cache without asking your server. A unique query string makes it fetch from the origin, so you see what your server actually does.
What does a slow first response followed by fast ones mean?
Usually a cold origin: the server or its cache was idle and took a few seconds to wake. The file is not refused, but a crawler with a short timeout may give up on that first slow read.
How does CrawlCheck decide a site's origin refuses crawlers?
It asks for the file again after a short pause. A refusal that repeats is reported as a finding with both readings attached; a refusal followed by a 200 is recorded as intermittent and raises no finding.
What does a crawler do when robots.txt returns 503?
Under RFC 9309 a server error on robots.txt means the crawler should assume it may not crawl anything until it can read the file again, though it may keep using an earlier copy for a while. A 404 is treated as no rules at all.
What if the refusal only comes from the CDN's cache?
Then the CDN stored an error and keeps serving it. Purge the file and stop the CDN from caching error responses.
Related findings
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.