Robots unavailable
A robots.txt request that cannot be completed because of server or network failure.
What Robots unavailable does not show
RFC behavior depends on failure class and duration; it is not equivalent to an empty file.
How CrawlCheck measures Robots unavailable
2 finding codes measure it. Each one’s share of all scans is computed from the live corpus when this page is served, not typed into it.
| Finding code | Share of scans | Guide |
|---|---|---|
ROBOTS_NOT_200 | 6.1% | How to check whether AI crawlers and agents can read your site |
ROBOTS_BLOCKED | 0.4% | robots.txt 404, 403, 429 and 503: what each tells a crawler |
Every code, its rule and its current rate: the dataset page. Scan a site to see which apply to it.
Terms this definition uses
Related terms in Crawling and indexing
How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.
Crawler identity · Product token · Identification string · Crawl frontier · Crawl queue · Crawl scheduler · Seed URL · URL discovery · Link extraction · Recrawl interval · Crawl frequency · Host politeness · Crawl rate · Crawl-delay · Robots group merging · Longest-match rule · Wildcard rule · End-anchor rule · Robots cache · Robots parse error · Robots redirect · Sitemap · Sitemap URL set · Sitemap lastmod · Sitemap hreflang · Image sitemap · Video sitemap · News sitemap · Canonical cluster · Duplicate cluster · Index selection · Crawl demand · Rendering queue · Rendered HTML · Indexing pipeline · Index freshness · Index lag · Deindexing · Removal request · URL inspection · Crawl anomaly · Bot spoofing · Shared IP range · Web Bot Auth
Questions about Robots unavailable
What is Robots unavailable?
A robots.txt request that cannot be completed because of server or network failure.
What does Robots unavailable not show or guarantee?
A robots.txt request that cannot be completed because of server or network failure. RFC behavior depends on failure class and duration; it is not equivalent to an empty file.
How does CrawlCheck measure Robots unavailable?
Through the finding codes ROBOTS_NOT_200, ROBOTS_BLOCKED, each with a stated rule, evidence bytes and a live share of scans on the dataset page.
Which area of the glossary does Robots unavailable belong to?
Crawling and indexing: How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.
← Robots redirect · Sitemap →
See it in the full glossary · 731 terms across 19 areas.