homepage refused
A homepage answering 401, 403, 429 or similar to this client while robots.txt is served normally. The machine layer stays scored; every section that needs the homepage is unmeasured rather than scored, and the report says why.
What homepage refused means
Reported as HOMEPAGE_REFUSED.
What homepage refused does not show
It is marked unconfirmed until checked from a second address, because a datacentre refusal may be about CrawlCheck rather than about crawlers.
How CrawlCheck measures homepage refused
One finding code measures it. Its share of all scans is computed from the live corpus when this page is served, not typed into it.
| Finding code | Share of scans | Guide |
|---|---|---|
HOMEPAGE_REFUSED | 1.7% | How to block AI crawlers and agents, and what each block costs you |
Every code, its rule and its current rate: the dataset page. Scan a site to see which apply to it.
Where homepage refused comes up on this site
- What AI crawlers get from Wix, Shopify, Squarespace and Webflow sites: 980 sites measuredOne Squarespace homepage refused all three AI crawlers with a plain 403 from a proxy in front of the site, again with a robots.txt that allowed them.
Terms this definition uses
robots.txt · machine layer · refusal
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · crawl budget · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about homepage refused
What is homepage refused?
A homepage answering 401, 403, 429 or similar to this client while robots.txt is served normally. The machine layer stays scored; every section that needs the homepage is unmeasured rather than scored, and the report says why.
What does homepage refused not show or guarantee?
The machine layer stays scored; every section that needs the homepage is unmeasured rather than scored, and the report says why. It is marked unconfirmed until checked from a second address, because a datacentre refusal may be about CrawlCheck rather than about crawlers.
How does CrawlCheck measure homepage refused?
Through the finding code HOMEPAGE_REFUSED, each with a stated rule, evidence bytes and a live share of scans on the dataset page.
Which area of the glossary does homepage refused belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← off-host redirect · FCrDNS →
See it in the full glossary · 731 terms across 19 areas.