Crawl demand
An engine’s estimated value in revisiting a URL based on importance, expected change, and other signals.
What Crawl demand does not show
It is inferred externally, not published as a per-page score.
Where Crawl demand comes up on this site
- Is your host blocking ClaudeBot? Group the 403s by IP firstA few dozen refusals on one small file explains more lost coverage than any amount of theorising about crawl demand.
Related terms in Crawling and indexing
How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.
Sitemap directive · Crawler identity · Product token · Identification string · Crawl frontier · Crawl queue · Crawl scheduler · Seed URL · URL discovery · Link extraction · Recrawl interval · Crawl frequency · Host politeness · Crawl rate · Crawl-delay · Robots group merging · Longest-match rule · Wildcard rule · End-anchor rule · Robots cache · Robots parse error · Robots redirect · Robots unavailable · Sitemap · Sitemap URL set · Sitemap lastmod · Sitemap hreflang · Image sitemap · Video sitemap · News sitemap · Canonical cluster · Duplicate cluster · Index selection · Rendering queue · Rendered HTML · Indexing pipeline · Index freshness · Index lag · Deindexing · Removal request · URL inspection · Crawl anomaly · Bot spoofing · Shared IP range · Web Bot Auth
Questions about Crawl demand
What is Crawl demand?
An engine’s estimated value in revisiting a URL based on importance, expected change, and other signals.
What does Crawl demand not show or guarantee?
It is inferred externally, not published as a per-page score.
Which area of the glossary does Crawl demand belong to?
Crawling and indexing: How a crawler discovers, schedules, fetches and selects pages, and why a fetched page is not an indexed one.
← Index selection · Rendering queue →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.