crawl budget
The effort a crawler will spend on a site before moving on.
What crawl budget does not show
Wasting it on redirect chains and unreachable URLs is measurable; the budget itself is not published by any operator.
Where crawl budget comes up on this site
- Crawl budget: measure what a crawler reads in 50 fetchesGoogle describes that crawl budget as a mix of how fast your server answers and how much demand there is for your pages, and says it matters most for large or fast-changing sites ( Google's crawl budget guide ).
- Canonical URL audit: do tags, redirects and sitemap agree?The crawl budget guide shows how often sites' own links carry query strings.
- Naming a crawler in robots.txt drops your Disallow rulesFor search crawlers that means crawl budget spent on duplicates and thin pages, and sometimes those pages appearing in results.
- Fake GPTBot and ClaudeBot requests scan for .env filesA forged user-agent is usually discussed as a statistics problem: someone inflating a crawl budget, or a scraper hiding behind a familiar name.
- Linked from every page, our llms.txt was fetched 8 timesA site with a larger crawl budget may see a different pattern, and so may a site that lists its machine files in its sitemap.
Terms this definition uses
Definitions that name crawl budget
Related terms in Crawlers and access
Who is fetching, whether they are who they claim, and what your rules actually permit.
visitor resolve · published crawler ranges · operator origin · signed agent · agent arrival · agent conduct record · agent traffic index · user-triggered fetch exemption · retrieval crawler · user-triggered fetch · verified crawler · forged crawler identity · unverifiable · robots.txt · user-agent group · AI opt-out · Content-Signal · cloaking · challenge page at 200 · uniform refusal · nonexistent-path control · blocked render resource · off-host redirect · homepage refused · FCrDNS · Crawler trap · Conditional request · ASN blocking · operator feed · residential proxy · Google-Extended · Google-Agent · GPTBot vs OAI-SearchBot · ClaudeBot vs Claude-User · PerplexityBot vs Perplexity-User · CCBot · Bytespider · agentic traffic · crawler classification · user agent · datacenter IP · vantage point · rate limit · delivery comparison · honeypot
Questions about crawl budget
What is crawl budget?
The effort a crawler will spend on a site before moving on.
What does crawl budget not show or guarantee?
Wasting it on redirect chains and unreachable URLs is measurable; the budget itself is not published by any operator.
Which area of the glossary does crawl budget belong to?
Crawlers and access: Who is fetching, whether they are who they claim, and what your rules actually permit.
← Content-Signal · cloaking →
See it in the full glossary · 772 terms across 19 areas.
Free browser extension
See this on any site you visit, in one click.
CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.