Guides · 2026-10-18 · By VSNARY | Emmanuel Orta

Crawl budget: measure what a crawler reads in 50 fetches

A crawler spends a limited number of fetches and leaves. How to measure what yours buys for 25, 50 or 100 fetches, which pages come last, and where fetches are wasted, with the waste signals measured on 111 sites.

A search engine or AI crawler does not read a whole site every visit. It spends a limited number of fetches, then leaves. Google describes that crawl budget as a mix of how fast your server answers and how much demand there is for your pages, and says it matters most for large or fast-changing sites (Google's crawl budget guide). Small sites feel it too: a visit spent on broken links and redirects never reaches the pages you care about. You cannot see a crawler's budget from outside, but you can measure what any budget buys.

What does a fetch buy? #

Every URL a crawler requests costs one fetch, whether it answers with a real page, a 404, a redirect or a page marked noindex. Only some fetches end at a useful page: one that answers 200, carries no noindex, and whose canonical points to itself. The question is never how many pages you have. It is how many useful pages a crawler reaches before its fetches run out, and in what order.

A crawler finds pages two ways: by following links from pages it has read, starting at the homepage, or by reading your sitemap and queueing every address it lists. Most do both, and the difference is diagnostic. Link-following reaches a page late when it sits many clicks deep, and never reaches it when nothing links to it. The sitemap wastes fetches when it lists addresses that redirect, return 404 or are marked noindex.

How do I measure it? #

Take every page your sitemap declares, or follow links from the homepage if there is no sitemap, and record which pages link to which. Then replay three crawlers over that map, one fetch per URL, no refetches:

For budgets of 10, 25, 50 and 100 fetches, count the useful pages each one reaches and the fetches it spends elsewhere. Note the pages link-following reaches last, because those are the first lost when a crawler spends less, and every sitemap entry or link that cost a fetch without finding a useful page.

If you extract links from HTML yourself, decode HTML entities in each href before you parse it. Ampersands in attributes are written &, and splitting the raw value on & produces parameter names like amp;utm_medium that do not exist. We shipped exactly that mistake in our own crawl-waste readout for a week: the counts were right, but the names a person reads to decide what to fix were not. Test with links copied from real pages, not typed by hand.

What two real sites showed #

A tree-service site we operate (77 pages, every one in its sitemap and linked from its navigation) is the clean case. Link-following reached 10, 25 and 50 useful pages for 10, 25 and 50 fetches, and all 77 by the 78th fetch.

A 154-page personal blog was the instructive one. Reading the sitemap first reached all 154 useful pages in 172 fetches. Following links reached 149 in 250 fetches, and 5 useful pages could not be reached by any link. Eighteen fetches went to addresses that answered 404: links inside posts with a typo, such as a missing slash or a pasted fragment of another URL, none of them visible to a reader skimming the page.

Where do sites waste fetches? #

On 16 September 2026 we read the crawl-waste signals on 111 scanned sites. The table is a one-day reading, not a live rate; current figures are on the dataset page. The site with the most query-string links was checked by hand against an independent fetch.

The redirecting-sitemap row matters more than its share suggests. It is usually caused by something nobody considers a defect, a trailing-slash rule or an http-to-https upgrade applied after the sitemap generator ran, and it costs a fetch on every visit. The apex-and-www row hides in plain sight too: this site carried it for months with a correct canonical tag.

What to fix, in order #

1. Remove dead addresses from the sitemap. An entry that redirects or returns 404 costs a fetch every visit and finds nothing.
2. Fix broken internal links. The internal link audit guide shows how to find them.
3. Link the unreachable pages from a page that is already linked, so a crawler that ignores the sitemap can still find them.
4. Shorten the path to important pages. A page five clicks deep is fetched after everything shallower.
5. Keep internal links on the canonical address: no query strings, no utm_ tags, no http://, one host.
6. Keep the sitemap accurate. The XML sitemap audit covers status, content type and stale entries.

Doing it by hand #

On a small site, any crawler tool that exports one row per URL will do. Export status, indexability, canonical and clicks from the homepage for every page, then sort by clicks. Pages nothing links to, pages five or more clicks deep, and sitemap entries that do not answer 200 are the list to fix. The free scan checks your homepage, robots.txt and sitemaps.

What this does not measure #

Real crawlers also spend budget on refetching pages they know, on images and scripts, and on links found elsewhere on the web. So the replay is a best case: a crawler with 50 fetches gets at most what it shows. It cannot tell you a crawler's actual budget for your site, which the crawler sets; Google says it adjusts it to what your server can handle (Google's crawler overview).

Questions this post answers

What is crawl budget?

The number of pages a crawler is willing to fetch from a site in a visit. Google sets it from how fast the server answers and how much demand there is for the pages.

Does crawl budget matter for a small site?

Less than for a large one, but a small site can still waste it: every link to a 404, every redirect and every noindex page in the sitemap costs a fetch that finds nothing.

How can I see which pages a crawler reaches first?

Replay a crawl over your own link graph. Following links from the homepage, nearest first, gives the order; the pages reached last are the first ones lost when a crawler spends less.

Why does reading the sitemap reach more pages than following links?

A sitemap lists pages directly, including ones no link points to. Following links can only reach pages that something links to.

What wastes crawl budget most often?

Sitemap entries that redirect or return 404, broken internal links, internal links carrying query strings or tracking parameters, and both www and non-www serving pages. Each costs fetches that find no new useful page.

Why do I see parameter names starting with amp; in my link report?

The links were split on the ampersand before their HTML entities were decoded. Ampersands in an href are written as &, so decode the attribute first and the stray amp; prefixes disappear.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works