Guides · 2026-09-16 · By VSNARY | Emmanuel Orta

XML sitemap audit: check the file and every URL in it

A sitemap can return 200, open in a browser and still be unusable. The audit has two layers, verify the document and then verify every URL it declares, and most audits stop after the first.

A sitemap can return 200, open in a browser, look orderly under the browser's XML stylesheet, and still be unusable. The response may be an HTML login wall, a cached error page, a challenge interstitial, malformed XML, or a clean list of URLs that redirect, canonicalize elsewhere, carry noindex, or no longer exist. A sitemap is not healthy because a validator found angle brackets. It is healthy when its bytes parse and every entry expresses the same canonical plan as the site it describes.

The audit has two layers, and most of them stop after the first: verify the document, then verify the population it declares. The first proves a file exists. Only the second proves the file tells crawlers the truth. This guide covers both, with the failure rate for each layer read live from the scans in the public dataset, so you can see how often each check fires before you run it.

The short answer #

A useful XML sitemap returns a successful response with parseable XML, contains absolute canonical URLs on one host and one protocol, stays within the published size limits, and lists only pages you want discovered. Every listed URL resolves directly to 200, is crawlable and indexable, and declares itself canonical. It is named in robots.txt with an absolute Sitemap: line that does not itself redirect.

Neither the file nor its declaration guarantees indexing. A sitemap is a discovery and canonicalization signal. It is not permission, not a quality judgement, and not evidence that an answer engine will cite a page.

How often each layer fails, measured #

The rates below are the share of scans in the public dataset carrying each failure, read live when this page loads rather than typed into it. First what each failure means, then the live rates in the same order.

Live share of scanned sites: 10.2% (live share of scanned sites: 8.6%) (live share of scanned sites: 0.9%) (live share of scanned sites: 0.6%) (live share of scanned sites: 0.4%) (live share of scanned sites: 0.5%) (live share of scanned sites: 2%) (live share of scanned sites: 0.2%) (live share of scanned sites: 6.7%).

Read the broken-declaration and blocked-sitemap rows together. A site can name its sitemap correctly and still hand a crawler nothing, either because the named URL is dead or because a Disallow line three lines below forbids the path the Sitemap: line just recommended. Both halves of that file are individually valid. Only something that reads the file in order and follows what it says can see the contradiction, which is why the robots.txt guide spends as long on discovery as on blocking.

Find every sitemap #

Start with robots.txt, because that is where a crawler starts:

curl -sS https://example.com/robots.txt | grep -i '^sitemap:'

Then probe the conventional locations, treating them as discovery aids rather than as the official file. A CMS often leaves an old sitemap at a common path while publishing a newer index somewhere else, and both will answer.

for path in /sitemap.xml /sitemap_index.xml /sitemap-index.xml /wp-sitemap.xml; do
  curl -sS -o /dev/null -w "$path | %{http_code} | %{content_type} | %{size_download}\n" "https://example.com$path"
done

Record the declarations on every host you own: apex, www, and any subdomain that serves pages. A migration commonly updates the public pages and leaves robots.txt on one host naming a sitemap on another. If there is no declaration at all, add one, absolute, on the final canonical host and protocol. The declaration itself should never redirect.

Verify the actual response, then tell it apart from a wall #

Fetch the headers and the body separately and read both:

curl -sS -D /tmp/sitemap.headers -o /tmp/sitemap.xml https://example.com/sitemap.xml
cat /tmp/sitemap.headers
head -c 500 /tmp/sitemap.xml

Note the status and any redirect chain, the final URL, the Content-Type, the byte count, the cache headers, and the first meaningful token of the body. A 200 with text/html is not automatically wrong, since some servers mislabel XML, but it is a warning that requires parsing the body. If the first token is <!DOCTYPE html> you are not reading a sitemap, whatever the status said.

Then run the one check that separates a sitemap from a firewall wearing its URL: request a path that cannot exist and compare.

curl -sS -L -o /dev/null -w '%{http_code} %{content_type} %{size_download}\n' https://example.com/not-real-sitemap-91a7.xml

If the impossible path returns the same status, type and approximate size as /sitemap.xml, an application fallback or a bot wall is answering everything, and nothing you measured through it describes the site. A CrawlCheck scan sends the same control, so its report tells you when a wall answered in place of your site instead of describing the wall as if it were the site. A file that answered 200 and could not be read is the case that made the control mandatory.

Parse, do not eyeball #

A browser applies a stylesheet to XML and makes malformed or mislabeled content look orderly. Parse the downloaded bytes instead:

xmllint --noout /tmp/sitemap.xml
xmllint --xpath 'name(/*)' /tmp/sitemap.xml; echo

An ordinary sitemap has a urlset root; an index has sitemapindex. The common syntax defects are bare ampersands in query strings where & was required, control characters, generation truncated by a timeout, HTML error text inserted mid-document, relative URLs, and malformed dates. Every one of those renders fine in a browser and fails in a parser.

Audit an index by fetching every child #

An index delegates to child files, and an index audit that stops at the index has audited a table of contents. Extract the locations and fetch each one:

xmllint --xpath '//*[local-name()="sitemap"]/*[local-name()="loc"]/text()' /tmp/sitemap.xml | tr ' ' '\n'

For each child, record status, final URL, type, bytes and parse result. Flag children that redirect, return 404, 403, 429 or a 5xx, return HTML, are empty, name a retired host, or duplicate another child's URL set. Compressed children still need valid XML after decompression, and large files must be split before the published limits rather than after a generator starts truncating under load.

The redirect that made one sitemap the slowest file on its site #

A detail from a site we operate, found while reading its speed section rather than its sitemap section. The site's /sitemap.xml is a WordPress 301 to /sitemap_index.xml. The index itself answers in 10 to 16 milliseconds from cache. The redirect in front of it is uncached at both the CDN and the origin's page cache, so every request for the conventional path pays a full origin round trip: 954, 1,562 and 1,138 milliseconds on three consecutive scans. The sitemap was correct. The path everyone types to reach it was the one slow file on the domain, and it flipped a speed row on and off for a week before anyone read which file was doing it.

Two lessons travel. A redirect is a fetch that returns nothing, which is why a listed URL that redirects should be replaced with its destination. And the conventional path is a real URL that real crawlers request, so it deserves the same cache treatment as the file it forwards to.

Extract, normalize, then request every listed URL #

Pull the locations out with a namespace-aware parser, then look for inventory defects before you fetch anything:

xmllint --xpath '//*[local-name()="url"]/*[local-name()="loc"]/text()' /tmp/sitemap.xml | tr ' ' '\n' > urls.txt
grep -E '^http://' urls.txt
grep -E '://www\.' urls.txt
grep -E '\?.*(utm_|session|sort=)' urls.txt
sort urls.txt | uniq -d

Every URL should be absolute, on the canonical host and protocol, with one trailing-slash policy, no fragments, no tracking or session parameters, and no staging or preview hosts. Do not rewrite entries blindly; a parameter can identify a genuinely distinct resource, so classify its function before removing it.

Then leave the XML and check the pages, because that is where a sitemap audit becomes useful:

while IFS= read -r url; do
  curl -sS -L -o /dev/null -w "$url\t%{http_code}\t%{url_effective}\t%{content_type}\t%{num_redirects}\n" "$url"
done < urls.txt > sitemap-status.tsv

Rate-limit responsibly, identify yourself honestly, and run from infrastructure you control. Flag any non-200 final status, any redirect from a listed URL, suspiciously tiny or identical bodies, X-Robots-Tag: noindex or a meta noindex, a robots.txt disallow for the crawlers you want, and any canonical pointing elsewhere. A listed URL that 404s should be removed unless it is about to be restored. A listed URL returning 403 needs a layer diagnosis, not deletion; removing an important page to make an error count disappear is how sites end up with a clean sitemap and no landing pages.

A free scan samples the declared URLs and checks each for status, redirect, and whether what came back is an HTML page at all.

Compare canonicals #

For each 200 HTML page, extract the canonical. The expected relation is exact:

sitemap URL = final URL = HTML canonical

Persistent differences weaken the file as a canonical signal. The failure classes worth checking by name: a missing canonical on an indexable page, a relative or malformed one, a canonical to a redirected or 404 URL, a cross-domain canonical copied in with a template, every paginated page canonicalized to page one, localized pages canonicalized to the default language, and product variants canonicalized to a materially different parent. Fix the page template and the sitemap generator from the same source of truth. When two systems construct URLs independently, drift returns.

Compare the sitemap with the site #

A sitemap can be internally clean and still omit what matters. Compare three sets: what the sitemaps declare, what internal links reach, and what receives impressions, crawler visits or external links. The differences have names. Orphans are declared but never linked. Undeclared pages are linked and indexable but absent from the file. Ghosts are declared but redirected, removed or excluded. Unreachable business pages exist in the CMS and in neither set.

The over-promising direction is the one a validator cannot see. On four home-service sites we operate, the structured data declared 19 to 23 service areas each; three of the four had a real landing page for one or seven of them. The schema claimed coverage the sitemap could not back, and the pages are the half a search engine can land on. Whatever else lives on the domain, llms.txt and any entity map included, should not recommend URLs the sitemap and the pages reject.

Dates and freshness #

Use lastmod only when it records a meaningful content change: substantive revision, a change to product facts or availability, a corrected policy, a material structured-data fix. Regeneration, a footer change applied site-wide, an analytics script swap and a cache refresh are not changes to the page. Stamping every URL with the build clock destroys the field's only information value, and a scanner that reads it will count the whole population as changed today.

Derive the date from the content record. If that cannot be done reliably, omit the field; a missing date is honest and a false one is not. Keep visible dateModified, the structured-data date and the sitemap date consistent where they describe the same event.

Diagnose generator ownership before fixing entries #

Identify what writes the file: CMS core, an SEO plugin, the commerce platform, a framework build, a server route, an edge script, a database job or an external service. Trace each field to its source. The host and protocol usually come from an environment variable, inclusion from publication status, lastmod from a generic row timestamp, exclusions from a plugin toggle. Then write tests at the generator boundary: no forbidden host, no non-canonical protocol, no duplicate <loc>, no excluded content type, XML that parses, child counts within limits, and a sample of output URLs that returns 200 and self-canonicalizes. The goal is to stop bad XML deploying, not to find it weeks later on a dashboard.

Monitor it as a contract #

Two schedules. A frequent file-level check for status, type, parse validity, child availability and a hash of the bytes. A slower URL-level crawl for status, redirect, robots, indexability and canonical agreement. Alert on transitions rather than states: XML to HTML, 200 to an error, a child count dropping, a URL count jumping past a release, the canonical host changing, new redirects or noindex entries, an old host reappearing, lastmod resetting across the whole population. Store the exact bytes and the date so a change can be traced to the release that caused it.

Fix order #

1. Make the official sitemap URL return the real, parseable file, with no redirect in front of it.
2. Repair broken child sitemaps.
3. Remove staging, foreign-host and malformed URLs.
4. Replace redirected entries with their final URLs.
5. Remove 404, 410 and intentionally excluded pages.
6. Resolve noindex, robots and canonical conflicts.
7. Add the missing canonical business pages.
8. Repair internal links and orphan architecture.
9. Make lastmod honest.
10. Add generation tests and monitoring.

The order repairs the instrument before using it to evaluate the inventory.

Run the same test on your site #

The free scan fetches robots.txt and every sitemap it names on both the apex and www, checks the response against a path that cannot exist, samples the declared URLs, and reports each of the failures in the table above in plain words. Open the sitemap evidence in the report and verify the population, not the status. If the same pages also disagree on their canonical, the canonical URL audit covers that layer.

Findings behind this #

Every figure above comes from a measurement written up in full, with the date, the method and the raw numbers.

• robots.txt returned 200 and no crawler could read it: challenge pages
• Your schema lists service areas your sitemap has pages for a fraction of them
• Our canonical tag was correct and the site was still served twice

Questions this post answers

What should an XML sitemap contain?

Absolute canonical URLs for the pages you want discovered, on one host and one protocol. Each listed page should return 200, be crawlable and indexable, and declare itself canonical.

Is a 200 status enough for a sitemap?

No. A 200 can carry an HTML login page, a challenge page or a catch-all fallback. Read the content type, the first bytes of the body and the XML parse result, and compare the response with a request for a path that cannot exist.

Should redirected URLs be in a sitemap?

Usually not. A redirect is a fetch that returns nothing, so replace each redirected entry with its final canonical destination and make sure the sitemap URL itself does not redirect either.

Does a sitemap guarantee indexing or AI citations?

No. It helps discovery and supplies a canonical signal. The page still needs access, readable content, consistent indexing directives and enough value to be selected.

Should every page carry today's lastmod?

No. Use the date of a meaningful change to that page. Resetting every URL on each build makes the field worthless, and omitting it is more honest than publishing a false date.

Where should the sitemap be declared?

In robots.txt with an absolute Sitemap: line on the final host and protocol, and submitted through the search tools you use. The declared URL must resolve directly, and the same robots.txt must not disallow its path.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works