CrawlCheck

Guides · 2026-09-16 · By

Canonical URL audit: do tags, redirects and sitemap agree?

The canonical tag is one line, and most audits stop at it. The audit that matters counts how many addresses answer 200 for the same page, whether the sitemap and links agree, and whether the hostname is a site at all. With the curl commands to check each.

One page should have one address that answers 200, and every other way of reaching it (www or not, http or https, with or without a trailing slash) should redirect there in one hop. The canonical tag should name exactly that address, and the sitemap should list it and no other. The tag is a hint, not an instruction: when the redirects, internal links and sitemap disagree with it, search engines choose for themselves, and not always the address you meant.

Check your own site free

About a minute. No account, no card, nothing installed.

The canonical tag declares the one address a site wants treated as the page, and most audits check only that the tag exists. The failures that matter are elsewhere: two hostnames answering 200 for the same bytes, a sitemap listing a different form of the address, a template pointing a whole section's canonicals somewhere else, a domain that is only half an alias. Each is measurable from the responses, and none shows up in a tag check.

http://example.com/301 → https, same hosthttps://www.example.com/200, stays on wwwhttps://example.com/200, canonical = selfhttps://old-brand.com/homepage 301 → other hostone pagehousekeeping: finetwo hosts serve the pagethe address it namesalias, complete or half-done
A canonical audit is about how many addresses answer 200 for the same bytes, and whether every one of them agrees which address is the page. The shaded two need fixing or finishing.

Why is a tag check not an audit? #

Search engines weigh the canonical tag against redirects, internal links, sitemap entries and the addresses other sites link to, and when the signals disagree the tag loses. A page whose tag says https://example.com/services/, while the server answers 200 on the www form, the sitemap lists the version without the trailing slash and the nav links the www one, has declared four canonicals. Only one of them is in the tag. The question is not "is there a tag" but "do the tag, the response, the sitemap and the links all name the same address".

How often does each failure turn up? #

FailureHow to see itShare of scans
Both apex and www serve the pageThe www request stays on www and answers 200 with the page6.2%
Homepage ends on another hostThe redirect chain's final host differs from the one requested2.9%
The whole domain is an aliasHomepage, robots.txt, sitemap and other machine files all redirect to the same other host0.8%
A staging or preview host in the page sourceHostnames in the HTML that name a staging environment0.1%

Host duplicates: two addresses, one set of bytes #

The most common canonical failure is not on any page; it is the hostname. A site that answers 200 on both example.com and www.example.com, staying on whichever was requested, has published every page twice. The canonical tag can be correct on both copies and the duplicate still exists, because a crawler arriving through a link to the wrong host gets a full page instead of a redirect and has to decide which host is real. We had exactly this on our own site for months: the canonical was right and the site was still served twice.

The same failure hides behind schemes and slashes. http:// should 301 to https:// on the same host in one hop, never serve a page. A path should answer 200 with or without its trailing slash, never both.

Aliases and off-host redirects #

The opposite failure is a hostname with nothing of its own: an old brand domain, a defensive registration, a market domain pointed at the main site. When the homepage, robots.txt, the sitemap and the other machine files all redirect to the same host, the domain is a complete alias, and that is a valid setup. A domain that redirects its homepage but still serves its own robots.txt is half an alias: a crawler reading the file gets a policy for a site that does not exist there.

Between the two sits a site whose homepage ends on another host while the rest stays put. Check it by hand, because two explanations look identical from the status line: a migration in progress, or an alias that was only partly configured. The chain's final host says where the entity lives now; the canonical tags on the remaining pages say whether the site knows.

The canonical on every template #

A canonical pointing elsewhere is not wrong on its face: paginated lists, tracking parameters and print views legitimately point at a parent. But a page reached through the site's own navigation, answering 200, whose canonical names a different URL, has told crawlers it is not the page to index. Across a template that is usually one line in the theme, and it quietly removes a whole section from the index while every page still answers 200. Check one page per template, not one page per site.

Does the sitemap agree? #

The sitemap is the site's own list of canonical addresses, so it is the easiest place to catch a disagreement. Fetch each listed URL and compare its canonical with the entry. A listed URL whose canonical points elsewhere is declared twice with two answers; a listed URL that redirects means the sitemap is out of date. The sitemap audit guide covers that comparison starting from the file.

Parameters: addresses you did not mean to publish #

Every tracking parameter, session id and sort option that reaches a 200 page is a canonical decision made without noticing. With a self-referencing canonical it is a duplicate; with a canonical to the clean URL it is a hint the crawler may or may not take. The safe shape is a parameter that never produces a distinct 200 page: strip it at the edge or redirect it, and keep internal links on the clean address. The crawl budget guide shows how often sites' own links carry query strings.

Test it yourself #

Start with the hostname. Of the four homepage forms, three should land on the fourth in one hop, and an inner page requested on the wrong host should redirect rather than serve:

# The four homepage forms: final URL, status and hop count for each
for u in http://example.com/ http://www.example.com/ https://example.com/ https://www.example.com/; do
  curl -s -o /dev/null -L -w "$u -> %{url_effective} %{http_code} hops:%{num_redirects}\n" "$u"
done

# An inner page on the wrong host should redirect, not serve
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://www.example.com/services/

Then the page-level checks: the canonical names the landed URL exactly, scheme and slash included; only one slash form answers 200; a made-up parameter gets a redirect or a canonical to the clean URL, never a self-canonical; and the sitemap's entries are the same addresses.

# The canonical the landed page declares
curl -sL https://example.com/services/ | grep -io '<link[^>]*rel="canonical"[^>]*>'

# Trailing slash: one form answers 200, the other redirects
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://example.com/services
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' https://example.com/services/

# A made-up parameter: page, redirect, or page with a canonical to the clean URL?
curl -s 'https://example.com/services/?zz=test' | grep -io '<link[^>]*rel="canonical"[^>]*>'

# Sitemap entries to spot-check against the canonicals
curl -s https://example.com/sitemap.xml | grep -o '<loc>[^<]*' | head

Redirect chains are read hop by hop in the status-code guide. The free scan runs the host and alias checks on every domain it scans.

What to fix first #

  1. The host: one hostname answers, the other three forms redirect to it in one hop.
  2. Any off-host homepage redirect: a migration to finish or an alias to complete, machine files included.
  3. The template canonical, checked on one page per template.
  4. The sitemap, regenerated from the canonical addresses rather than edited by hand.
  5. Parameters: the most numerous and least damaging; the edge rule that strips them also cuts crawl waste.
Share of scanned sites with each finding in this articlewww and bare domain both serve the page6.2%Homepage redirects to another domain2.9%Domain is an alias of another0.8%Staging host linked from live pages0.1%Share of scanned sites with eachfinding in this articlewww and bare domain both serve the page6.2%Homepage redirects to another domain2.9%Domain is an alias of another0.8%Staging host linked from live pages0.1%
Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%.

The main product

Found this on your own site? We fix it for $100.

Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.

Free browser extension

Check this on any site you open, without leaving the page.

CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account.

Questions this post answers

Is a canonical tag enough to fix duplicate content?

No. The tag is a hint that search engines weigh against redirects, sitemap entries and internal links. If the server answers 200 on two hosts, or the sitemap lists a different address, the tag can be overruled. Make one address the only one that answers 200 and have every other signal name it.

How do I check whether www and non-www both serve my site?

Request the four homepage forms, http and https on the apex and on www, with curl following redirects. Three should end on the fourth after one hop. If both www and non-www end on themselves with a 200, the site is published twice.

What does it mean when a domain is an alias?

Every path on the domain, machine files included, redirects to the same other host. That is a valid configuration when it is complete, and a defect when only the homepage redirects while robots.txt or the sitemap still answer on the alias.

Why is a canonical pointing at another page sometimes correct?

Paginated lists, print views and parameterised copies legitimately point at a parent page. The failure is a page reached through the site's own navigation, answering 200, whose canonical points elsewhere across a whole template.

How do query parameters create canonical problems?

A parameter that produces a distinct 200 page with a self-referencing canonical is a duplicate. The safe shapes are a redirect to the clean address, or the page served with a canonical to the clean address while internal links never carry the parameter.

Should the canonical tag include the trailing slash?

It should match exactly the address the server answers 200 on, scheme, host and slash included. Pick one slash form, redirect the other to it, and use the same form in the sitemap and internal links.

Related findings

Comments

Comments are read before they appear. Nothing is published automatically, and no account is needed.

Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.

All guides · The dataset · How the dataset works