Findings · 2026-08-29 · By VSNARY | Emmanuel Orta · 1 view
Our canonical tag was correct and the site was still served twice
Every page on this site existed at two addresses for months. The canonical pointed at the right one. That turns out not to be the part that matters.
An external audit of this site came back with a Health Score of 60 and four 404s. Two of the URLs were paths that do not exist for an anonymous visitor. The other two were the same two paths again, on a different hostname.
That second detail was the real finding. www.crawlcheck.io was answering 200 for every page on the site, serving a full copy alongside the apex. It had been doing that for as long as the domain has been live.
Why it looked fine #
Because on the measure everyone checks, it was fine. Every page carried a correct canonical tag pointing at the apex:
| Request | https://www.crawlcheck.io/guides |
| Status | 200 |
| Canonical | https://crawlcheck.io/guides |
Nothing in that is wrong. The canonical is accurate, it points where it should, a validator passes it. And it does not solve the problem, because a canonical tag is read after the response arrives. The crawler resolved the hostname, opened a connection, and received the full page. The tag then told it which address to prefer. The second fetch had already happened.
Sixty-five pages, two hosts. Every crawl of this site cost twice what it should, and half of the requests were spent confirming that the other half were the ones that counted.
Why it survives #
Three reasons, and they compound.
A browser never shows you. You type the domain without the prefix, you get the site, everything works. Nobody visits their own site by typing www. in front of it.
The tooling reports it as correct. A canonical checker looks at the tag and finds the tag is right. It is not asking the prior question, which is whether the request should have been served at all.
The default is to serve, not to redirect. Point a wildcard record or a proxied hostname at an origin and both names answer. Nothing has to be misconfigured for this to happen; it is what you get when nobody decides otherwise.
The fix #
One redirect, before routing:
| Before | www.crawlcheck.io/guides → 200, full page |
| After | www.crawlcheck.io/guides → 301 → crawlcheck.io/guides |
Verified on the root, /blog, /guides and /docs/api. One address per page.
The direction does not matter. Redirecting the apex to www is equally correct. What matters is that one of the two stops answering with content.
What we added to CrawlCheck #
This is exactly the class of defect CrawlCheck exists for: valid markup, healthy status codes, correct configuration by every local measure, and a crawler doing twice the work. So it is now a check.
CrawlCheck now requests the other hostname and reports it when it answers with a second copy of the site instead of redirecting. A host that redirects passes silently. A www host serving something genuinely different — a landing page, a legacy app — is not this defect and is not reported, and if you deliberately scan a www hostname, the duplicate is the address you asked about, so it is not reported back to you either.
Two results from the first runs #
Pointed at real sites, with the peer host confirmed by hand first:
| Domain | Apex | www | Drift | Fired |
|---|---|---|---|---|
| ahrefs.com | 200 | 301 to apex | — | no |
| sqlite.org | 8,886 bytes | 8,886 bytes, stays on www | 0% | yes |
| apache.org | 54,021 bytes | 54,021 bytes, stays on www | 0% | yes |
Byte-identical on both, in both cases. These are not obscure sites run by people who do not know what a redirect is — they are among the most linked-to domains on the web. The defect is not a competence problem. It is a default nobody revisited.
The uncomfortable part #
We shipped a scanner that reports what a machine actually receives, and we were serving our own site twice to every machine that asked. It went unnoticed because we checked the thing everyone checks — is the canonical correct — instead of the thing that decides the outcome, which is how many times the site answers.
The check exists now because we failed it. That is the only reason any of these checks exist.
Every figure above came out of CrawlCheck.
Point it at your own domain and see the same measurements, free.
The main product
Found this on your own site? We fix it for $749.
Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after.
Questions this post answers
Does a canonical tag stop duplicate content across www and the apex?
No. A canonical tag is read after the response arrives, so the crawler has already resolved the hostname, opened a connection and received the full page. The tag expresses a preference about which address to index; it does not prevent the second fetch.
How do I check whether my site is served on two hostnames?
Request your homepage on both the apex and the www hostname and compare the final URL after redirects. If both return 200 and the www request is still on www at the end, both hosts are serving the site. Byte counts within a few percent of each other confirm they are the same pages.
Should www redirect to the apex or the other way round?
Either is correct. What matters is that only one of the two answers with content and the other issues a 301 to it, so every page has a single address.
Related findings
Comments
Comments are read before they appear. Nothing is published automatically, and no account is needed.
Writing about this? Facts, live figures and marks — every number on that page is dated and traceable to a scan.