[Datasets](/datasets) · [by problem](/datasets/problems)

# www and the apex both serve the site instead of one redirecting

Both www and the bare domain answer 200 with the same pages, so every URL on this site exists at two addresses and a crawler fetches the whole site twice. A canonical tag does not prevent this: it is read after the response arrives, so the second fetch has already been spent. Nothing looks broken from a browser, which is why this survives for years. The fix is a 301 from one host to the other so there is one address per page. 13,579 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

- **13,579** sites, measured in the last 7 days
- **5939** US local businesses
- **2026-11-22** last counted, recounted every hour
- **$299** once, about 2.2¢ a row, 30 days of downloads

[Buy the list — $299](/credits?item=prob_host-duplicate-200#buy)[Free sample of this list (CSV)](/api/datasets/sample?problem=host-duplicate-200)[Other problems](/datasets/problems) · [all lanes, $899/month](/datasets#plan-all)

**At a glance:** 87% of these sites show no CMS CrawlCheck could detect, and the most common grade among them is D (49%). [How CrawlCheck measures](/research)

Who has it

Platform

**No CMS detected** 87% **Next.js** 5% **WordPress** 4% **Drupal** 2% **Webflow** 1% **Squarespace** 1%

Grade

**D** 49% **C** 44% **F** 7%

Where

**US local businesses** 44% **SaaS and software** 26% **Government and education** 16% **Wider corpus** 14%

| Where | Category | Grade | Platform | AI crawlers |
| --- | --- | --- | --- | --- |
| SaaS and software | Other | C | No CMS detected | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Saas | C | No CMS detected | allowed |
| SaaS and software | Other | C | No CMS detected | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Other | F | No CMS detected | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |
| SaaS and software | Other | D | No CMS detected | allowed |

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: [what a row holds](/datasets#fields) · [free sample of this list](/api/datasets/sample?problem=host-duplicate-200) · a problem, country or industry we do not list: [order a custom dataset](/datasets#order).

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [Listing check](/listing-check) · [SDK](/sdk) · [Certificates](/certificates) · [Conduct: Duckduckbot](/agents/conduct/duckduckbot) | **Registry:** [MCP: Operator](/registry/mcp/operator) · [Trade: Phoenix](/registry/trade/roofing/az/phoenix) · [studioxnews.ca](/registry/capabilities/studioxnews.ca) | **Datasets:** [Technologies: Stripe](/datasets/technologies/stripe) · [Problems: Host duplicate 200](/datasets/problems/host-duplicate-200) | **AI crawlers:** [Ccbot vs GPTBot](/ai-crawlers/compare/ccbot-vs-gptbot) | **Blog:** [Six in ten websites already use post-quantum encryption. Whether yours does mostly depends on one thing](/blog/post-quantum-key-exchange-six-in-ten-websites-november-2026) | **Glossary:** [E-E-A-T](/glossary/e-e-a-t)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
