{"kind":"crawlcheck-census-similarity","v":1,"frame":"top1k","label":"top 1,000, daily","run":51,"finished_at":"2026-11-21T09:35:00.723Z","sites":991,"fingerprinted":429,"distinct_landing_hosts":394,"families":{"families":0,"sites_in_families":0,"size_distribution":{"2":0,"3_to_9":0,"10_plus":0},"largest":[],"max_bits":6},"parity":{"gptbot":{"compared":350,"same":311,"small":27,"different":9,"other_page":3,"other_host":8},"claudebot":{"compared":341,"same":304,"small":24,"different":7,"other_page":6,"other_host":9},"oaisearch":{"compared":369,"same":323,"small":27,"different":12,"other_page":7,"other_host":11},"perplexity":{"compared":353,"same":312,"small":28,"different":9,"other_page":4,"other_host":9}},"reading":"Fingerprints are 64-bit SimHashes of a homepage's visible text with the site's own name, URLs, emails and numbers masked; pages under 40 words are skipped. A family is homepages on different domains at most 6 bits apart. Domains that land on the same host count once; the fingerprint covers the whole page however long; an identity sent to a different host or a different URL than the browser (a mobile, country or language edition) is counted as other_host, not compared. Parity bands: same 0-3 bits, small 4-10, different 11-20, other_page over 20. A difference is not an accusation: rotating content, consent banners and personalisation move text too; per-site readings stay in the census records.","method":"https://crawlcheck.io/api/census","families_of_mcp_servers":"https://crawlcheck.io/api/v1/mcp/families","code":{"build_id":"9219fb519ff0eaf2","script_sha16":"b00e794a299c5a1b","state":"attested","deployed":{"at":"2026-11-21T18:41:28.011Z"},"build_log_entry":2899,"score_version":43,"scoring_policy_sha256":"3e08fccfbbce356dc82ecd302b32a226a36fc0fb3150a4ab72248fb5795372af","decision_rules_sha256":"18dd5636e0896692ab518050cb401744edea2e6f1086dff94fead314959bdc3e","check":"https://crawlcheck.io/api/v1/builds/9219fb519ff0eaf2"},"sha256":"9ff437f435fb5cfa5ecf76e6caa6e4c6cb06f16c75f42b31ac3bb1ec07d91ed8","signature":{"v":1,"alg":"Ed25519","kid":"1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps","over":"sha256","message":"crawlcheck-data-v1\n9ff437f435fb5cfa5ecf76e6caa6e4c6cb06f16c75f42b31ac3bb1ec07d91ed8","sig":"12UbwC4dEYmzUBsbrJwFZOJ4FaWP9xuZQLpVzjWz4pQ3QqBViMFyDacqLmiDgHkKZMxBCn28hHR9oD8yKTqOAA","signed_at":"2026-11-22T01:33:57.080Z","key":{"kid":"1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps","jwk":{"kty":"OKP","crv":"Ed25519","x":"5TmKCOx0TTHepq7HwyXBHvGQpVCSBd2lV8cLzYVqFxQ"},"published_in":"https://crawlcheck.io/.well-known/http-message-signatures-directory"},"canonical":"sha256 over canonical JSON of every field except sha256 and signature"}}