{"ok":true,"relay_note":"Until 13 November 2026 16:04 UTC a mirror of this site that we ran ourselves relayed crawler requests here from a CDN address under each crawler's own name. Counts recorded before then include that relayed traffic: most Meta-ExternalAgent hits, and part of the failed-verification counts of crawlers that publish address ranges (Amazonbot above all). The mirror is closed; the earlier counts cannot be separated and should not be quoted as a forgery rate.","spoof_note":"A failed verification on /api/, /r/ or /tools/ is somebody checking a report we produced, not a crawler impersonating an operator on a page. Those are counted in spoofed_hits_tooling and should be excluded before quoting a forgery rate. Hits recorded before this split shipped carry no path attribution and are reported as spoofed_hits_unattributed rather than assigned to either side.","bots_seen":45,"total_hits":357551,"claimed_hits":437166,"verified_hits":184958,"spoofed_hits":24713,"spoofed_hits_tooling":317,"spoofed_hits_content":1530,"spoofed_hits_unattributed":22866,"unverifiable_hits":227526,"by_bot":[{"bot":"Meta-ExternalAgent","hits":103620,"claimed_hits":103620,"verified":0,"spoofed":0,"unverifiable":103620,"tls_verified":null,"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"this operator publishes no IP range feed — identity is unverifiable","hosts":{"crawlcheck.io":102898,"worker preview host":713,"www.crawlcheck.io":9},"prefers":"apex","confident":false},{"bot":"PerplexityBot","hits":34927,"claimed_hits":35256,"verified":34927,"spoofed":334,"unverifiable":0,"tls_verified":{"TLSv1.3":18641},"spoofed_tooling":3,"spoofed_content":100,"spoofed_unattributed":231,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":34927},"prefers":"apex","confident":true},{"bot":"Amazonbot","hits":33909,"claimed_hits":87867,"verified":33909,"spoofed":8915,"unverifiable":45044,"tls_verified":{"TLSv1.3":31637},"spoofed_tooling":248,"spoofed_content":119,"spoofed_unattributed":8548,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":33909},"prefers":"apex","confident":true},{"bot":"SemrushBot","hits":29913,"claimed_hits":29913,"verified":0,"spoofed":0,"unverifiable":29915,"tls_verified":null,"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"this operator publishes no IP range feed — identity is unverifiable","hosts":{"crawlcheck.io":29913},"prefers":"apex","confident":false},{"bot":"YandexBot","hits":28025,"claimed_hits":30461,"verified":28025,"spoofed":20,"unverifiable":2438,"tls_verified":{"TLSv1.3":16605},"spoofed_tooling":0,"spoofed_content":20,"spoofed_unattributed":0,"verification":"source IP checked by forward-confirmed reverse DNS to the operator's own domains (from 10 October 2026)","hosts":{"crawlcheck.io":28025},"prefers":"apex","confident":true},{"bot":"Bytespider","hits":19702,"claimed_hits":19702,"verified":0,"spoofed":0,"unverifiable":19702,"tls_verified":null,"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"this operator publishes no IP range feed — identity is unverifiable","hosts":{"crawlcheck.io":19694,"www.crawlcheck.io":8},"prefers":"apex","confident":false},{"bot":"DataForSeoBot","hits":16743,"claimed_hits":16743,"verified":0,"spoofed":0,"unverifiable":16743,"tls_verified":null,"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"this operator publishes no IP range feed — identity is unverifiable","hosts":{"crawlcheck.io":16743},"prefers":"apex","confident":false},{"bot":"AhrefsBot","hits":14550,"claimed_hits":19193,"verified":14550,"spoofed":4,"unverifiable":4640,"tls_verified":{"TLSv1.3":11685},"spoofed_tooling":0,"spoofed_content":3,"spoofed_unattributed":1,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":14550},"prefers":"apex","confident":true},{"bot":"Applebot","hits":12452,"claimed_hits":12751,"verified":12452,"spoofed":299,"unverifiable":0,"tls_verified":{"TLSv1.3":7579},"spoofed_tooling":2,"spoofed_content":84,"spoofed_unattributed":213,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":12306,"www.crawlcheck.io":146},"prefers":"apex","confident":true},{"bot":"Googlebot","hits":12235,"claimed_hits":22936,"verified":12235,"spoofed":10562,"unverifiable":139,"tls_verified":{"TLSv1.3":5497},"spoofed_tooling":1,"spoofed_content":261,"spoofed_unattributed":10300,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":12235},"prefers":"apex","confident":true},{"bot":"ClaudeBot","hits":11458,"claimed_hits":11791,"verified":11458,"spoofed":333,"unverifiable":0,"tls_verified":{"TLSv1.3":5046},"spoofed_tooling":4,"spoofed_content":123,"spoofed_unattributed":206,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":11416,"www.crawlcheck.io":42},"prefers":"apex","confident":true},{"bot":"Bingbot","hits":10851,"claimed_hits":11158,"verified":10851,"spoofed":168,"unverifiable":139,"tls_verified":{"TLSv1.3":8026},"spoofed_tooling":0,"spoofed_content":52,"spoofed_unattributed":116,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":10849,"www.crawlcheck.io":2},"prefers":"apex","confident":true},{"bot":"GPTBot","hits":10413,"claimed_hits":10905,"verified":10413,"spoofed":492,"unverifiable":0,"tls_verified":{"TLSv1.3":4558},"spoofed_tooling":5,"spoofed_content":183,"spoofed_unattributed":304,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":10413},"prefers":"apex","confident":true},{"bot":"OAI-SearchBot","hits":8865,"claimed_hits":10887,"verified":8865,"spoofed":2022,"unverifiable":0,"tls_verified":{"TLSv1.3":6640},"spoofed_tooling":3,"spoofed_content":111,"spoofed_unattributed":1908,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":8865},"prefers":"apex","confident":true},{"bot":"YouBot","hits":2519,"claimed_hits":2519,"verified":0,"spoofed":0,"unverifiable":2519,"tls_verified":null,"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"this operator publishes no IP range feed — identity is unverifiable","hosts":{"crawlcheck.io":2519},"prefers":"apex","confident":false},{"bot":"ChatGPT-User","hits":1747,"claimed_hits":2112,"verified":1747,"spoofed":365,"unverifiable":0,"tls_verified":{"TLSv1.3":940},"spoofed_tooling":3,"spoofed_content":89,"spoofed_unattributed":273,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":1747},"prefers":"apex","confident":true},{"bot":"Claude-User","hits":1540,"claimed_hits":2169,"verified":1540,"spoofed":629,"unverifiable":0,"tls_verified":{"TLSv1.3":1223},"spoofed_tooling":6,"spoofed_content":55,"spoofed_unattributed":568,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":1540},"prefers":"apex","confident":true},{"bot":"SeznamBot","hits":1135,"claimed_hits":1135,"verified":1135,"spoofed":0,"unverifiable":0,"tls_verified":{"TLSv1.3":953},"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"source IP checked by forward-confirmed reverse DNS to the operator's own domains (from 10 October 2026)","hosts":{"crawlcheck.io":1135},"prefers":"apex","confident":true},{"bot":"Baiduspider","hits":683,"claimed_hits":1711,"verified":683,"spoofed":61,"unverifiable":967,"tls_verified":{"TLSv1.3":569},"spoofed_tooling":3,"spoofed_content":45,"spoofed_unattributed":13,"verification":"source IP checked by forward-confirmed reverse DNS to the operator's own domains (from 10 October 2026)","hosts":{"crawlcheck.io":683},"prefers":"apex","confident":true},{"bot":"CCBot","hits":636,"claimed_hits":2164,"verified":636,"spoofed":14,"unverifiable":1514,"tls_verified":{"TLSv1.3":633},"spoofed_tooling":2,"spoofed_content":9,"spoofed_unattributed":3,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":636},"prefers":"apex","confident":true},{"bot":"Google-CWS","hits":478,"claimed_hits":478,"verified":478,"spoofed":0,"unverifiable":0,"tls_verified":{"TLSv1.3":225},"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":478},"prefers":"apex","confident":true},{"bot":"Google-InspectionTool","hits":361,"claimed_hits":367,"verified":361,"spoofed":6,"unverifiable":0,"tls_verified":{"TLSv1.3":205},"spoofed_tooling":1,"spoofed_content":5,"spoofed_unattributed":0,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":361},"prefers":"apex","confident":true},{"bot":"Google-Agent","hits":281,"claimed_hits":281,"verified":281,"spoofed":0,"unverifiable":0,"tls_verified":{"TLSv1.3":168},"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":281},"prefers":"apex","confident":true},{"bot":"DuckDuckBot","hits":193,"claimed_hits":243,"verified":193,"spoofed":0,"unverifiable":50,"tls_verified":{"TLSv1.3":162},"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":193},"prefers":"apex","confident":true},{"bot":"GoogleOther","hits":178,"claimed_hits":315,"verified":178,"spoofed":137,"unverifiable":0,"tls_verified":{"TLSv1.3":177},"spoofed_tooling":7,"spoofed_content":94,"spoofed_unattributed":36,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":178},"prefers":"apex","confident":true},{"bot":"cohere-ai","hits":62,"claimed_hits":62,"verified":0,"spoofed":0,"unverifiable":62,"tls_verified":null,"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"this operator publishes no IP range feed — identity is unverifiable","hosts":{"crawlcheck.io":62},"prefers":"apex","confident":false},{"bot":"Scrapy","hits":18,"claimed_hits":18,"verified":0,"spoofed":0,"unverifiable":18,"tls_verified":null,"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"this operator publishes no IP range feed — identity is unverifiable","hosts":{"crawlcheck.io":18},"prefers":"apex","confident":false},{"bot":"Perplexity-User","hits":9,"claimed_hits":128,"verified":9,"spoofed":119,"unverifiable":0,"tls_verified":{"TLSv1.3":1},"spoofed_tooling":8,"spoofed_content":52,"spoofed_unattributed":59,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":9},"prefers":"apex","confident":false},{"bot":"MistralAI-User","hits":9,"claimed_hits":9,"verified":0,"spoofed":0,"unverifiable":9,"tls_verified":null,"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":9},"prefers":"apex","confident":false},{"bot":"Google-NotebookLM","hits":8,"claimed_hits":8,"verified":8,"spoofed":0,"unverifiable":0,"tls_verified":{"TLSv1.3":1},"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":8},"prefers":"apex","confident":false},{"bot":"Google-Read-Aloud","hits":7,"claimed_hits":7,"verified":7,"spoofed":0,"unverifiable":0,"tls_verified":{"TLSv1.3":4},"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":7},"prefers":"apex","confident":false},{"bot":"Yeti","hits":6,"claimed_hits":6,"verified":6,"spoofed":0,"unverifiable":0,"tls_verified":{"TLSv1.3":6},"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"source IP checked by forward-confirmed reverse DNS to the operator's own domains (from 10 October 2026)","hosts":{"crawlcheck.io":6},"prefers":"apex","confident":false},{"bot":"AdsBot-Google","hits":5,"claimed_hits":5,"verified":5,"spoofed":0,"unverifiable":0,"tls_verified":{"TLSv1.3":5},"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":5},"prefers":"apex","confident":false},{"bot":"Timpibot","hits":4,"claimed_hits":4,"verified":0,"spoofed":0,"unverifiable":4,"tls_verified":null,"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"this operator publishes no IP range feed — identity is unverifiable","hosts":{"crawlcheck.io":4},"prefers":"apex","confident":false},{"bot":"Google-Safety","hits":4,"claimed_hits":4,"verified":4,"spoofed":0,"unverifiable":0,"tls_verified":{"TLSv1.3":4},"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":4},"prefers":"apex","confident":false},{"bot":"Diffbot","hits":2,"claimed_hits":2,"verified":0,"spoofed":0,"unverifiable":2,"tls_verified":null,"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"this operator publishes no IP range feed — identity is unverifiable","hosts":{"crawlcheck.io":2},"prefers":"apex","confident":false},{"bot":"Google-Site-Verification","hits":2,"claimed_hits":2,"verified":2,"spoofed":0,"unverifiable":0,"tls_verified":null,"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"source IP checked against the operator's published ranges","hosts":{"crawlcheck.io":2},"prefers":"apex","confident":false},{"bot":"FacebookBot","hits":1,"claimed_hits":1,"verified":0,"spoofed":0,"unverifiable":1,"tls_verified":null,"spoofed_tooling":0,"spoofed_content":0,"spoofed_unattributed":0,"verification":"this operator publishes no IP range feed — identity is unverifiable","hosts":{"crawlcheck.io":1},"prefers":"apex","confident":false},{"bot":"Google-Extended","hits":0,"claimed_hits":89,"verified":0,"spoofed":89,"unverifiable":0,"tls_verified":null,"spoofed_tooling":4,"spoofed_content":49,"spoofed_unattributed":36,"verification":"source IP checked against the operator's published ranges","hosts":{},"prefers":"no preference observed","confident":false},{"bot":"Claude-SearchBot","hits":0,"claimed_hits":109,"verified":0,"spoofed":109,"unverifiable":0,"tls_verified":null,"spoofed_tooling":2,"spoofed_content":58,"spoofed_unattributed":49,"verification":"source IP checked against the operator's published ranges","hosts":{},"prefers":"no preference observed","confident":false},{"bot":"anthropic-ai","hits":0,"claimed_hits":5,"verified":0,"spoofed":5,"unverifiable":0,"tls_verified":null,"spoofed_tooling":1,"spoofed_content":4,"spoofed_unattributed":0,"verification":"source IP checked against the operator's published ranges","hosts":{},"prefers":"no preference observed","confident":false},{"bot":"Applebot-Extended","hits":0,"claimed_hits":3,"verified":0,"spoofed":3,"unverifiable":0,"tls_verified":null,"spoofed_tooling":1,"spoofed_content":2,"spoofed_unattributed":0,"verification":"source IP checked against the operator's published ranges","hosts":{},"prefers":"no preference observed","confident":false},{"bot":"FeedFetcher-Google","hits":0,"claimed_hits":1,"verified":0,"spoofed":1,"unverifiable":0,"tls_verified":null,"spoofed_tooling":0,"spoofed_content":1,"spoofed_unattributed":0,"verification":"source IP checked against the operator's published ranges","hosts":{},"prefers":"no preference observed","confident":false},{"bot":"Google-CloudVertexBot","hits":0,"claimed_hits":3,"verified":0,"spoofed":3,"unverifiable":0,"tls_verified":null,"spoofed_tooling":0,"spoofed_content":3,"spoofed_unattributed":0,"verification":"source IP checked against the operator's published ranges","hosts":{},"prefers":"no preference observed","confident":false},{"bot":"Storebot-Google","hits":0,"claimed_hits":23,"verified":0,"spoofed":23,"unverifiable":0,"tls_verified":null,"spoofed_tooling":13,"spoofed_content":8,"spoofed_unattributed":2,"verification":"source IP checked against the operator's published ranges","hosts":{},"prefers":"no preference observed","confident":false}],"recent":[{"at":"2026-11-20T13:29:56.139Z","bot":"ClaudeBot","host":"crawlcheck.io","path":"/registry/onecallconstruction.us","verified":true,"country":"US","network":"216.73.216.0/24"},{"at":"2026-11-20T13:29:56.072Z","bot":"ClaudeBot","host":"crawlcheck.io","path":"/registry/idealfencingcorp.com","verified":true,"country":"US","network":"216.73.216.0/24"},{"at":"2026-11-20T13:29:54.511Z","bot":"ClaudeBot","host":"crawlcheck.io","path":"/glossary/lighthouse.json","verified":true,"country":"US","network":"216.73.216.0/24"},{"at":"2026-11-20T13:29:54.109Z","bot":"ClaudeBot","host":"crawlcheck.io","path":"/certificate/d-id.com","verified":true,"country":"US","network":"216.73.216.0/24"},{"at":"2026-11-20T13:29:53.353Z","bot":"ClaudeBot","host":"crawlcheck.io","path":"/registry/rankability.com","verified":true,"country":"US","network":"216.73.216.0/24"},{"at":"2026-11-20T13:29:52.696Z","bot":"ClaudeBot","host":"crawlcheck.io","path":"/registry/facebook.com","verified":true,"country":"US","network":"216.73.216.0/24"},{"at":"2026-11-20T13:29:52.246Z","bot":"ClaudeBot","host":"crawlcheck.io","path":"/registry/amicited.com","verified":true,"country":"US","network":"216.73.216.0/24"},{"at":"2026-11-20T13:29:12.923Z","bot":"Bytespider","host":"crawlcheck.io","path":"/","verified":null,"country":"SG","network":"47.128.56.0/24"},{"at":"2026-11-20T13:28:45.782Z","bot":"Bytespider","host":"crawlcheck.io","path":"/registry/servicelifter.com/claim","verified":null,"country":"SG","network":"47.128.56.0/24"},{"at":"2026-11-20T13:28:23.050Z","bot":"Bytespider","host":"crawlcheck.io","path":"/","verified":null,"country":"SG","network":"47.128.56.0/24"},{"at":"2026-11-20T13:28:08.939Z","bot":"Bingbot","host":"crawlcheck.io","path":"/glossary/strict-dynamic","verified":true,"country":"US","network":"52.167.144.0/24"},{"at":"2026-11-20T13:28:04.263Z","bot":"SemrushBot","host":"crawlcheck.io","path":"/datasets/technologies/nuxt","verified":null,"country":"US","network":"85.208.96.0/24"},{"at":"2026-11-20T13:27:58.421Z","bot":"OAI-SearchBot","host":"crawlcheck.io","path":"/api/form/config","verified":true,"country":"US","network":"40.67.175.0/24"},{"at":"2026-11-20T13:27:19.905Z","bot":"YandexBot","host":"crawlcheck.io","path":"/directory/brandwatch","verified":true,"country":"RU","network":"95.108.213.0/24"},{"at":"2026-11-20T13:26:42.504Z","bot":"Bingbot","host":"crawlcheck.io","path":"/certificate/sciencepark.se","verified":true,"country":"US","network":"52.167.144.0/24"},{"at":"2026-11-20T13:26:32.752Z","bot":"Bingbot","host":"crawlcheck.io","path":"/registry/swedroid.se","verified":true,"country":"US","network":"40.77.167.0/24"},{"at":"2026-11-20T13:26:19.874Z","bot":"AhrefsBot","host":"crawlcheck.io","path":"/registry/nyheter24.se","verified":true,"country":"FR","network":"92.222.108.0/24"},{"at":"2026-11-20T13:25:10.212Z","bot":"YandexBot","host":"crawlcheck.io","path":"/registry/logoswebdesigns.com","verified":true,"country":"RU","network":"95.108.213.0/24"},{"at":"2026-11-20T13:25:10.123Z","bot":"YandexBot","host":"crawlcheck.io","path":"/registry/trade/concrete","verified":true,"country":"RU","network":"213.180.203.0/24"},{"at":"2026-11-20T13:25:03.026Z","bot":"YandexBot","host":"crawlcheck.io","path":"/registry/trade/landscaping","verified":true,"country":"RU","network":"95.108.213.0/24"},{"at":"2026-11-20T13:24:54.860Z","bot":"YandexBot","host":"crawlcheck.io","path":"/robots.txt","verified":true,"country":"RU","network":"95.108.213.0/24"},{"at":"2026-11-20T13:24:08.736Z","bot":"YandexBot","host":"crawlcheck.io","path":"/glossary/redirected-internal-link","verified":true,"country":"RU","network":"5.255.231.0/24"},{"at":"2026-11-20T13:23:49.141Z","bot":"AhrefsBot","host":"crawlcheck.io","path":"/registry/omtanke.today","verified":true,"country":"FR","network":"176.31.139.0/24"},{"at":"2026-11-20T13:23:01.390Z","bot":"YandexBot","host":"crawlcheck.io","path":"/directory/bolt","verified":true,"country":"RU","network":"95.108.213.0/24"},{"at":"2026-11-20T13:23:01.419Z","bot":"YandexBot","host":"crawlcheck.io","path":"/api/registry","verified":true,"country":"RU","network":"95.108.213.0/24"}],"method":"First-party observation of requests to this endpoint. The Host header is recorded on every hit — the field the default access-log format discards, which is why apex-vs-www crawler behaviour has no published answer.","caveat":"A census of ONE property, not a sample of the web. It reports what crawlers did here. `confident` is false below 20 hits, where a host split is noise rather than a preference. `total_hits` counts only hits whose source IP falls inside the operator's own published ranges; `claimed_hits` counts every request that merely said so. A user-agent is a claim. Where an operator publishes no feed the hits are reported as unverifiable and can never be marked confident — that is a limit of the evidence, not a judgement about the crawler."}