{"kind":"crawlcheck-conduct","v":1,"generated_at":"2026-11-08T23:56:06.036Z","population":"verifiable agents observed on crawlcheck.io, and on the other sites that report through their edge Workers (portfolio)","signature_coverage":{"agents_assessed":30,"agents_signing":2,"verified_requests":107942,"signed_requests":3116,"failed_signatures":2827,"by_agent":[{"agent":"AhrefsBot","operator":"Ahrefs","signed_requests":3098,"verified_requests":9804,"signed_share":0.316},{"agent":"DuckDuckBot","operator":"DuckDuckGo","signed_requests":18,"verified_requests":128,"signed_share":0.141}],"signed_share":0.029,"counted_correctly_since":"2026-10-29T17:45:00Z","reading":"signed_requests are verified requests that carried a Web Bot Auth signature which verified against the operator's own key directory; signed_share is that count over verified_requests. An agent whose operator publishes no key directory cannot sign, so a low share is a measure of adoption, not of conduct"},"portfolio":{"sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com","emmanuelorta.com"],"sites_with_verified_fetches":6,"days":14,"site_days_judged":12,"judged_from":"2026-11-07","reading":"the other sites that report to CrawlCheck (the operator's own sites), whose edge Workers report every verified crawler fetch with its path; a fetch is judged against that site's robots.txt as read on the day of the fetch and the day before (both must disallow it); site-days without both readings are counted as not judged"},"reading":"A census of one property, not a sample of the web: what each agent did HERE, verified by source address against the operator's own published ranges. conduct is no_violations_observed when at least 50 verified requests show no fetch of a path robots.txt disallows; violations_observed when a disallowed path was fetched; insufficient_data below 50. The busiest minute is reported, not judged: this site publishes no crawl rate. site_checks_14d are counts from Visitor Resolve calls other sites made about this agent.","thresholds":{"min_verified_requests":50,"published_rate_limit":null,"disallowed_rules":["/r/","/backup/","/admin-old/"]},"verdicts":{"respected":"robots.txt fetched, and no disallowed path fetched","robots_txt_not_fetched":"no disallowed path fetched, but robots.txt was never fetched under this agent's name here, so it cannot be credited with reading the rules","disallowed_paths_fetched":"a verified fetch of a disallowed path, kept with its time, path and network in disallowed_fetch_records","disallowed_fetch_unrecorded":"disallowed fetches were counted before 2026-10-15, when no per-fetch record was kept; reported, not judged (conduct: not_judged)"},"grades":{"A":"read robots.txt and obeyed it","B":"obeyed, but never fetched robots.txt under its own name here","D":"recorded fetches of disallowed paths, under 0.5% of its verified requests","F":"a fetch of a decoy path named only in robots.txt, or disallowed fetches at 0.5% or more","none":"not enough verified requests, or not judgeable"},"agents":[{"agent":"Amazonbot","operator":"Amazon","purpose":"indexing","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-13T20:30:33.503Z","last_seen":"2026-11-08T23:54:18.252Z"},"identity":{"verified_requests":23731,"forged_claims":107,"forgery_share":0.004,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Amazon publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":0,"disallowed_fetches":52,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-18T08:45:12.519Z","path":"/r/xufza280d1","net":"18.215.77.0/24"},{"at":"2026-10-18T18:02:03.808Z","path":"/r/xufza280d1","net":"52.205.113.0/24"},{"at":"2026-10-18T18:04:19.333Z","path":"/r/xufza280d1","net":"34.227.156.0/24"},{"at":"2026-10-18T19:28:54.132Z","path":"/r/xufza280d1","net":"54.163.169.0/24"},{"at":"2026-10-18T20:50:07.376Z","path":"/r/xufza280d1","net":"44.220.2.0/24"},{"at":"2026-10-19T23:29:49.080Z","path":"/r/xufza280d1","net":"23.23.137.0/24"},{"at":"2026-10-23T04:17:44.813Z","path":"/r/hsem60u0pb","net":"18.214.186.0/24"},{"at":"2026-10-24T07:16:41.220Z","path":"/r/hsem60u0pb","net":"184.73.68.0/24"},{"at":"2026-10-24T07:16:45.273Z","path":"/r/hsem60u0pb","net":"3.221.156.0/24"},{"at":"2026-10-24T13:42:45.118Z","path":"/r/hsem60u0pb","net":"18.209.137.0/24"},{"at":"2026-10-26T02:57:16.905Z","path":"/r/cz9abl7ob3","net":"18.215.77.0/24"},{"at":"2026-10-27T12:38:34.291Z","path":"/r/cz9abl7ob3","net":"44.197.76.0/24"},{"at":"2026-10-27T14:53:27.270Z","path":"/r/cz9abl7ob3","net":"52.1.106.0/24"},{"at":"2026-10-27T17:03:47.170Z","path":"/r/cz9abl7ob3","net":"107.23.62.0/24"},{"at":"2026-10-27T22:34:39.166Z","path":"/r/cz9abl7ob3","net":"34.231.77.0/24"},{"at":"2026-10-27T23:05:38.747Z","path":"/r/1fb80ss94n","net":"52.45.29.0/24"},{"at":"2026-10-28T06:16:36.232Z","path":"/r/1fb80ss94n","net":"44.207.69.0/24"},{"at":"2026-10-28T08:23:26.019Z","path":"/r/up0rftpp5z","net":"54.147.238.0/24"},{"at":"2026-10-28T11:12:05.504Z","path":"/r/1fb80ss94n","net":"52.204.89.0/24"},{"at":"2026-10-28T11:12:06.136Z","path":"/r/1fb80ss94n","net":"44.194.139.0/24"}],"counted_since":"2026-10-18T06:50:57.435Z","reason":null,"robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":34,"peak_at":"2026-11-05T10:13","published_limit":null,"measured_since":"2026-10-18T06:50:57.435Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-18T06:50:57.435Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-26":861,"2026-10-27":1018,"2026-10-28":916,"2026-10-29":1009,"2026-10-30":914,"2026-10-31":534,"2026-11-01":605,"2026-11-02":626,"2026-11-03":1015,"2026-11-04":1454,"2026-11-05":2406,"2026-11-06":2351,"2026-11-07":1015,"2026-11-08":693},"portfolio":{"agent":"Amazonbot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":167,"verified_fetches_not_judged":742,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["52 recorded fetches of disallowed paths, 0.22% of its 23,731 verified requests"]},{"agent":"PerplexityBot","operator":"Perplexity","purpose":"indexing","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-13T02:14:50.805Z","last_seen":"2026-11-08T23:52:19.451Z"},"identity":{"verified_requests":19059,"forged_claims":87,"forgery_share":0.005,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Perplexity publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":198,"disallowed_fetches":36,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-22T08:38:56.498Z","path":"/r/hsem60u0pb","net":"18.97.9.0/24"},{"at":"2026-10-22T15:59:08.322Z","path":"/r/hsem60u0pb","net":"18.97.9.0/24"},{"at":"2026-10-22T15:59:20.457Z","path":"/r/uazt7utoub","net":"18.97.9.0/24"},{"at":"2026-10-24T07:38:31.838Z","path":"/r/hsem60u0pb","net":"18.97.9.0/24"},{"at":"2026-10-24T07:38:34.319Z","path":"/r/phuaa0zazw","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:44.938Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:45.149Z","path":"/r/phuaa0zazw","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:45.922Z","path":"/r/phuaa0zazw","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:46.331Z","path":"/r/phuaa0zazw","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:47.936Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:02.461Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:04.413Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:07.938Z","path":"/r/up0rftpp5z","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:08.999Z","path":"/r/1fb80ss94n","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:09.428Z","path":"/r/up0rftpp5z","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:11.912Z","path":"/r/1fb80ss94n","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:18.485Z","path":"/r/up0rftpp5z","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:18.928Z","path":"/r/up0rftpp5z","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:46.518Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"},{"at":"2026-10-28T20:49:42.689Z","path":"/r/khlmjzqasm","net":"18.97.9.0/24"}],"counted_since":"2026-10-07T04:51:16.683Z","reason":null,"robots_txt_own_fetches":198,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":119,"peak_at":"2026-10-11T23:32","published_limit":null,"measured_since":"2026-10-07T04:51:16.683Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T04:51:16.683Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":250,"verified":163,"spoofed":87},"daily_verified":{"2026-10-26":121,"2026-10-27":250,"2026-10-28":181,"2026-10-29":96,"2026-10-30":173,"2026-10-31":289,"2026-11-01":144,"2026-11-02":878,"2026-11-03":853,"2026-11-04":814,"2026-11-05":613,"2026-11-06":687,"2026-11-07":972,"2026-11-08":1521},"portfolio":{"agent":"PerplexityBot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":41,"verified_fetches_not_judged":324,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["36 recorded fetches of disallowed paths, 0.19% of its 19,059 verified requests"]},{"agent":"Googlebot","operator":"Google","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-13T02:14:51.611Z","last_seen":"2026-11-08T23:47:29.563Z"},"identity":{"verified_requests":9829,"forged_claims":197,"forgery_share":0.02,"signed_requests":0,"failed_signatures":392,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":176,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T03:18:09.861Z","reason":null,"robots_txt_own_fetches":176,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":13,"peak_at":"2026-10-19T02:10","published_limit":null,"measured_since":"2026-10-07T03:18:09.861Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":1132,"share_of_verified_since_measured":0.115,"measured_since":"2026-10-07T03:18:09.861Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":1678,"verified":1198,"spoofed":480},"daily_verified":{"2026-10-26":89,"2026-10-27":69,"2026-10-28":48,"2026-10-29":139,"2026-10-30":103,"2026-10-31":193,"2026-11-01":162,"2026-11-02":215,"2026-11-03":264,"2026-11-04":83,"2026-11-05":117,"2026-11-06":204,"2026-11-07":134,"2026-11-08":98},"portfolio":{"agent":"Googlebot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":330,"verified_fetches_not_judged":3132,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"AhrefsBot","operator":"Ahrefs","purpose":"tool","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-14T02:11:27.871Z","last_seen":"2026-11-08T23:51:43.431Z"},"identity":{"verified_requests":9804,"forged_claims":1,"forgery_share":0,"signed_requests":3098,"failed_signatures":2356,"signed_share":0.316,"strongest_proof":"signature","reading":"verified_requests came from the ranges Ahrefs publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":193,"disallowed_fetches":7,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-26T07:48:14.796Z","path":"/r/1fb80ss94n","net":"51.68.247.0/24"},{"at":"2026-10-26T08:03:14.376Z","path":"/r/1fb80ss94n","net":"51.75.236.0/24"},{"at":"2026-10-26T08:44:02.288Z","path":"/r/1fb80ss94n","net":"5.39.109.0/24"},{"at":"2026-11-03T14:28:07.385Z","path":"/r/6gaxqt2gao","net":"51.81.110.0/24"}],"counted_since":"2026-10-07T23:53:27.541Z","reason":null,"robots_txt_own_fetches":193,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":64,"peak_at":"2026-11-01T16:47","published_limit":null,"measured_since":"2026-10-07T23:53:27.541Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":36,"share_of_verified_since_measured":0.004,"measured_since":"2026-10-07T23:53:27.541Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-26":315,"2026-10-27":158,"2026-10-28":134,"2026-10-29":217,"2026-10-30":30,"2026-10-31":30,"2026-11-01":1118,"2026-11-02":395,"2026-11-03":174,"2026-11-04":148,"2026-11-05":619,"2026-11-06":223,"2026-11-07":175,"2026-11-08":130},"portfolio":{"agent":"AhrefsBot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":332,"verified_fetches_not_judged":2751,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["7 recorded fetches of disallowed paths, 0.07% of its 9,804 verified requests"]},{"agent":"ClaudeBot","operator":"Anthropic","purpose":"training","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-13T02:14:49.992Z","last_seen":"2026-11-08T02:39:23.314Z"},"identity":{"verified_requests":8920,"forged_claims":104,"forgery_share":0.012,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Anthropic publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":273,"disallowed_fetches":3,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-16T09:27:57.482Z","path":"/r/uazt7utoub","net":"216.73.216.0/24"},{"at":"2026-10-16T09:31:12.061Z","path":"/r/uazt7utoub","net":"216.73.216.0/24"},{"at":"2026-10-16T09:38:15.950Z","path":"/r/uazt7utoub","net":"216.73.216.0/24"}],"counted_since":"2026-10-07T03:48:54.788Z","reason":null,"robots_txt_own_fetches":273,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":61,"peak_at":"2026-10-19T01:34","published_limit":null,"measured_since":"2026-10-07T03:48:54.788Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T03:48:54.788Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":283,"verified":182,"spoofed":101},"daily_verified":{"2026-10-26":14,"2026-10-27":430,"2026-10-28":20,"2026-10-29":18,"2026-10-30":81,"2026-10-31":22,"2026-11-01":17,"2026-11-02":17,"2026-11-03":1213,"2026-11-04":84,"2026-11-05":28,"2026-11-06":15,"2026-11-07":2,"2026-11-08":2},"portfolio":{"agent":"ClaudeBot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":167,"verified_fetches_not_judged":2481,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["3 recorded fetches of disallowed paths, 0.03% of its 8,920 verified requests"]},{"agent":"Applebot","operator":"Apple","purpose":"indexing","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-13T12:32:42.174Z","last_seen":"2026-11-08T23:54:41.326Z"},"identity":{"verified_requests":7829,"forged_claims":84,"forgery_share":0.011,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Apple publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":80,"disallowed_fetches":40,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-23T13:32:15.215Z","path":"/r/8dsc0mu5ks","net":"17.166.233.0/24"},{"at":"2026-10-23T13:39:38.180Z","path":"/r/8dsc0mu5ks","net":"17.166.232.0/24"},{"at":"2026-10-24T07:28:50.585Z","path":"/r/8dsc0mu5ks","net":"17.166.235.0/24"},{"at":"2026-10-24T07:32:58.177Z","path":"/r/8dsc0mu5ks","net":"17.166.236.0/24"},{"at":"2026-10-24T16:13:56.254Z","path":"/r/mkx08rapce","net":"17.166.232.0/24"},{"at":"2026-10-24T16:17:40.243Z","path":"/r/mkx08rapce","net":"17.166.235.0/24"},{"at":"2026-10-26T15:31:58.651Z","path":"/r/8dsc0mu5ks","net":"17.166.232.0/24"},{"at":"2026-10-26T15:34:20.954Z","path":"/r/8dsc0mu5ks","net":"17.166.232.0/24"},{"at":"2026-10-26T16:01:58.942Z","path":"/r/8dsc0mu5ks","net":"17.166.25.0/24"},{"at":"2026-10-26T16:06:50.241Z","path":"/r/8dsc0mu5ks","net":"17.166.21.0/24"},{"at":"2026-10-27T20:22:26.222Z","path":"/r/mkx08rapce","net":"17.166.235.0/24"},{"at":"2026-10-27T20:37:56.625Z","path":"/r/mkx08rapce","net":"17.166.233.0/24"},{"at":"2026-10-28T01:54:46.237Z","path":"/r/mkx08rapce","net":"17.166.21.0/24"},{"at":"2026-10-28T01:55:55.986Z","path":"/r/mkx08rapce","net":"17.166.22.0/24"},{"at":"2026-10-28T12:27:38.731Z","path":"/r/mkx08rapce","net":"17.166.153.0/24"},{"at":"2026-10-28T12:33:52.496Z","path":"/r/mkx08rapce","net":"17.166.152.0/24"},{"at":"2026-10-30T18:31:10.592Z","path":"/r/khlmjzqasm","net":"17.166.153.0/24"},{"at":"2026-10-30T18:31:54.377Z","path":"/r/khlmjzqasm","net":"17.166.154.0/24"},{"at":"2026-11-02T05:43:52.587Z","path":"/r/aqxn4lxkaw","net":"17.166.151.0/24"},{"at":"2026-11-02T05:51:03.321Z","path":"/r/aqxn4lxkaw","net":"17.166.154.0/24"}],"counted_since":"2026-10-07T03:15:19.709Z","reason":null,"robots_txt_own_fetches":80,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":8,"peak_at":"2026-10-21T17:47","published_limit":null,"measured_since":"2026-10-07T03:15:19.709Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T03:15:19.709Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":627,"verified":582,"spoofed":45},"daily_verified":{"2026-10-26":31,"2026-10-27":39,"2026-10-28":49,"2026-10-29":35,"2026-10-30":117,"2026-10-31":94,"2026-11-01":40,"2026-11-02":48,"2026-11-03":53,"2026-11-04":51,"2026-11-05":114,"2026-11-06":99,"2026-11-07":124,"2026-11-08":779},"portfolio":{"agent":"Applebot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":207,"verified_fetches_not_judged":1164,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"F","grade_reasons":["40 recorded fetches of disallowed paths, 0.51% of its 7,829 verified requests"]},{"agent":"Bingbot","operator":"Microsoft","purpose":"indexing","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-14T06:25:41.331Z","last_seen":"2026-11-08T23:49:12.661Z"},"identity":{"verified_requests":7647,"forged_claims":49,"forgery_share":0.006,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Microsoft publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":104,"disallowed_fetches":1,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-27T19:17:43.386Z","path":"/r/uazt7utoub","net":"40.77.167.0/24"}],"counted_since":"2026-10-07T03:20:08.676Z","reason":null,"robots_txt_own_fetches":104,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":8,"peak_at":"2026-11-01T02:34","published_limit":null,"measured_since":"2026-10-07T03:20:08.676Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":579,"share_of_verified_since_measured":0.076,"measured_since":"2026-10-07T03:20:08.676Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":897,"verified":816,"spoofed":81},"daily_verified":{"2026-10-26":186,"2026-10-27":242,"2026-10-28":229,"2026-10-29":229,"2026-10-30":199,"2026-10-31":182,"2026-11-01":361,"2026-11-02":322,"2026-11-03":270,"2026-11-04":212,"2026-11-05":199,"2026-11-06":124,"2026-11-07":221,"2026-11-08":161},"portfolio":{"agent":"Bingbot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":224,"verified_fetches_not_judged":1490,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["1 recorded fetch of disallowed paths, 0.01% of its 7,647 verified requests"]},{"agent":"GPTBot","operator":"OpenAI","purpose":"training","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-12T06:20:42.861Z","last_seen":"2026-11-08T14:01:35.892Z"},"identity":{"verified_requests":6604,"forged_claims":155,"forgery_share":0.023,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges OpenAI publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"robots_txt_not_fetched","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T18:09:36.470Z","reason":null,"robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":120,"peak_at":"2026-11-02T11:28","published_limit":null,"measured_since":"2026-10-07T18:09:36.470Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T18:09:36.470Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":261,"verified":24,"spoofed":237},"daily_verified":{"2026-10-26":2,"2026-10-27":2,"2026-10-28":48,"2026-10-29":4,"2026-10-30":2,"2026-10-31":6,"2026-11-01":21,"2026-11-02":2130,"2026-11-03":2,"2026-11-04":6,"2026-11-05":2,"2026-11-06":24,"2026-11-07":2,"2026-11-08":57},"portfolio":{"agent":"GPTBot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":26,"verified_fetches_not_judged":365,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"B","grade_reasons":["no disallowed path fetched, but it never fetched robots.txt under its own name here, so it cannot be credited with reading the rules"]},{"agent":"OAI-SearchBot","operator":"OpenAI","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-13T02:14:49.166Z","last_seen":"2026-11-08T23:54:53.556Z"},"identity":{"verified_requests":5918,"forged_claims":98,"forgery_share":0.016,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges OpenAI publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":276,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T03:30:28.967Z","reason":null,"robots_txt_own_fetches":276,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":119,"peak_at":"2026-10-29T12:58","published_limit":null,"measured_since":"2026-10-07T03:30:28.967Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T03:30:28.967Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":756,"verified":686,"spoofed":70},"daily_verified":{"2026-10-26":67,"2026-10-27":74,"2026-10-28":35,"2026-10-29":1054,"2026-10-30":234,"2026-10-31":77,"2026-11-01":96,"2026-11-02":420,"2026-11-03":308,"2026-11-04":208,"2026-11-05":160,"2026-11-06":131,"2026-11-07":392,"2026-11-08":240},"portfolio":{"agent":"OAI-SearchBot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":211,"verified_fetches_not_judged":1137,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"YandexBot","operator":"Yandex","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-19T13:53:12.960Z","last_seen":"2026-11-08T23:48:00.799Z"},"identity":{"verified_requests":5156,"forged_claims":15,"forgery_share":0.003,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"reverse_dns","reading":"verified_requests passed forward-confirmed reverse DNS to Yandex's own domains (yandex.ru, yandex.net, yandex.com); forged_claims used this agent's name from addresses that did not (impersonators, not the agent). Reverse-DNS checks began 10 October 2026; earlier hits stay unverifiable"},"robots_txt":{"verdict":"respected","robots_txt_fetches":504,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-10T12:54:23.939Z","reason":null,"robots_txt_own_fetches":504,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":26,"peak_at":"2026-11-06T03:13","published_limit":null,"measured_since":"2026-10-10T12:54:23.939Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-10T12:54:23.939Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-25":7,"2026-10-26":6,"2026-10-28":7,"2026-10-29":384,"2026-10-30":163,"2026-10-31":221,"2026-11-01":243,"2026-11-02":884,"2026-11-03":149,"2026-11-04":147,"2026-11-05":173,"2026-11-06":749,"2026-11-07":172,"2026-11-08":130},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"ChatGPT-User","operator":"OpenAI","purpose":"live fetch","user_triggered":true,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-13T22:52:23.214Z","last_seen":"2026-11-08T22:46:26.638Z"},"identity":{"verified_requests":1241,"forged_claims":85,"forgery_share":0.064,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges OpenAI publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"not_applicable","robots_txt_fetches":1,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T03:26:16.042Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":1,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":13,"peak_at":"2026-10-15T19:36","published_limit":null,"measured_since":"2026-10-07T03:26:16.042Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T03:26:16.042Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":139,"verified":113,"spoofed":26},"daily_verified":{"2026-10-26":26,"2026-10-27":26,"2026-10-28":27,"2026-10-29":21,"2026-10-30":29,"2026-10-31":19,"2026-11-01":13,"2026-11-02":21,"2026-11-03":29,"2026-11-04":25,"2026-11-05":28,"2026-11-06":30,"2026-11-07":33,"2026-11-08":48},"portfolio":{"agent":"ChatGPT-User","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":23,"verified_fetches_not_judged":135,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"A","grade_reasons":["fetches pages a person asked for; its operator states robots.txt does not govern those fetches, and every counted request came from its published addresses"]},{"agent":"SeznamBot","operator":"Seznam","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-10-10T13:04:32.776Z","last_seen":"2026-11-08T16:18:09.672Z"},"identity":{"verified_requests":854,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"reverse_dns","reading":"verified_requests passed forward-confirmed reverse DNS to Seznam's own domains (seznam.cz); forged_claims used this agent's name from addresses that did not (impersonators, not the agent). Reverse-DNS checks began 10 October 2026; earlier hits stay unverifiable"},"robots_txt":{"verdict":"respected","robots_txt_fetches":22,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-10T13:04:32.776Z","reason":null,"robots_txt_own_fetches":22,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":3,"peak_at":"2026-10-10T22:12","published_limit":null,"measured_since":"2026-10-10T13:04:32.776Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":712,"share_of_verified_since_measured":0.834,"measured_since":"2026-10-10T13:04:32.776Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-26":29,"2026-10-27":35,"2026-10-28":24,"2026-10-29":19,"2026-10-30":50,"2026-10-31":30,"2026-11-01":22,"2026-11-02":52,"2026-11-03":2,"2026-11-04":44,"2026-11-05":3,"2026-11-06":43,"2026-11-07":9,"2026-11-08":2},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"Baiduspider","operator":"Baidu","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-13T21:31:01.445Z","last_seen":"2026-11-08T22:52:14.054Z"},"identity":{"verified_requests":472,"forged_claims":33,"forgery_share":0.065,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"reverse_dns","reading":"verified_requests passed forward-confirmed reverse DNS to Baidu's own domains (baidu.com, baidu.jp); forged_claims used this agent's name from addresses that did not (impersonators, not the agent). Reverse-DNS checks began 10 October 2026; earlier hits stay unverifiable"},"robots_txt":{"verdict":"robots_txt_not_fetched","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-10T13:14:25.713Z","reason":null,"robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":5,"peak_at":"2026-10-26T00:08","published_limit":null,"measured_since":"2026-10-10T13:14:25.713Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-10T13:14:25.713Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-26":29,"2026-10-27":12,"2026-10-28":17,"2026-10-29":5,"2026-10-30":24,"2026-10-31":29,"2026-11-01":10,"2026-11-02":21,"2026-11-03":1,"2026-11-04":12,"2026-11-05":19,"2026-11-06":18,"2026-11-07":10,"2026-11-08":16},"grade":"B","grade_reasons":["no disallowed path fetched, but it never fetched robots.txt under its own name here, so it cannot be credited with reading the rules"]},{"agent":"Google-CWS","operator":"Google","purpose":"store listing fetch","user_triggered":true,"conduct":"no_violations_observed","window":{"first_seen":"2026-09-18T16:25:39.959Z","last_seen":"2026-11-08T22:47:48.665Z"},"identity":{"verified_requests":349,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"not_applicable","robots_txt_fetches":176,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T11:31:33.296Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":176,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":3,"peak_at":"2026-10-07T20:31","published_limit":null,"measured_since":"2026-10-07T11:31:33.296Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T11:31:33.296Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-24":5,"2026-10-25":1,"2026-10-26":2,"2026-10-29":10,"2026-10-30":8,"2026-10-31":14,"2026-11-01":15,"2026-11-02":19,"2026-11-03":8,"2026-11-04":9,"2026-11-05":3,"2026-11-06":8,"2026-11-07":14,"2026-11-08":18},"grade":"A","grade_reasons":["fetches pages a person asked for; its operator states robots.txt does not govern those fetches, and every counted request came from its published addresses"]},{"agent":"Google-InspectionTool","operator":"Google","purpose":null,"user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-10-27T11:15:26.957Z","last_seen":"2026-11-08T21:51:27.847Z"},"identity":{"verified_requests":216,"forged_claims":4,"forgery_share":0.018,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":176,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-29T14:57:13.241Z","reason":"Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 176 times here","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":176,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":7,"peak_at":"2026-10-29T21:37","published_limit":null,"measured_since":"2026-10-29T14:57:13.241Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-29T14:57:13.241Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":1,"verified":0,"spoofed":1},"daily_verified":{"2026-10-29":57,"2026-10-30":18,"2026-10-31":10,"2026-11-01":11,"2026-11-02":4,"2026-11-03":5,"2026-11-04":4,"2026-11-06":8,"2026-11-07":29,"2026-11-08":10},"grade":"A","grade_reasons":["Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 176 times here, and it fetched no path robots.txt disallows"]},{"agent":"DuckDuckBot","operator":"DuckDuckGo","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-12T20:29:50.246Z","last_seen":"2026-11-08T17:36:16.431Z"},"identity":{"verified_requests":128,"forged_claims":0,"forgery_share":0,"signed_requests":18,"failed_signatures":60,"signed_share":0.141,"strongest_proof":"signature","reading":"verified_requests came from the ranges DuckDuckGo publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"robots_txt_not_fetched","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-10T21:21:35.448Z","reason":null,"robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":2,"peak_at":"2026-10-10T21:21","published_limit":null,"measured_since":"2026-10-10T21:21:35.448Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-10T21:21:35.448Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-26":7,"2026-10-27":6,"2026-10-28":7,"2026-10-29":8,"2026-10-30":1,"2026-10-31":4,"2026-11-01":3,"2026-11-02":7,"2026-11-03":6,"2026-11-04":4,"2026-11-05":6,"2026-11-06":6,"2026-11-07":4,"2026-11-08":5},"grade":"B","grade_reasons":["no disallowed path fetched, but it never fetched robots.txt under its own name here, so it cannot be credited with reading the rules"]},{"agent":"Claude-User","operator":"Anthropic","purpose":"live fetch","user_triggered":true,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-12T22:57:31.977Z","last_seen":"2026-11-08T21:23:56.512Z"},"identity":{"verified_requests":97,"forged_claims":54,"forgery_share":0.358,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Anthropic publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"not_applicable","robots_txt_fetches":14,"disallowed_fetches":1,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-17T04:44:31.376Z","path":"/r/zqdnb5b58f","net":"160.79.106.0/24"}],"counted_since":"2026-10-07T11:03:04.563Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":14,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":3,"peak_at":"2026-10-08T16:11","published_limit":null,"measured_since":"2026-10-07T11:03:04.563Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T11:03:04.563Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":24,"verified":0,"spoofed":24},"daily_verified":{"2026-10-25":1},"grade":"A","grade_reasons":["fetches pages a person asked for; its operator states robots.txt does not govern those fetches, and every counted request came from its published addresses"]},{"agent":"GoogleOther","operator":"Google","purpose":"training","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-10-28T07:22:36.856Z","last_seen":"2026-11-08T05:05:54.233Z"},"identity":{"verified_requests":55,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":176,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-28T07:22:36.856Z","reason":"Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 176 times here","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":176,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":3,"peak_at":"2026-11-04T13:58","published_limit":null,"measured_since":"2026-10-28T07:22:36.856Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-28T07:22:36.856Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":1,"verified":0,"spoofed":1},"daily_verified":{"2026-10-28":6,"2026-10-29":5,"2026-10-30":2,"2026-10-31":1,"2026-11-01":2,"2026-11-02":4,"2026-11-03":2,"2026-11-04":9,"2026-11-05":1,"2026-11-06":10,"2026-11-07":11,"2026-11-08":2},"grade":"A","grade_reasons":["Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 176 times here, and it fetched no path robots.txt disallows"]},{"agent":"Perplexity-User","operator":"Perplexity","purpose":"live fetch","user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-08-14T01:04:08.513Z","last_seen":"2026-11-07T19:14:23.425Z"},"identity":{"verified_requests":9,"forged_claims":51,"forgery_share":0.85,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Perplexity publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":1,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-11-07T19:14:23.425Z","path":"/r/a7qr2s89eu.md","net":"18.97.43.0/24"}],"counted_since":"2026-11-07T19:14:23.425Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":1,"peak_at":"2026-11-07T19:14","published_limit":null,"measured_since":"2026-11-07T19:14:23.425Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-11-07T19:14:23.425Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":13,"verified":0,"spoofed":13},"daily_verified":{"2026-11-07":1},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-NotebookLM","operator":"Google","purpose":null,"user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-10-09T19:50:34.360Z","last_seen":"2026-10-13T20:31:29.773Z"},"identity":{"verified_requests":7,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":176,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-09T19:50:34.360Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":176,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":4,"peak_at":"2026-10-09T19:50","published_limit":null,"measured_since":"2026-10-09T19:50:34.360Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-09T19:50:34.360Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Yeti","operator":"Naver","purpose":"indexing","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-20T06:52:03.909Z","last_seen":"2026-11-04T01:12:54.549Z"},"identity":{"verified_requests":5,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"reverse_dns","reading":"verified_requests passed forward-confirmed reverse DNS to Naver's own domains (naver.com); forged_claims used this agent's name from addresses that did not (impersonators, not the agent). Reverse-DNS checks began 10 October 2026; earlier hits stay unverifiable"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":2,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-20T06:52:03.909Z","reason":null,"robots_txt_own_fetches":2,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":1,"peak_at":"2026-10-20T06:52","published_limit":null,"measured_since":"2026-10-20T06:52:03.909Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-20T06:52:03.909Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-20":1,"2026-10-29":2,"2026-11-04":2},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-Agent","operator":"Google","purpose":"live fetch","user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-10-31T20:31:56.218Z","last_seen":"2026-11-01T22:11:40.865Z"},"identity":{"verified_requests":5,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":5,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":176,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-31T20:31:56.218Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":176,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":3,"peak_at":"2026-10-31T20:31","published_limit":null,"measured_since":"2026-10-31T20:31:56.218Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-31T20:31:56.218Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-31":4,"2026-11-01":1},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"AdsBot-Google","operator":"Google","purpose":"tool","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-27T09:03:19.135Z","last_seen":"2026-10-30T02:16:38.887Z"},"identity":{"verified_requests":4,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":176,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-27T09:03:19.135Z","reason":"its operator states it ignores the * group and obeys only a group that names it; this robots.txt names it in no group, so no rule here binds it and disallowed fetches are reported, not judged","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":176,"reading":"this agent its operator states it ignores the * group and obeys only a group that names it; this robots.txt names it in no group, so no rule here binds it and disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":2,"peak_at":"2026-10-27T09:03","published_limit":null,"measured_since":"2026-10-27T09:03:19.135Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-27T09:03:19.135Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-27":2,"2026-10-30":2},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-Site-Verification","operator":"Google","purpose":"ownership check","user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-10-03T21:54:40.584Z","last_seen":"2026-10-03T21:54:40.980Z"},"identity":{"verified_requests":2,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":176,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":176,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-Safety","operator":null,"purpose":null,"user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-29T05:16:06.085Z","last_seen":"2026-10-29T05:16:06.085Z"},"identity":{"verified_requests":1,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":176,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-29T05:16:06.085Z","reason":"its operator states it ignores robots.txt rules (a safety crawler), so disallowed fetches are reported, not judged","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":176,"reading":"this agent its operator states it ignores robots.txt rules (a safety crawler), so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":1,"peak_at":"2026-10-29T05:16","published_limit":null,"measured_since":"2026-10-29T05:16:06.085Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-29T05:16:06.085Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-29":1},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Claude-SearchBot","operator":"Anthropic","purpose":"indexing","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-08-16T01:38:23.331Z","last_seen":"2026-10-29T21:47:27.912Z"},"identity":{"verified_requests":0,"forged_claims":55,"forgery_share":1,"signed_requests":0,"failed_signatures":0,"signed_share":null,"strongest_proof":"address","reading":"verified_requests came from the ranges Anthropic publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":null,"robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":55,"verified":0,"spoofed":55},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"MistralAI-User","operator":"Mistral","purpose":"live fetch","user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-08-16T01:38:24.858Z","last_seen":"2026-09-12T16:50:37.743Z"},"identity":{"verified_requests":0,"forged_claims":0,"forgery_share":null,"signed_requests":0,"failed_signatures":0,"signed_share":null,"strongest_proof":"address","reading":"verified_requests came from the ranges Mistral publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"FeedFetcher-Google","operator":"Google","purpose":"feed fetch","user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-09-28T08:19:13.036Z","last_seen":"2026-09-28T08:19:13.036Z"},"identity":{"verified_requests":0,"forged_claims":1,"forgery_share":1,"signed_requests":0,"failed_signatures":0,"signed_share":null,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":176,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":176,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-CloudVertexBot","operator":"Google","purpose":"training","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-30T15:10:18.766Z","last_seen":"2026-10-30T15:10:25.755Z"},"identity":{"verified_requests":0,"forged_claims":3,"forgery_share":1,"signed_requests":0,"failed_signatures":0,"signed_share":null,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":176,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":"Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 176 times here","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":176,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Storebot-Google","operator":"Google","purpose":"indexing","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-31T06:04:36.487Z","last_seen":"2026-11-07T07:39:36.811Z"},"identity":{"verified_requests":0,"forged_claims":7,"forgery_share":1,"signed_requests":0,"failed_signatures":14,"signed_share":null,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":176,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":"Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 176 times here","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":176,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]}],"not_assessed":[{"agent":"Meta-ExternalAgent","claimed_hits":61222,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"Bytespider","claimed_hits":4481,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"Google-Extended","claimed_hits":49,"why":"a robots.txt control token, not a crawler: its operator sends no requests under this name, so nothing here can be judged as its conduct"},{"agent":"CCBot","claimed_hits":1467,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"Diffbot","claimed_hits":2,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"SemrushBot","claimed_hits":19648,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"DataForSeoBot","claimed_hits":11774,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"anthropic-ai","claimed_hits":4,"why":"a robots.txt control token, not a crawler: its operator sends no requests under this name, so nothing here can be judged as its conduct"},{"agent":"Applebot-Extended","claimed_hits":1,"why":"a robots.txt control token, not a crawler: its operator sends no requests under this name, so nothing here can be judged as its conduct"},{"agent":"Timpibot","claimed_hits":4,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"YouBot","claimed_hits":2056,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"Scrapy","claimed_hits":15,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"cohere-ai","claimed_hits":41,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"FacebookBot","claimed_hits":1,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"}],"census":"https://crawlcheck.io/api/telemetry","visitor_checks":"https://crawlcheck.io/api/v1/visitor/stats","code":{"version_id":"e11d6140-2c4c-4704-ac4e-0f46d2dbc49e","script_sha16":"1345f0b7626f3279","state":"attested","deployed":{"at":"2026-11-08T06:36:18.216Z","commit":"d2c43f6ec932","run":"42455363596"},"build_log_entry":1921,"score_version":34,"scoring_policy_sha256":"21af5c858fca26199e01525fec7b19b353df30151b1d29fdeba63a28b90d44a2","decision_rules_sha256":"18dd5636e0896692ab518050cb401744edea2e6f1086dff94fead314959bdc3e","check":"https://crawlcheck.io/api/v1/builds/e11d6140-2c4c-4704-ac4e-0f46d2dbc49e"},"sha256":"37fb515400c3bd48cf52ac1dbeb8d4c4b5a1ed145adc45968d916fdef1501f9a","signature":{"v":1,"alg":"Ed25519","kid":"1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps","over":"sha256","message":"crawlcheck-data-v1\n37fb515400c3bd48cf52ac1dbeb8d4c4b5a1ed145adc45968d916fdef1501f9a","sig":"Tb4q1bcYP7Oq_Lv8NjAfbzc_pQwGpzdsNLy6tMTAQOTSDdk32izdMlLJ0o4nvqDPdf1iOdwwd0u-P2zeRj_oDQ","signed_at":"2026-11-08T23:56:06.036Z","key":{"kid":"1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps","jwk":{"kty":"OKP","crv":"Ed25519","x":"5TmKCOx0TTHepq7HwyXBHvGQpVCSBd2lV8cLzYVqFxQ"},"published_in":"https://crawlcheck.io/.well-known/http-message-signatures-directory"},"canonical":"sha256 over canonical JSON of every field except sha256 and signature"}}