{"kind":"crawlcheck-conduct","v":1,"generated_at":"2026-11-22T01:06:17.203Z","population":"verifiable agents observed on crawlcheck.io, and on the other sites that report through their edge code (portfolio)","signature_coverage":{"agents_assessed":32,"agents_signing":2,"verified_requests":194897,"signed_requests":8016,"failed_signatures":3018,"by_agent":[{"agent":"AhrefsBot","operator":"Ahrefs","signed_requests":7974,"verified_requests":15070,"signed_share":0.529},{"agent":"DuckDuckBot","operator":"DuckDuckGo","signed_requests":42,"verified_requests":203,"signed_share":0.207}],"signed_share":0.041,"counted_correctly_since":"2026-10-29T17:45:00Z","reading":"signed_requests are verified requests that carried a Web Bot Auth signature which verified against the operator's own key directory; signed_share is that count over verified_requests. An agent whose operator publishes no key directory cannot sign, so a low share is a measure of adoption, not of conduct"},"portfolio":{"sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com","emmanuelorta.com"],"sites_with_verified_fetches":7,"days":14,"site_days_judged":92,"judged_from":"2026-11-08","reading":"the other sites that report to CrawlCheck (the operator's own sites), whose edge code reports every verified crawler fetch with its path; a fetch is judged against that site's robots.txt as read on the day of the fetch and the day before (both must disallow it); site-days without both readings are counted as not judged"},"reading":"A census of one property, not a sample of the web: what each agent did HERE, verified by source address against the operator's own published ranges. conduct is no_violations_observed when at least 50 verified requests show no fetch of a path robots.txt disallows; violations_observed when a disallowed path was fetched; insufficient_data below 50. The busiest minute is reported, not judged: this site publishes no crawl rate. site_checks_14d are counts from Visitor Resolve calls other sites made about this agent.","thresholds":{"min_verified_requests":50,"published_rate_limit":null,"disallowed_rules":["/r/","/backup/","/admin-old/"]},"verdicts":{"respected":"robots.txt fetched, and no disallowed path fetched","robots_txt_not_fetched":"no disallowed path fetched, but robots.txt was never fetched under this agent's name here, so it cannot be credited with reading the rules","disallowed_paths_fetched":"a verified fetch of a disallowed path, kept with its time, path and network in disallowed_fetch_records","disallowed_fetch_unrecorded":"disallowed fetches were counted before 2026-10-15, when no per-fetch record was kept; reported, not judged (conduct: not_judged)"},"grades":{"A":"read robots.txt and obeyed it","B":"obeyed, but never fetched robots.txt under its own name here","D":"recorded fetches of disallowed paths, under 0.5% of its verified requests","F":"a fetch of a decoy path named only in robots.txt, or disallowed fetches at 0.5% or more","none":"not enough verified requests, or not judgeable"},"agents":[{"agent":"Amazonbot","operator":"Amazon","purpose":"training","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-13T20:30:33.503Z","last_seen":"2026-11-22T00:40:48.535Z"},"identity":{"verified_requests":37134,"forged_claims":119,"forgery_share":0.003,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Amazon publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":1,"disallowed_fetches":88,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-18T08:45:12.519Z","path":"/r/xufza280d1","net":"18.215.77.0/24"},{"at":"2026-10-18T18:02:03.808Z","path":"/r/xufza280d1","net":"52.205.113.0/24"},{"at":"2026-10-18T18:04:19.333Z","path":"/r/xufza280d1","net":"34.227.156.0/24"},{"at":"2026-10-18T19:28:54.132Z","path":"/r/xufza280d1","net":"54.163.169.0/24"},{"at":"2026-10-18T20:50:07.376Z","path":"/r/xufza280d1","net":"44.220.2.0/24"},{"at":"2026-10-19T23:29:49.080Z","path":"/r/xufza280d1","net":"23.23.137.0/24"},{"at":"2026-10-23T04:17:44.813Z","path":"/r/hsem60u0pb","net":"18.214.186.0/24"},{"at":"2026-10-24T07:16:41.220Z","path":"/r/hsem60u0pb","net":"184.73.68.0/24"},{"at":"2026-10-24T07:16:45.273Z","path":"/r/hsem60u0pb","net":"3.221.156.0/24"},{"at":"2026-10-24T13:42:45.118Z","path":"/r/hsem60u0pb","net":"18.209.137.0/24"},{"at":"2026-10-26T02:57:16.905Z","path":"/r/cz9abl7ob3","net":"18.215.77.0/24"},{"at":"2026-10-27T12:38:34.291Z","path":"/r/cz9abl7ob3","net":"44.197.76.0/24"},{"at":"2026-10-27T14:53:27.270Z","path":"/r/cz9abl7ob3","net":"52.1.106.0/24"},{"at":"2026-10-27T17:03:47.170Z","path":"/r/cz9abl7ob3","net":"107.23.62.0/24"},{"at":"2026-10-27T22:34:39.166Z","path":"/r/cz9abl7ob3","net":"34.231.77.0/24"},{"at":"2026-10-27T23:05:38.747Z","path":"/r/1fb80ss94n","net":"52.45.29.0/24"},{"at":"2026-10-28T06:16:36.232Z","path":"/r/1fb80ss94n","net":"44.207.69.0/24"},{"at":"2026-10-28T08:23:26.019Z","path":"/r/up0rftpp5z","net":"54.147.238.0/24"},{"at":"2026-10-28T11:12:05.504Z","path":"/r/1fb80ss94n","net":"52.204.89.0/24"},{"at":"2026-10-28T11:12:06.136Z","path":"/r/1fb80ss94n","net":"44.194.139.0/24"}],"counted_since":"2026-10-18T06:50:57.435Z","reason":null,"robots_txt_own_fetches":1,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":34,"peak_at":"2026-11-05T10:13","published_limit":null,"measured_since":"2026-10-18T06:50:57.435Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-18T06:50:57.435Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-09":957,"2026-11-10":1008,"2026-11-11":701,"2026-11-12":586,"2026-11-13":519,"2026-11-14":416,"2026-11-15":484,"2026-11-16":437,"2026-11-17":523,"2026-11-18":573,"2026-11-19":1278,"2026-11-20":1562,"2026-11-21":2227,"2026-11-22":4},"portfolio":{"agent":"Amazonbot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":1877,"verified_fetches_not_judged":0,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["88 recorded fetches of disallowed paths, 0.24% of its 37,134 verified requests"]},{"agent":"PerplexityBot","operator":"Perplexity","purpose":"indexing","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-13T02:14:50.805Z","last_seen":"2026-11-22T01:02:30.227Z"},"identity":{"verified_requests":35925,"forged_claims":101,"forgery_share":0.003,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Perplexity publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":267,"disallowed_fetches":74,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-22T08:38:56.498Z","path":"/r/hsem60u0pb","net":"18.97.9.0/24"},{"at":"2026-10-22T15:59:08.322Z","path":"/r/hsem60u0pb","net":"18.97.9.0/24"},{"at":"2026-10-22T15:59:20.457Z","path":"/r/uazt7utoub","net":"18.97.9.0/24"},{"at":"2026-10-24T07:38:31.838Z","path":"/r/hsem60u0pb","net":"18.97.9.0/24"},{"at":"2026-10-24T07:38:34.319Z","path":"/r/phuaa0zazw","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:44.938Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:45.149Z","path":"/r/phuaa0zazw","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:45.922Z","path":"/r/phuaa0zazw","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:46.331Z","path":"/r/phuaa0zazw","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:47.936Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:02.461Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:04.413Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:07.938Z","path":"/r/up0rftpp5z","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:08.999Z","path":"/r/1fb80ss94n","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:09.428Z","path":"/r/up0rftpp5z","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:11.912Z","path":"/r/1fb80ss94n","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:18.485Z","path":"/r/up0rftpp5z","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:18.928Z","path":"/r/up0rftpp5z","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:46.518Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"},{"at":"2026-10-28T20:49:42.689Z","path":"/r/khlmjzqasm","net":"18.97.9.0/24"}],"counted_since":"2026-10-07T04:51:16.683Z","reason":null,"robots_txt_own_fetches":267,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":120,"peak_at":"2026-11-20T06:12","published_limit":null,"measured_since":"2026-10-07T04:51:16.683Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T04:51:16.683Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":209,"verified":107,"spoofed":102},"daily_verified":{"2026-11-09":814,"2026-11-10":722,"2026-11-11":604,"2026-11-12":568,"2026-11-13":692,"2026-11-14":880,"2026-11-15":1250,"2026-11-16":821,"2026-11-17":1286,"2026-11-18":346,"2026-11-19":804,"2026-11-20":737,"2026-11-21":441,"2026-11-22":23},"portfolio":{"agent":"PerplexityBot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com","emmanuelorta.com"],"verified_fetches_judged":172,"verified_fetches_not_judged":0,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["74 recorded fetches of disallowed paths, 0.21% of its 35,925 verified requests"]},{"agent":"YandexBot","operator":"Yandex","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-19T13:53:12.960Z","last_seen":"2026-11-22T00:51:37.689Z"},"identity":{"verified_requests":29922,"forged_claims":20,"forgery_share":0.001,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"reverse_dns","reading":"verified_requests passed forward-confirmed reverse DNS to Yandex's own domains (yandex.ru, yandex.net, yandex.com); forged_claims used this agent's name from addresses that did not (impersonators, not the agent). Reverse-DNS checks began 10 October 2026; earlier hits stay unverifiable"},"robots_txt":{"verdict":"respected","robots_txt_fetches":867,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-10T12:54:23.939Z","reason":null,"robots_txt_own_fetches":867,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":26,"peak_at":"2026-11-06T03:13","published_limit":null,"measured_since":"2026-10-10T12:54:23.939Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-10T12:54:23.939Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-09":765,"2026-11-10":1139,"2026-11-11":982,"2026-11-12":959,"2026-11-13":697,"2026-11-14":1131,"2026-11-15":1003,"2026-11-16":1865,"2026-11-17":1242,"2026-11-18":534,"2026-11-19":1833,"2026-11-20":1602,"2026-11-21":485,"2026-11-22":23},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"AhrefsBot","operator":"Ahrefs","purpose":"tool","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-14T02:11:27.871Z","last_seen":"2026-11-22T01:04:34.481Z"},"identity":{"verified_requests":15070,"forged_claims":3,"forgery_share":0,"signed_requests":7974,"failed_signatures":2356,"signed_share":0.529,"strongest_proof":"signature","reading":"verified_requests came from the ranges Ahrefs publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":249,"disallowed_fetches":12,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-26T07:48:14.796Z","path":"/r/1fb80ss94n","net":"51.68.247.0/24"},{"at":"2026-10-26T08:03:14.376Z","path":"/r/1fb80ss94n","net":"51.75.236.0/24"},{"at":"2026-10-26T08:44:02.288Z","path":"/r/1fb80ss94n","net":"5.39.109.0/24"},{"at":"2026-11-03T14:28:07.385Z","path":"/r/6gaxqt2gao","net":"51.81.110.0/24"},{"at":"2026-11-09T05:19:38.753Z","path":"/r/j2mipohpuh","net":"37.59.204.0/24"},{"at":"2026-11-09T05:27:58.358Z","path":"/r/j2mipohpuh","net":"5.39.109.0/24"},{"at":"2026-11-09T05:36:18.899Z","path":"/r/j2mipohpuh","net":"51.75.236.0/24"},{"at":"2026-11-11T17:56:06.713Z","path":"/r/uazt7utoub","net":"51.222.95.0/24"},{"at":"2026-11-11T17:58:15.475Z","path":"/r/uazt7utoub","net":"15.235.27.0/24"}],"counted_since":"2026-10-07T23:53:27.541Z","reason":null,"robots_txt_own_fetches":249,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":64,"peak_at":"2026-11-01T16:47","published_limit":null,"measured_since":"2026-10-07T23:53:27.541Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":1285,"share_of_verified_since_measured":0.085,"measured_since":"2026-10-07T23:53:27.541Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-09":359,"2026-11-10":221,"2026-11-11":373,"2026-11-12":104,"2026-11-13":700,"2026-11-14":350,"2026-11-15":420,"2026-11-16":440,"2026-11-17":693,"2026-11-18":225,"2026-11-19":562,"2026-11-20":142,"2026-11-21":147,"2026-11-22":162},"portfolio":{"agent":"AhrefsBot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":2575,"verified_fetches_not_judged":0,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["12 recorded fetches of disallowed paths, 0.08% of its 15,070 verified requests"]},{"agent":"ClaudeBot","operator":"Anthropic","purpose":"training","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-13T02:14:49.992Z","last_seen":"2026-11-21T14:08:15.664Z"},"identity":{"verified_requests":13375,"forged_claims":124,"forgery_share":0.009,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Anthropic publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":295,"disallowed_fetches":3,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-16T09:27:57.482Z","path":"/r/uazt7utoub","net":"216.73.216.0/24"},{"at":"2026-10-16T09:31:12.061Z","path":"/r/uazt7utoub","net":"216.73.216.0/24"},{"at":"2026-10-16T09:38:15.950Z","path":"/r/uazt7utoub","net":"216.73.216.0/24"}],"counted_since":"2026-10-07T03:48:54.788Z","reason":null,"robots_txt_own_fetches":294,"robots_txt_sibling_fetches":1,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":69,"peak_at":"2026-11-20T13:40","published_limit":null,"measured_since":"2026-10-07T03:48:54.788Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T03:48:54.788Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":334,"verified":190,"spoofed":144},"daily_verified":{"2026-11-06":15,"2026-11-07":2,"2026-11-08":2,"2026-11-09":10,"2026-11-10":182,"2026-11-11":5,"2026-11-12":65,"2026-11-13":44,"2026-11-14":24,"2026-11-15":12,"2026-11-16":4,"2026-11-17":3,"2026-11-20":262,"2026-11-21":3},"portfolio":{"agent":"ClaudeBot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":1392,"verified_fetches_not_judged":0,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["3 recorded fetches of disallowed paths, 0.02% of its 13,375 verified requests"]},{"agent":"Applebot","operator":"Apple","purpose":"indexing","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-13T12:32:42.174Z","last_seen":"2026-11-22T00:58:55.991Z"},"identity":{"verified_requests":12683,"forged_claims":84,"forgery_share":0.007,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Apple publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":103,"disallowed_fetches":40,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-23T13:32:15.215Z","path":"/r/8dsc0mu5ks","net":"17.166.233.0/24"},{"at":"2026-10-23T13:39:38.180Z","path":"/r/8dsc0mu5ks","net":"17.166.232.0/24"},{"at":"2026-10-24T07:28:50.585Z","path":"/r/8dsc0mu5ks","net":"17.166.235.0/24"},{"at":"2026-10-24T07:32:58.177Z","path":"/r/8dsc0mu5ks","net":"17.166.236.0/24"},{"at":"2026-10-24T16:13:56.254Z","path":"/r/mkx08rapce","net":"17.166.232.0/24"},{"at":"2026-10-24T16:17:40.243Z","path":"/r/mkx08rapce","net":"17.166.235.0/24"},{"at":"2026-10-26T15:31:58.651Z","path":"/r/8dsc0mu5ks","net":"17.166.232.0/24"},{"at":"2026-10-26T15:34:20.954Z","path":"/r/8dsc0mu5ks","net":"17.166.232.0/24"},{"at":"2026-10-26T16:01:58.942Z","path":"/r/8dsc0mu5ks","net":"17.166.25.0/24"},{"at":"2026-10-26T16:06:50.241Z","path":"/r/8dsc0mu5ks","net":"17.166.21.0/24"},{"at":"2026-10-27T20:22:26.222Z","path":"/r/mkx08rapce","net":"17.166.235.0/24"},{"at":"2026-10-27T20:37:56.625Z","path":"/r/mkx08rapce","net":"17.166.233.0/24"},{"at":"2026-10-28T01:54:46.237Z","path":"/r/mkx08rapce","net":"17.166.21.0/24"},{"at":"2026-10-28T01:55:55.986Z","path":"/r/mkx08rapce","net":"17.166.22.0/24"},{"at":"2026-10-28T12:27:38.731Z","path":"/r/mkx08rapce","net":"17.166.153.0/24"},{"at":"2026-10-28T12:33:52.496Z","path":"/r/mkx08rapce","net":"17.166.152.0/24"},{"at":"2026-10-30T18:31:10.592Z","path":"/r/khlmjzqasm","net":"17.166.153.0/24"},{"at":"2026-10-30T18:31:54.377Z","path":"/r/khlmjzqasm","net":"17.166.154.0/24"},{"at":"2026-11-02T05:43:52.587Z","path":"/r/aqxn4lxkaw","net":"17.166.151.0/24"},{"at":"2026-11-02T05:51:03.321Z","path":"/r/aqxn4lxkaw","net":"17.166.154.0/24"}],"counted_since":"2026-10-07T03:15:19.709Z","reason":null,"robots_txt_own_fetches":103,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":8,"peak_at":"2026-10-21T17:47","published_limit":null,"measured_since":"2026-10-07T03:15:19.709Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T03:15:19.709Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":1104,"verified":1045,"spoofed":59},"daily_verified":{"2026-11-09":818,"2026-11-10":460,"2026-11-11":826,"2026-11-12":695,"2026-11-13":630,"2026-11-14":382,"2026-11-15":308,"2026-11-16":89,"2026-11-17":66,"2026-11-18":70,"2026-11-19":49,"2026-11-20":115,"2026-11-21":90,"2026-11-22":14},"portfolio":{"agent":"Applebot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":1761,"verified_fetches_not_judged":0,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["40 recorded fetches of disallowed paths, 0.32% of its 12,683 verified requests"]},{"agent":"Googlebot","operator":"Google","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-13T02:14:51.611Z","last_seen":"2026-11-22T00:58:44.836Z"},"identity":{"verified_requests":12437,"forged_claims":266,"forgery_share":0.021,"signed_requests":0,"failed_signatures":392,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":215,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T03:18:09.861Z","reason":null,"robots_txt_own_fetches":215,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":13,"peak_at":"2026-10-19T02:10","published_limit":null,"measured_since":"2026-10-07T03:18:09.861Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":1765,"share_of_verified_since_measured":0.142,"measured_since":"2026-10-07T03:18:09.861Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":1975,"verified":1237,"spoofed":738},"daily_verified":{"2026-11-09":90,"2026-11-10":133,"2026-11-11":265,"2026-11-12":397,"2026-11-13":326,"2026-11-14":168,"2026-11-15":125,"2026-11-16":152,"2026-11-17":158,"2026-11-18":126,"2026-11-19":238,"2026-11-20":174,"2026-11-21":110,"2026-11-22":10},"portfolio":{"agent":"Googlebot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com","emmanuelorta.com"],"verified_fetches_judged":3012,"verified_fetches_not_judged":0,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"Bingbot","operator":"Microsoft","purpose":"indexing","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-14T06:25:41.331Z","last_seen":"2026-11-22T00:55:54.476Z"},"identity":{"verified_requests":11274,"forged_claims":52,"forgery_share":0.005,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Microsoft publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":134,"disallowed_fetches":1,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-27T19:17:43.386Z","path":"/r/uazt7utoub","net":"40.77.167.0/24"}],"counted_since":"2026-10-07T03:20:08.676Z","reason":null,"robots_txt_own_fetches":134,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":12,"peak_at":"2026-11-18T00:05","published_limit":null,"measured_since":"2026-10-07T03:20:08.676Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":986,"share_of_verified_since_measured":0.087,"measured_since":"2026-10-07T03:20:08.676Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":1451,"verified":1334,"spoofed":117},"daily_verified":{"2026-11-09":114,"2026-11-10":105,"2026-11-11":224,"2026-11-12":257,"2026-11-13":237,"2026-11-14":324,"2026-11-15":273,"2026-11-16":538,"2026-11-17":255,"2026-11-18":91,"2026-11-19":402,"2026-11-20":432,"2026-11-21":228,"2026-11-22":19},"portfolio":{"agent":"Bingbot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com","emmanuelorta.com"],"verified_fetches_judged":1833,"verified_fetches_not_judged":0,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["1 recorded fetch of disallowed paths, 0.01% of its 11,274 verified requests"]},{"agent":"GPTBot","operator":"OpenAI","purpose":"training","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-12T06:20:42.861Z","last_seen":"2026-11-22T00:09:48.836Z"},"identity":{"verified_requests":10547,"forged_claims":184,"forgery_share":0.017,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges OpenAI publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"robots_txt_not_fetched","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T18:09:36.470Z","reason":null,"robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":120,"peak_at":"2026-11-02T11:28","published_limit":null,"measured_since":"2026-10-07T18:09:36.470Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T18:09:36.470Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":779,"verified":184,"spoofed":595},"daily_verified":{"2026-11-09":2,"2026-11-10":2,"2026-11-11":410,"2026-11-12":1,"2026-11-13":28,"2026-11-14":76,"2026-11-15":464,"2026-11-16":85,"2026-11-17":244,"2026-11-18":570,"2026-11-19":160,"2026-11-20":118,"2026-11-21":63,"2026-11-22":20},"portfolio":{"agent":"GPTBot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com","emmanuelorta.com"],"verified_fetches_judged":689,"verified_fetches_not_judged":0,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"B","grade_reasons":["no disallowed path fetched, but it never fetched robots.txt under its own name here, so it cannot be credited with reading the rules"]},{"agent":"OAI-SearchBot","operator":"OpenAI","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-13T02:14:49.166Z","last_seen":"2026-11-22T00:55:25.430Z"},"identity":{"verified_requests":9105,"forged_claims":111,"forgery_share":0.012,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges OpenAI publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":351,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T03:30:28.967Z","reason":null,"robots_txt_own_fetches":351,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":119,"peak_at":"2026-10-29T12:58","published_limit":null,"measured_since":"2026-10-07T03:30:28.967Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T03:30:28.967Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":826,"verified":737,"spoofed":89},"daily_verified":{"2026-11-09":338,"2026-11-10":203,"2026-11-11":182,"2026-11-12":131,"2026-11-13":185,"2026-11-14":283,"2026-11-15":216,"2026-11-16":207,"2026-11-17":273,"2026-11-18":192,"2026-11-19":407,"2026-11-20":204,"2026-11-21":140,"2026-11-22":20},"portfolio":{"agent":"OAI-SearchBot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com","emmanuelorta.com"],"verified_fetches_judged":1213,"verified_fetches_not_judged":0,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"ChatGPT-User","operator":"OpenAI","purpose":"live fetch","user_triggered":true,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-13T22:52:23.214Z","last_seen":"2026-11-22T01:00:21.761Z"},"identity":{"verified_requests":1778,"forged_claims":90,"forgery_share":0.048,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges OpenAI publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"not_applicable","robots_txt_fetches":1,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T03:26:16.042Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":1,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":13,"peak_at":"2026-10-15T19:36","published_limit":null,"measured_since":"2026-10-07T03:26:16.042Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T03:26:16.042Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":199,"verified":103,"spoofed":96},"daily_verified":{"2026-11-09":60,"2026-11-10":72,"2026-11-11":27,"2026-11-12":28,"2026-11-13":30,"2026-11-14":25,"2026-11-15":20,"2026-11-16":29,"2026-11-17":16,"2026-11-18":15,"2026-11-19":29,"2026-11-20":30,"2026-11-21":10,"2026-11-22":1},"portfolio":{"agent":"ChatGPT-User","sites":["supremefencingdenver.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":148,"verified_fetches_not_judged":0,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"A","grade_reasons":["fetches pages a person asked for; its operator states robots.txt does not govern those fetches, and every counted request came from its published addresses"]},{"agent":"Claude-User","operator":"Anthropic","purpose":"live fetch","user_triggered":true,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-12T22:57:31.977Z","last_seen":"2026-11-21T14:24:15.419Z"},"identity":{"verified_requests":1607,"forged_claims":55,"forgery_share":0.033,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Anthropic publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"not_applicable","robots_txt_fetches":18,"disallowed_fetches":1,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-17T04:44:31.376Z","path":"/r/zqdnb5b58f","net":"160.79.106.0/24"}],"counted_since":"2026-10-07T11:03:04.563Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":18,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":7,"peak_at":"2026-11-15T13:58","published_limit":null,"measured_since":"2026-10-07T11:03:04.563Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T11:03:04.563Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":120,"verified":25,"spoofed":95},"daily_verified":{"2026-10-25":1,"2026-11-13":143,"2026-11-14":284,"2026-11-15":208,"2026-11-16":208,"2026-11-17":151,"2026-11-18":98,"2026-11-19":112,"2026-11-20":52,"2026-11-21":21},"portfolio":{"agent":"Claude-User","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com","emmanuelorta.com"],"verified_fetches_judged":70,"verified_fetches_not_judged":0,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"A","grade_reasons":["fetches pages a person asked for; its operator states robots.txt does not govern those fetches, and every counted request came from its published addresses"]},{"agent":"SeznamBot","operator":"Seznam","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-10-10T13:04:32.776Z","last_seen":"2026-11-21T14:35:31.033Z"},"identity":{"verified_requests":1156,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"reverse_dns","reading":"verified_requests passed forward-confirmed reverse DNS to Seznam's own domains (seznam.cz); forged_claims used this agent's name from addresses that did not (impersonators, not the agent). Reverse-DNS checks began 10 October 2026; earlier hits stay unverifiable"},"robots_txt":{"verdict":"respected","robots_txt_fetches":28,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-10T13:04:32.776Z","reason":null,"robots_txt_own_fetches":28,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":3,"peak_at":"2026-10-10T22:12","published_limit":null,"measured_since":"2026-10-10T13:04:32.776Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":960,"share_of_verified_since_measured":0.83,"measured_since":"2026-10-10T13:04:32.776Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-06":43,"2026-11-07":9,"2026-11-08":2,"2026-11-09":35,"2026-11-10":43,"2026-11-11":15,"2026-11-13":14,"2026-11-14":13,"2026-11-15":59,"2026-11-16":4,"2026-11-17":42,"2026-11-19":28,"2026-11-20":24,"2026-11-21":20},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"Baiduspider","operator":"Baidu","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-13T21:31:01.445Z","last_seen":"2026-11-22T00:08:53.337Z"},"identity":{"verified_requests":691,"forged_claims":45,"forgery_share":0.061,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"reverse_dns","reading":"verified_requests passed forward-confirmed reverse DNS to Baidu's own domains (baidu.com, baidu.jp); forged_claims used this agent's name from addresses that did not (impersonators, not the agent). Reverse-DNS checks began 10 October 2026; earlier hits stay unverifiable"},"robots_txt":{"verdict":"robots_txt_not_fetched","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-10T13:14:25.713Z","reason":null,"robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":5,"peak_at":"2026-10-26T00:08","published_limit":null,"measured_since":"2026-10-10T13:14:25.713Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-10T13:14:25.713Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-08":16,"2026-11-09":5,"2026-11-10":17,"2026-11-11":54,"2026-11-12":9,"2026-11-13":29,"2026-11-14":14,"2026-11-15":23,"2026-11-16":16,"2026-11-17":4,"2026-11-18":5,"2026-11-19":11,"2026-11-20":23,"2026-11-22":1},"grade":"B","grade_reasons":["no disallowed path fetched, but it never fetched robots.txt under its own name here, so it cannot be credited with reading the rules"]},{"agent":"CCBot","operator":"Common Crawl","purpose":"training","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-15T17:58:20.715Z","last_seen":"2026-11-18T05:08:43.011Z"},"identity":{"verified_requests":636,"forged_claims":9,"forgery_share":0.014,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Common Crawl publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":2,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-11-09T16:57:56.236Z","reason":null,"robots_txt_own_fetches":2,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":7,"peak_at":"2026-11-09T18:40","published_limit":null,"measured_since":"2026-11-09T16:57:56.236Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-11-09T16:57:56.236Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-09":187,"2026-11-10":7,"2026-11-11":183,"2026-11-14":187,"2026-11-17":69},"portfolio":{"agent":"CCBot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":188,"verified_fetches_not_judged":0,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"Google-CWS","operator":"Google","purpose":"store listing fetch","user_triggered":true,"conduct":"no_violations_observed","window":{"first_seen":"2026-09-18T16:25:39.959Z","last_seen":"2026-11-19T06:28:26.432Z"},"identity":{"verified_requests":478,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"not_applicable","robots_txt_fetches":217,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T11:31:33.296Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":217,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":3,"peak_at":"2026-10-07T20:31","published_limit":null,"measured_since":"2026-10-07T11:31:33.296Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T11:31:33.296Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-05":3,"2026-11-06":8,"2026-11-07":14,"2026-11-08":18,"2026-11-09":12,"2026-11-10":5,"2026-11-11":2,"2026-11-13":4,"2026-11-14":11,"2026-11-15":8,"2026-11-16":5,"2026-11-17":8,"2026-11-18":6,"2026-11-19":1},"grade":"A","grade_reasons":["fetches pages a person asked for; its operator states robots.txt does not govern those fetches, and every counted request came from its published addresses"]},{"agent":"Google-InspectionTool","operator":"Google","purpose":null,"user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-10-27T11:15:26.957Z","last_seen":"2026-11-19T08:45:43.383Z"},"identity":{"verified_requests":361,"forged_claims":5,"forgery_share":0.014,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":217,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-29T14:57:13.241Z","reason":"Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 217 times here","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":217,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":7,"peak_at":"2026-10-29T21:37","published_limit":null,"measured_since":"2026-10-29T14:57:13.241Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-29T14:57:13.241Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-04":4,"2026-11-06":8,"2026-11-07":29,"2026-11-08":10,"2026-11-09":6,"2026-11-10":1,"2026-11-11":2,"2026-11-13":9,"2026-11-14":6,"2026-11-15":5,"2026-11-16":7,"2026-11-17":4,"2026-11-18":3,"2026-11-19":6},"grade":"A","grade_reasons":["Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 217 times here, and it fetched no path robots.txt disallows"]},{"agent":"Google-Agent","operator":"Google","purpose":"live fetch","user_triggered":true,"conduct":"no_violations_observed","window":{"first_seen":"2026-10-31T20:31:56.218Z","last_seen":"2026-11-16T14:11:05.215Z"},"identity":{"verified_requests":281,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":168,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"not_applicable","robots_txt_fetches":2,"disallowed_fetches":3,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-11-09T19:22:08.454Z","path":"/r/by2qvflvb4","net":"136.121.28.0/24"},{"at":"2026-11-09T19:22:38.694Z","path":"/r/by2qvflvb4","net":"136.121.28.0/24"},{"at":"2026-11-09T19:23:26.441Z","path":"/r/by2qvflvb4","net":"136.121.28.0/24"}],"counted_since":"2026-10-31T20:31:56.218Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":2,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":17,"peak_at":"2026-11-09T19:13","published_limit":null,"measured_since":"2026-10-31T20:31:56.218Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":14,"share_of_verified_since_measured":0.05,"measured_since":"2026-10-31T20:31:56.218Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-31":4,"2026-11-01":1,"2026-11-09":120,"2026-11-13":42,"2026-11-16":1},"grade":"A","grade_reasons":["fetches pages a person asked for; its operator states robots.txt does not govern those fetches, and every counted request came from its published addresses"]},{"agent":"DuckDuckBot","operator":"DuckDuckGo","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-12T20:29:50.246Z","last_seen":"2026-11-22T00:56:15.014Z"},"identity":{"verified_requests":203,"forged_claims":0,"forgery_share":0,"signed_requests":42,"failed_signatures":86,"signed_share":0.207,"strongest_proof":"signature","reading":"verified_requests came from the ranges DuckDuckGo publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":3,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-10T21:21:35.448Z","reason":null,"robots_txt_own_fetches":3,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":2,"peak_at":"2026-10-10T21:21","published_limit":null,"measured_since":"2026-10-10T21:21:35.448Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":4,"share_of_verified_since_measured":0.02,"measured_since":"2026-10-10T21:21:35.448Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-09":3,"2026-11-10":3,"2026-11-11":2,"2026-11-12":7,"2026-11-13":4,"2026-11-14":8,"2026-11-15":4,"2026-11-16":7,"2026-11-17":1,"2026-11-18":2,"2026-11-19":4,"2026-11-20":10,"2026-11-21":4,"2026-11-22":2},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows","42 of its requests carried a Web Bot Auth signature that verified against its operator's own key (the strongest identity proof)"]},{"agent":"GoogleOther","operator":"Google","purpose":"training","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-10-28T07:22:36.856Z","last_seen":"2026-11-21T14:31:49.482Z"},"identity":{"verified_requests":193,"forged_claims":134,"forgery_share":0.41,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":217,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-28T07:22:36.856Z","reason":"Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 217 times here","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":217,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":5,"peak_at":"2026-11-15T19:39","published_limit":null,"measured_since":"2026-10-28T07:22:36.856Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-28T07:22:36.856Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-08":2,"2026-11-09":24,"2026-11-10":1,"2026-11-11":3,"2026-11-12":4,"2026-11-13":10,"2026-11-14":4,"2026-11-15":31,"2026-11-16":10,"2026-11-17":13,"2026-11-18":10,"2026-11-19":10,"2026-11-20":7,"2026-11-21":10},"grade":"A","grade_reasons":["Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 217 times here, and it fetched no path robots.txt disallows"]},{"agent":"Perplexity-User","operator":"Perplexity","purpose":"live fetch","user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-08-14T01:04:08.513Z","last_seen":"2026-11-18T05:08:40.728Z"},"identity":{"verified_requests":9,"forged_claims":52,"forgery_share":0.852,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Perplexity publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":1,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-11-07T19:14:23.425Z","path":"/r/a7qr2s89eu.md","net":"18.97.43.0/24"}],"counted_since":"2026-11-07T19:14:23.425Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":1,"peak_at":"2026-11-07T19:14","published_limit":null,"measured_since":"2026-11-07T19:14:23.425Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-11-07T19:14:23.425Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":59,"verified":0,"spoofed":59},"daily_verified":{"2026-11-07":1},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-NotebookLM","operator":"Google","purpose":null,"user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-10-09T19:50:34.360Z","last_seen":"2026-11-14T11:56:22.335Z"},"identity":{"verified_requests":8,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":217,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-09T19:50:34.360Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":217,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":4,"peak_at":"2026-10-09T19:50","published_limit":null,"measured_since":"2026-10-09T19:50:34.360Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-09T19:50:34.360Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-14":1},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-Read-Aloud","operator":null,"purpose":null,"user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-11-09T02:47:57.686Z","last_seen":"2026-11-09T02:48:00.569Z"},"identity":{"verified_requests":7,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":217,"disallowed_fetches":2,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-11-09T02:47:57.686Z","path":"/r/j2mipohpuh","net":"66.249.83.0/24"},{"at":"2026-11-09T02:47:58.799Z","path":"/r/j2mipohpuh","net":"66.249.83.0/24"}],"counted_since":"2026-11-09T02:47:57.686Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":217,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":3,"peak_at":"2026-11-09T02:47","published_limit":null,"measured_since":"2026-11-09T02:47:57.686Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-11-09T02:47:57.686Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-09":4},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Yeti","operator":"Naver","purpose":"indexing","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-20T06:52:03.909Z","last_seen":"2026-11-10T11:49:02.423Z"},"identity":{"verified_requests":6,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"reverse_dns","reading":"verified_requests passed forward-confirmed reverse DNS to Naver's own domains (naver.com); forged_claims used this agent's name from addresses that did not (impersonators, not the agent). Reverse-DNS checks began 10 October 2026; earlier hits stay unverifiable"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":3,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-20T06:52:03.909Z","reason":null,"robots_txt_own_fetches":3,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":1,"peak_at":"2026-10-20T06:52","published_limit":null,"measured_since":"2026-10-20T06:52:03.909Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-20T06:52:03.909Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-20":1,"2026-10-29":2,"2026-11-04":2,"2026-11-10":1},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"AdsBot-Google","operator":"Google","purpose":"tool","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-27T09:03:19.135Z","last_seen":"2026-11-13T13:13:52.654Z"},"identity":{"verified_requests":5,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":217,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-27T09:03:19.135Z","reason":"its operator states it ignores the * group and obeys only a group that names it; this robots.txt names it in no group, so no rule here binds it and disallowed fetches are reported, not judged","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":217,"reading":"this agent its operator states it ignores the * group and obeys only a group that names it; this robots.txt names it in no group, so no rule here binds it and disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":2,"peak_at":"2026-10-27T09:03","published_limit":null,"measured_since":"2026-10-27T09:03:19.135Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-27T09:03:19.135Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-27":2,"2026-10-30":2,"2026-11-13":1},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-Safety","operator":null,"purpose":null,"user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-29T05:16:06.085Z","last_seen":"2026-11-15T18:57:35.722Z"},"identity":{"verified_requests":4,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":217,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-29T05:16:06.085Z","reason":"its operator states it ignores robots.txt rules (a safety crawler), so disallowed fetches are reported, not judged","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":217,"reading":"this agent its operator states it ignores robots.txt rules (a safety crawler), so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":2,"peak_at":"2026-11-15T18:49","published_limit":null,"measured_since":"2026-10-29T05:16:06.085Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-29T05:16:06.085Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-29":1,"2026-11-15":3},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-Site-Verification","operator":"Google","purpose":"ownership check","user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-10-03T21:54:40.584Z","last_seen":"2026-10-03T21:54:40.980Z"},"identity":{"verified_requests":2,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":217,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":217,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Claude-SearchBot","operator":"Anthropic","purpose":"indexing","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-08-16T01:38:23.331Z","last_seen":"2026-11-18T05:08:39.337Z"},"identity":{"verified_requests":0,"forged_claims":58,"forgery_share":1,"signed_requests":0,"failed_signatures":0,"signed_share":null,"strongest_proof":"address","reading":"verified_requests came from the ranges Anthropic publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":1,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":"robots.txt was fetched 1 time under its name from another of its operator’s published crawler ranges","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":1,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":87,"verified":0,"spoofed":87},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"MistralAI-User","operator":"Mistral","purpose":"live fetch","user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-08-16T01:38:24.858Z","last_seen":"2026-09-12T16:50:37.743Z"},"identity":{"verified_requests":0,"forged_claims":0,"forgery_share":null,"signed_requests":0,"failed_signatures":0,"signed_share":null,"strongest_proof":"address","reading":"verified_requests came from the ranges Mistral publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"FeedFetcher-Google","operator":"Google","purpose":"feed fetch","user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-09-28T08:19:13.036Z","last_seen":"2026-09-28T08:19:13.036Z"},"identity":{"verified_requests":0,"forged_claims":1,"forgery_share":1,"signed_requests":0,"failed_signatures":0,"signed_share":null,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":217,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":217,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-CloudVertexBot","operator":"Google","purpose":"tool","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-30T15:10:18.766Z","last_seen":"2026-10-30T15:10:25.755Z"},"identity":{"verified_requests":0,"forged_claims":3,"forgery_share":1,"signed_requests":0,"failed_signatures":0,"signed_share":null,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":217,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":"Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 217 times here","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":217,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Storebot-Google","operator":"Google","purpose":"indexing","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-31T06:04:36.487Z","last_seen":"2026-11-17T00:47:58.531Z"},"identity":{"verified_requests":0,"forged_claims":8,"forgery_share":1,"signed_requests":0,"failed_signatures":16,"signed_share":null,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":217,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":"Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 217 times here","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":217,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]}],"not_assessed":[{"agent":"Meta-ExternalAgent","claimed_hits":126727,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"Bytespider","claimed_hits":20109,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"Google-Extended","claimed_hits":49,"why":"a robots.txt control token, not a crawler: its operator sends no requests under this name, so nothing here can be judged as its conduct"},{"agent":"Diffbot","claimed_hits":2,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"SemrushBot","claimed_hits":30353,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"DataForSeoBot","claimed_hits":16743,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"anthropic-ai","claimed_hits":4,"why":"a robots.txt control token, not a crawler: its operator sends no requests under this name, so nothing here can be judged as its conduct"},{"agent":"Applebot-Extended","claimed_hits":2,"why":"a robots.txt control token, not a crawler: its operator sends no requests under this name, so nothing here can be judged as its conduct"},{"agent":"Timpibot","claimed_hits":4,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"YouBot","claimed_hits":2524,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"Scrapy","claimed_hits":18,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"cohere-ai","claimed_hits":62,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"FacebookBot","claimed_hits":1,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"}],"census":"https://crawlcheck.io/api/telemetry","visitor_checks":"https://crawlcheck.io/api/v1/visitor/stats","code":{"build_id":"92219fb5c4286f0a","script_sha16":"b00e794a299c5a1b","state":"attested","deployed":{"at":"2026-11-21T18:41:30.008Z"},"build_log_entry":2900,"score_version":43,"scoring_policy_sha256":"3e08fccfbbce356dc82ecd302b32a226a36fc0fb3150a4ab72248fb5795372af","decision_rules_sha256":"18dd5636e0896692ab518050cb401744edea2e6f1086dff94fead314959bdc3e","check":"https://crawlcheck.io/api/v1/builds/92219fb5c4286f0a"},"sha256":"c59ab23e4c4499821830075a3a5ed4f5ba9c596d29ef837c3345ea53bb7814d6","signature":{"v":1,"alg":"Ed25519","kid":"1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps","over":"sha256","message":"crawlcheck-data-v1\nc59ab23e4c4499821830075a3a5ed4f5ba9c596d29ef837c3345ea53bb7814d6","sig":"QL7ET8wYoL4dlULipCk_Qiu3q3ULAnXEJrqhpEgEWFqaiQabYfPlsxnX_f-sdWQBz8-znToMf4Nz-Fs_xW9TDA","signed_at":"2026-11-22T01:06:17.203Z","key":{"kid":"1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps","jwk":{"kty":"OKP","crv":"Ed25519","x":"5TmKCOx0TTHepq7HwyXBHvGQpVCSBd2lV8cLzYVqFxQ"},"published_in":"https://crawlcheck.io/.well-known/http-message-signatures-directory"},"canonical":"sha256 over canonical JSON of every field except sha256 and signature"}}