{"kind":"crawlcheck-conduct","v":1,"generated_at":"2026-11-15T23:51:15.108Z","population":"verifiable agents observed on crawlcheck.io, and on the other sites that report through their edge Workers (portfolio)","signature_coverage":{"agents_assessed":32,"agents_signing":2,"verified_requests":150609,"signed_requests":5637,"failed_signatures":3006,"by_agent":[{"agent":"AhrefsBot","operator":"Ahrefs","signed_requests":5608,"verified_requests":12464,"signed_share":0.45},{"agent":"DuckDuckBot","operator":"DuckDuckGo","signed_requests":29,"verified_requests":168,"signed_share":0.173}],"signed_share":0.037,"counted_correctly_since":"2026-10-29T17:45:00Z","reading":"signed_requests are verified requests that carried a Web Bot Auth signature which verified against the operator's own key directory; signed_share is that count over verified_requests. An agent whose operator publishes no key directory cannot sign, so a low share is a measure of adoption, not of conduct"},"portfolio":{"sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com","emmanuelorta.com"],"sites_with_verified_fetches":7,"days":14,"site_days_judged":56,"judged_from":"2026-11-07","reading":"the other sites that report to CrawlCheck (the operator's own sites), whose edge Workers report every verified crawler fetch with its path; a fetch is judged against that site's robots.txt as read on the day of the fetch and the day before (both must disallow it); site-days without both readings are counted as not judged"},"reading":"A census of one property, not a sample of the web: what each agent did HERE, verified by source address against the operator's own published ranges. conduct is no_violations_observed when at least 50 verified requests show no fetch of a path robots.txt disallows; violations_observed when a disallowed path was fetched; insufficient_data below 50. The busiest minute is reported, not judged: this site publishes no crawl rate. site_checks_14d are counts from Visitor Resolve calls other sites made about this agent.","thresholds":{"min_verified_requests":50,"published_rate_limit":null,"disallowed_rules":["/r/","/backup/","/admin-old/"]},"verdicts":{"respected":"robots.txt fetched, and no disallowed path fetched","robots_txt_not_fetched":"no disallowed path fetched, but robots.txt was never fetched under this agent's name here, so it cannot be credited with reading the rules","disallowed_paths_fetched":"a verified fetch of a disallowed path, kept with its time, path and network in disallowed_fetch_records","disallowed_fetch_unrecorded":"disallowed fetches were counted before 2026-10-15, when no per-fetch record was kept; reported, not judged (conduct: not_judged)"},"grades":{"A":"read robots.txt and obeyed it","B":"obeyed, but never fetched robots.txt under its own name here","D":"recorded fetches of disallowed paths, under 0.5% of its verified requests","F":"a fetch of a decoy path named only in robots.txt, or disallowed fetches at 0.5% or more","none":"not enough verified requests, or not judgeable"},"agents":[{"agent":"Amazonbot","operator":"Amazon","purpose":"training","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-13T20:30:33.503Z","last_seen":"2026-11-15T23:32:43.236Z"},"identity":{"verified_requests":29126,"forged_claims":119,"forgery_share":0.004,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Amazon publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":1,"disallowed_fetches":80,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-18T08:45:12.519Z","path":"/r/xufza280d1","net":"18.215.77.0/24"},{"at":"2026-10-18T18:02:03.808Z","path":"/r/xufza280d1","net":"52.205.113.0/24"},{"at":"2026-10-18T18:04:19.333Z","path":"/r/xufza280d1","net":"34.227.156.0/24"},{"at":"2026-10-18T19:28:54.132Z","path":"/r/xufza280d1","net":"54.163.169.0/24"},{"at":"2026-10-18T20:50:07.376Z","path":"/r/xufza280d1","net":"44.220.2.0/24"},{"at":"2026-10-19T23:29:49.080Z","path":"/r/xufza280d1","net":"23.23.137.0/24"},{"at":"2026-10-23T04:17:44.813Z","path":"/r/hsem60u0pb","net":"18.214.186.0/24"},{"at":"2026-10-24T07:16:41.220Z","path":"/r/hsem60u0pb","net":"184.73.68.0/24"},{"at":"2026-10-24T07:16:45.273Z","path":"/r/hsem60u0pb","net":"3.221.156.0/24"},{"at":"2026-10-24T13:42:45.118Z","path":"/r/hsem60u0pb","net":"18.209.137.0/24"},{"at":"2026-10-26T02:57:16.905Z","path":"/r/cz9abl7ob3","net":"18.215.77.0/24"},{"at":"2026-10-27T12:38:34.291Z","path":"/r/cz9abl7ob3","net":"44.197.76.0/24"},{"at":"2026-10-27T14:53:27.270Z","path":"/r/cz9abl7ob3","net":"52.1.106.0/24"},{"at":"2026-10-27T17:03:47.170Z","path":"/r/cz9abl7ob3","net":"107.23.62.0/24"},{"at":"2026-10-27T22:34:39.166Z","path":"/r/cz9abl7ob3","net":"34.231.77.0/24"},{"at":"2026-10-27T23:05:38.747Z","path":"/r/1fb80ss94n","net":"52.45.29.0/24"},{"at":"2026-10-28T06:16:36.232Z","path":"/r/1fb80ss94n","net":"44.207.69.0/24"},{"at":"2026-10-28T08:23:26.019Z","path":"/r/up0rftpp5z","net":"54.147.238.0/24"},{"at":"2026-10-28T11:12:05.504Z","path":"/r/1fb80ss94n","net":"52.204.89.0/24"},{"at":"2026-10-28T11:12:06.136Z","path":"/r/1fb80ss94n","net":"44.194.139.0/24"}],"counted_since":"2026-10-18T06:50:57.435Z","reason":null,"robots_txt_own_fetches":1,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":34,"peak_at":"2026-11-05T10:13","published_limit":null,"measured_since":"2026-10-18T06:50:57.435Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-18T06:50:57.435Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-02":626,"2026-11-03":1015,"2026-11-04":1454,"2026-11-05":2406,"2026-11-06":2351,"2026-11-07":1015,"2026-11-08":696,"2026-11-09":957,"2026-11-10":1008,"2026-11-11":701,"2026-11-12":586,"2026-11-13":519,"2026-11-14":416,"2026-11-15":469},"portfolio":{"agent":"Amazonbot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":1103,"verified_fetches_not_judged":337,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["80 recorded fetches of disallowed paths, 0.27% of its 29,126 verified requests"]},{"agent":"PerplexityBot","operator":"Perplexity","purpose":"indexing","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-13T02:14:50.805Z","last_seen":"2026-11-15T23:08:39.037Z"},"identity":{"verified_requests":27844,"forged_claims":97,"forgery_share":0.003,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Perplexity publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":238,"disallowed_fetches":54,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-22T08:38:56.498Z","path":"/r/hsem60u0pb","net":"18.97.9.0/24"},{"at":"2026-10-22T15:59:08.322Z","path":"/r/hsem60u0pb","net":"18.97.9.0/24"},{"at":"2026-10-22T15:59:20.457Z","path":"/r/uazt7utoub","net":"18.97.9.0/24"},{"at":"2026-10-24T07:38:31.838Z","path":"/r/hsem60u0pb","net":"18.97.9.0/24"},{"at":"2026-10-24T07:38:34.319Z","path":"/r/phuaa0zazw","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:44.938Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:45.149Z","path":"/r/phuaa0zazw","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:45.922Z","path":"/r/phuaa0zazw","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:46.331Z","path":"/r/phuaa0zazw","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:47.936Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:02.461Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:04.413Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:07.938Z","path":"/r/up0rftpp5z","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:08.999Z","path":"/r/1fb80ss94n","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:09.428Z","path":"/r/up0rftpp5z","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:11.912Z","path":"/r/1fb80ss94n","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:18.485Z","path":"/r/up0rftpp5z","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:18.928Z","path":"/r/up0rftpp5z","net":"18.97.9.0/24"},{"at":"2026-10-27T06:50:46.518Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"},{"at":"2026-10-28T20:49:42.689Z","path":"/r/khlmjzqasm","net":"18.97.9.0/24"}],"counted_since":"2026-10-07T04:51:16.683Z","reason":null,"robots_txt_own_fetches":238,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":119,"peak_at":"2026-10-11T23:32","published_limit":null,"measured_since":"2026-10-07T04:51:16.683Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T04:51:16.683Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":228,"verified":131,"spoofed":97},"daily_verified":{"2026-11-02":878,"2026-11-03":853,"2026-11-04":814,"2026-11-05":613,"2026-11-06":687,"2026-11-07":972,"2026-11-08":1521,"2026-11-09":814,"2026-11-10":722,"2026-11-11":604,"2026-11-12":568,"2026-11-13":692,"2026-11-14":880,"2026-11-15":1250},"portfolio":{"agent":"PerplexityBot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com","emmanuelorta.com"],"verified_fetches_judged":129,"verified_fetches_not_judged":163,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["54 recorded fetches of disallowed paths, 0.19% of its 27,844 verified requests"]},{"agent":"YandexBot","operator":"Yandex","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-19T13:53:12.960Z","last_seen":"2026-11-15T23:48:07.977Z"},"identity":{"verified_requests":15496,"forged_claims":20,"forgery_share":0.001,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"reverse_dns","reading":"verified_requests passed forward-confirmed reverse DNS to Yandex's own domains (yandex.ru, yandex.net, yandex.com); forged_claims used this agent's name from addresses that did not (impersonators, not the agent). Reverse-DNS checks began 10 October 2026; earlier hits stay unverifiable"},"robots_txt":{"verdict":"respected","robots_txt_fetches":688,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-10T12:54:23.939Z","reason":null,"robots_txt_own_fetches":688,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":26,"peak_at":"2026-11-06T03:13","published_limit":null,"measured_since":"2026-10-10T12:54:23.939Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-10T12:54:23.939Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-02":884,"2026-11-03":149,"2026-11-04":147,"2026-11-05":173,"2026-11-06":749,"2026-11-07":172,"2026-11-08":130,"2026-11-09":765,"2026-11-10":1139,"2026-11-11":982,"2026-11-12":959,"2026-11-13":697,"2026-11-14":1131,"2026-11-15":988},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"AhrefsBot","operator":"Ahrefs","purpose":"tool","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-14T02:11:27.871Z","last_seen":"2026-11-15T23:02:40.266Z"},"identity":{"verified_requests":12464,"forged_claims":3,"forgery_share":0,"signed_requests":5608,"failed_signatures":2356,"signed_share":0.45,"strongest_proof":"signature","reading":"verified_requests came from the ranges Ahrefs publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":227,"disallowed_fetches":12,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-26T07:48:14.796Z","path":"/r/1fb80ss94n","net":"51.68.247.0/24"},{"at":"2026-10-26T08:03:14.376Z","path":"/r/1fb80ss94n","net":"51.75.236.0/24"},{"at":"2026-10-26T08:44:02.288Z","path":"/r/1fb80ss94n","net":"5.39.109.0/24"},{"at":"2026-11-03T14:28:07.385Z","path":"/r/6gaxqt2gao","net":"51.81.110.0/24"},{"at":"2026-11-09T05:19:38.753Z","path":"/r/j2mipohpuh","net":"37.59.204.0/24"},{"at":"2026-11-09T05:27:58.358Z","path":"/r/j2mipohpuh","net":"5.39.109.0/24"},{"at":"2026-11-09T05:36:18.899Z","path":"/r/j2mipohpuh","net":"51.75.236.0/24"},{"at":"2026-11-11T17:56:06.713Z","path":"/r/uazt7utoub","net":"51.222.95.0/24"},{"at":"2026-11-11T17:58:15.475Z","path":"/r/uazt7utoub","net":"15.235.27.0/24"}],"counted_since":"2026-10-07T23:53:27.541Z","reason":null,"robots_txt_own_fetches":227,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":64,"peak_at":"2026-11-01T16:47","published_limit":null,"measured_since":"2026-10-07T23:53:27.541Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":933,"share_of_verified_since_measured":0.075,"measured_since":"2026-10-07T23:53:27.541Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-02":395,"2026-11-03":174,"2026-11-04":148,"2026-11-05":619,"2026-11-06":223,"2026-11-07":175,"2026-11-08":130,"2026-11-09":359,"2026-11-10":221,"2026-11-11":373,"2026-11-12":104,"2026-11-13":700,"2026-11-14":350,"2026-11-15":420},"portfolio":{"agent":"AhrefsBot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":1537,"verified_fetches_not_judged":896,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["12 recorded fetches of disallowed paths, 0.1% of its 12,464 verified requests"]},{"agent":"Applebot","operator":"Apple","purpose":"indexing","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-13T12:32:42.174Z","last_seen":"2026-11-15T23:43:42.459Z"},"identity":{"verified_requests":12130,"forged_claims":84,"forgery_share":0.007,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Apple publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":96,"disallowed_fetches":40,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-23T13:32:15.215Z","path":"/r/8dsc0mu5ks","net":"17.166.233.0/24"},{"at":"2026-10-23T13:39:38.180Z","path":"/r/8dsc0mu5ks","net":"17.166.232.0/24"},{"at":"2026-10-24T07:28:50.585Z","path":"/r/8dsc0mu5ks","net":"17.166.235.0/24"},{"at":"2026-10-24T07:32:58.177Z","path":"/r/8dsc0mu5ks","net":"17.166.236.0/24"},{"at":"2026-10-24T16:13:56.254Z","path":"/r/mkx08rapce","net":"17.166.232.0/24"},{"at":"2026-10-24T16:17:40.243Z","path":"/r/mkx08rapce","net":"17.166.235.0/24"},{"at":"2026-10-26T15:31:58.651Z","path":"/r/8dsc0mu5ks","net":"17.166.232.0/24"},{"at":"2026-10-26T15:34:20.954Z","path":"/r/8dsc0mu5ks","net":"17.166.232.0/24"},{"at":"2026-10-26T16:01:58.942Z","path":"/r/8dsc0mu5ks","net":"17.166.25.0/24"},{"at":"2026-10-26T16:06:50.241Z","path":"/r/8dsc0mu5ks","net":"17.166.21.0/24"},{"at":"2026-10-27T20:22:26.222Z","path":"/r/mkx08rapce","net":"17.166.235.0/24"},{"at":"2026-10-27T20:37:56.625Z","path":"/r/mkx08rapce","net":"17.166.233.0/24"},{"at":"2026-10-28T01:54:46.237Z","path":"/r/mkx08rapce","net":"17.166.21.0/24"},{"at":"2026-10-28T01:55:55.986Z","path":"/r/mkx08rapce","net":"17.166.22.0/24"},{"at":"2026-10-28T12:27:38.731Z","path":"/r/mkx08rapce","net":"17.166.153.0/24"},{"at":"2026-10-28T12:33:52.496Z","path":"/r/mkx08rapce","net":"17.166.152.0/24"},{"at":"2026-10-30T18:31:10.592Z","path":"/r/khlmjzqasm","net":"17.166.153.0/24"},{"at":"2026-10-30T18:31:54.377Z","path":"/r/khlmjzqasm","net":"17.166.154.0/24"},{"at":"2026-11-02T05:43:52.587Z","path":"/r/aqxn4lxkaw","net":"17.166.151.0/24"},{"at":"2026-11-02T05:51:03.321Z","path":"/r/aqxn4lxkaw","net":"17.166.154.0/24"}],"counted_since":"2026-10-07T03:15:19.709Z","reason":null,"robots_txt_own_fetches":96,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":8,"peak_at":"2026-10-21T17:47","published_limit":null,"measured_since":"2026-10-07T03:15:19.709Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T03:15:19.709Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":749,"verified":687,"spoofed":62},"daily_verified":{"2026-11-02":48,"2026-11-03":53,"2026-11-04":51,"2026-11-05":114,"2026-11-06":99,"2026-11-07":124,"2026-11-08":779,"2026-11-09":818,"2026-11-10":460,"2026-11-11":826,"2026-11-12":695,"2026-11-13":630,"2026-11-14":382,"2026-11-15":306},"portfolio":{"agent":"Applebot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":1153,"verified_fetches_not_judged":409,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["40 recorded fetches of disallowed paths, 0.33% of its 12,130 verified requests"]},{"agent":"Googlebot","operator":"Google","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-13T02:14:51.611Z","last_seen":"2026-11-15T23:44:32.869Z"},"identity":{"verified_requests":11413,"forged_claims":241,"forgery_share":0.021,"signed_requests":0,"failed_signatures":392,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":204,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T03:18:09.861Z","reason":null,"robots_txt_own_fetches":204,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":13,"peak_at":"2026-10-19T02:10","published_limit":null,"measured_since":"2026-10-07T03:18:09.861Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":1502,"share_of_verified_since_measured":0.132,"measured_since":"2026-10-07T03:18:09.861Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":1562,"verified":1088,"spoofed":474},"daily_verified":{"2026-11-02":215,"2026-11-03":264,"2026-11-04":83,"2026-11-05":117,"2026-11-06":204,"2026-11-07":134,"2026-11-08":98,"2026-11-09":90,"2026-11-10":133,"2026-11-11":265,"2026-11-12":397,"2026-11-13":326,"2026-11-14":168,"2026-11-15":124},"portfolio":{"agent":"Googlebot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com","emmanuelorta.com"],"verified_fetches_judged":1721,"verified_fetches_not_judged":1477,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"ClaudeBot","operator":"Anthropic","purpose":"training","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-13T02:14:49.992Z","last_seen":"2026-11-15T15:32:59.324Z"},"identity":{"verified_requests":10932,"forged_claims":117,"forgery_share":0.011,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Anthropic publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":292,"disallowed_fetches":3,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-16T09:27:57.482Z","path":"/r/uazt7utoub","net":"216.73.216.0/24"},{"at":"2026-10-16T09:31:12.061Z","path":"/r/uazt7utoub","net":"216.73.216.0/24"},{"at":"2026-10-16T09:38:15.950Z","path":"/r/uazt7utoub","net":"216.73.216.0/24"}],"counted_since":"2026-10-07T03:48:54.788Z","reason":null,"robots_txt_own_fetches":292,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":61,"peak_at":"2026-10-19T01:34","published_limit":null,"measured_since":"2026-10-07T03:48:54.788Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T03:48:54.788Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":293,"verified":173,"spoofed":120},"daily_verified":{"2026-11-02":17,"2026-11-03":1213,"2026-11-04":84,"2026-11-05":28,"2026-11-06":15,"2026-11-07":2,"2026-11-08":2,"2026-11-09":10,"2026-11-10":182,"2026-11-11":5,"2026-11-12":65,"2026-11-13":44,"2026-11-14":24,"2026-11-15":12},"portfolio":{"agent":"ClaudeBot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":898,"verified_fetches_not_judged":970,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["3 recorded fetches of disallowed paths, 0.03% of its 10,932 verified requests"]},{"agent":"Bingbot","operator":"Microsoft","purpose":"indexing","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-14T06:25:41.331Z","last_seen":"2026-11-15T23:49:16.910Z"},"identity":{"verified_requests":9220,"forged_claims":50,"forgery_share":0.005,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Microsoft publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":120,"disallowed_fetches":1,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-27T19:17:43.386Z","path":"/r/uazt7utoub","net":"40.77.167.0/24"}],"counted_since":"2026-10-07T03:20:08.676Z","reason":null,"robots_txt_own_fetches":120,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":8,"peak_at":"2026-11-01T02:34","published_limit":null,"measured_since":"2026-10-07T03:20:08.676Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":792,"share_of_verified_since_measured":0.086,"measured_since":"2026-10-07T03:20:08.676Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":1059,"verified":955,"spoofed":104},"daily_verified":{"2026-11-02":322,"2026-11-03":270,"2026-11-04":212,"2026-11-05":199,"2026-11-06":124,"2026-11-07":221,"2026-11-08":161,"2026-11-09":114,"2026-11-10":105,"2026-11-11":224,"2026-11-12":257,"2026-11-13":237,"2026-11-14":324,"2026-11-15":271},"portfolio":{"agent":"Bingbot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com","emmanuelorta.com"],"verified_fetches_judged":1138,"verified_fetches_not_judged":600,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"D","grade_reasons":["1 recorded fetch of disallowed paths, 0.01% of its 9,220 verified requests"]},{"agent":"GPTBot","operator":"OpenAI","purpose":"training","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-12T06:20:42.861Z","last_seen":"2026-11-15T22:55:44.711Z"},"identity":{"verified_requests":8350,"forged_claims":173,"forgery_share":0.02,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges OpenAI publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"robots_txt_not_fetched","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T18:09:36.470Z","reason":null,"robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":120,"peak_at":"2026-11-02T11:28","published_limit":null,"measured_since":"2026-10-07T18:09:36.470Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T18:09:36.470Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":566,"verified":66,"spoofed":500},"daily_verified":{"2026-11-02":2130,"2026-11-03":2,"2026-11-04":6,"2026-11-05":2,"2026-11-06":24,"2026-11-07":2,"2026-11-08":57,"2026-11-09":2,"2026-11-10":2,"2026-11-11":410,"2026-11-12":1,"2026-11-13":28,"2026-11-14":76,"2026-11-15":463},"portfolio":{"agent":"GPTBot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com","emmanuelorta.com"],"verified_fetches_judged":309,"verified_fetches_not_judged":157,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"B","grade_reasons":["no disallowed path fetched, but it never fetched robots.txt under its own name here, so it cannot be credited with reading the rules"]},{"agent":"OAI-SearchBot","operator":"OpenAI","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-13T02:14:49.166Z","last_seen":"2026-11-15T23:45:42.426Z"},"identity":{"verified_requests":7562,"forged_claims":107,"forgery_share":0.014,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges OpenAI publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":316,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T03:30:28.967Z","reason":null,"robots_txt_own_fetches":316,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":119,"peak_at":"2026-10-29T12:58","published_limit":null,"measured_since":"2026-10-07T03:30:28.967Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T03:30:28.967Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":929,"verified":847,"spoofed":82},"daily_verified":{"2026-11-02":420,"2026-11-03":308,"2026-11-04":208,"2026-11-05":160,"2026-11-06":131,"2026-11-07":392,"2026-11-08":241,"2026-11-09":338,"2026-11-10":203,"2026-11-11":182,"2026-11-12":131,"2026-11-13":185,"2026-11-14":283,"2026-11-15":213},"portfolio":{"agent":"OAI-SearchBot","sites":["supremefencingdenver.com","supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com","emmanuelorta.com"],"verified_fetches_judged":927,"verified_fetches_not_judged":467,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"ChatGPT-User","operator":"OpenAI","purpose":"live fetch","user_triggered":true,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-13T22:52:23.214Z","last_seen":"2026-11-15T21:57:18.721Z"},"identity":{"verified_requests":1631,"forged_claims":88,"forgery_share":0.051,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges OpenAI publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"not_applicable","robots_txt_fetches":1,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T03:26:16.042Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":1,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":13,"peak_at":"2026-10-15T19:36","published_limit":null,"measured_since":"2026-10-07T03:26:16.042Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T03:26:16.042Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":150,"verified":94,"spoofed":56},"daily_verified":{"2026-11-02":21,"2026-11-03":29,"2026-11-04":25,"2026-11-05":28,"2026-11-06":30,"2026-11-07":33,"2026-11-08":48,"2026-11-09":60,"2026-11-10":72,"2026-11-11":27,"2026-11-12":28,"2026-11-13":30,"2026-11-14":25,"2026-11-15":20},"portfolio":{"agent":"ChatGPT-User","sites":["supremefencingdenver.com","supremearvadafencing.com","arvadatreeservicellc.com","terrariumstation.com","treeservicedenverllc.com"],"verified_fetches_judged":87,"verified_fetches_not_judged":55,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"A","grade_reasons":["fetches pages a person asked for; its operator states robots.txt does not govern those fetches, and every counted request came from its published addresses"]},{"agent":"SeznamBot","operator":"Seznam","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-10-10T13:04:32.776Z","last_seen":"2026-11-15T20:44:03.474Z"},"identity":{"verified_requests":1037,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"reverse_dns","reading":"verified_requests passed forward-confirmed reverse DNS to Seznam's own domains (seznam.cz); forged_claims used this agent's name from addresses that did not (impersonators, not the agent). Reverse-DNS checks began 10 October 2026; earlier hits stay unverifiable"},"robots_txt":{"verdict":"respected","robots_txt_fetches":25,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-10T13:04:32.776Z","reason":null,"robots_txt_own_fetches":25,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":3,"peak_at":"2026-10-10T22:12","published_limit":null,"measured_since":"2026-10-10T13:04:32.776Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":865,"share_of_verified_since_measured":0.834,"measured_since":"2026-10-10T13:04:32.776Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-01":22,"2026-11-02":52,"2026-11-03":2,"2026-11-04":44,"2026-11-05":3,"2026-11-06":43,"2026-11-07":9,"2026-11-08":2,"2026-11-09":35,"2026-11-10":43,"2026-11-11":15,"2026-11-13":14,"2026-11-14":13,"2026-11-15":59},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"Claude-User","operator":"Anthropic","purpose":"live fetch","user_triggered":true,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-12T22:57:31.977Z","last_seen":"2026-11-15T23:49:35.337Z"},"identity":{"verified_requests":853,"forged_claims":55,"forgery_share":0.061,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Anthropic publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"not_applicable","robots_txt_fetches":17,"disallowed_fetches":1,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-17T04:44:31.376Z","path":"/r/zqdnb5b58f","net":"160.79.106.0/24"}],"counted_since":"2026-10-07T11:03:04.563Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":17,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":7,"peak_at":"2026-11-15T13:58","published_limit":null,"measured_since":"2026-10-07T11:03:04.563Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T11:03:04.563Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":62,"verified":6,"spoofed":56},"daily_verified":{"2026-10-25":1,"2026-11-13":143,"2026-11-14":284,"2026-11-15":205},"portfolio":{"agent":"Claude-User","sites":["supremefencingdenver.com","treeservicedenverllc.com"],"verified_fetches_judged":17,"verified_fetches_not_judged":0,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"A","grade_reasons":["fetches pages a person asked for; its operator states robots.txt does not govern those fetches, and every counted request came from its published addresses"]},{"agent":"Baiduspider","operator":"Baidu","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-13T21:31:01.445Z","last_seen":"2026-11-15T23:24:23.279Z"},"identity":{"verified_requests":630,"forged_claims":37,"forgery_share":0.055,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"reverse_dns","reading":"verified_requests passed forward-confirmed reverse DNS to Baidu's own domains (baidu.com, baidu.jp); forged_claims used this agent's name from addresses that did not (impersonators, not the agent). Reverse-DNS checks began 10 October 2026; earlier hits stay unverifiable"},"robots_txt":{"verdict":"robots_txt_not_fetched","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-10T13:14:25.713Z","reason":null,"robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":5,"peak_at":"2026-10-26T00:08","published_limit":null,"measured_since":"2026-10-10T13:14:25.713Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-10T13:14:25.713Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-02":21,"2026-11-03":1,"2026-11-04":12,"2026-11-05":19,"2026-11-06":18,"2026-11-07":10,"2026-11-08":16,"2026-11-09":5,"2026-11-10":17,"2026-11-11":54,"2026-11-12":9,"2026-11-13":29,"2026-11-14":14,"2026-11-15":23},"grade":"B","grade_reasons":["no disallowed path fetched, but it never fetched robots.txt under its own name here, so it cannot be credited with reading the rules"]},{"agent":"CCBot","operator":"Common Crawl","purpose":"training","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-15T17:58:20.715Z","last_seen":"2026-11-14T09:28:58.099Z"},"identity":{"verified_requests":567,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Common Crawl publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":1,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-11-09T16:57:56.236Z","reason":null,"robots_txt_own_fetches":1,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":7,"peak_at":"2026-11-09T18:40","published_limit":null,"measured_since":"2026-11-09T16:57:56.236Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-11-09T16:57:56.236Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-09":187,"2026-11-10":7,"2026-11-11":183,"2026-11-14":187},"portfolio":{"agent":"CCBot","sites":["supremefencinglakewood.com","supremearvadafencing.com","arvadatreeservicellc.com","treeservicedenverllc.com"],"verified_fetches_judged":82,"verified_fetches_not_judged":0,"disallowed_fetches":0,"disallowed_fetch_records":[]},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"Google-CWS","operator":"Google","purpose":"store listing fetch","user_triggered":true,"conduct":"no_violations_observed","window":{"first_seen":"2026-09-18T16:25:39.959Z","last_seen":"2026-11-15T19:03:09.558Z"},"identity":{"verified_requests":442,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"not_applicable","robots_txt_fetches":206,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T11:31:33.296Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":206,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":3,"peak_at":"2026-10-07T20:31","published_limit":null,"measured_since":"2026-10-07T11:31:33.296Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T11:31:33.296Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-01":15,"2026-11-02":19,"2026-11-03":8,"2026-11-04":9,"2026-11-05":3,"2026-11-06":8,"2026-11-07":14,"2026-11-08":18,"2026-11-09":12,"2026-11-10":5,"2026-11-11":2,"2026-11-13":4,"2026-11-14":11,"2026-11-15":8},"grade":"A","grade_reasons":["fetches pages a person asked for; its operator states robots.txt does not govern those fetches, and every counted request came from its published addresses"]},{"agent":"Google-InspectionTool","operator":"Google","purpose":null,"user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-10-27T11:15:26.957Z","last_seen":"2026-11-15T14:09:07.989Z"},"identity":{"verified_requests":291,"forged_claims":5,"forgery_share":0.017,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":206,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-29T14:57:13.241Z","reason":"Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 206 times here","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":206,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":7,"peak_at":"2026-10-29T21:37","published_limit":null,"measured_since":"2026-10-29T14:57:13.241Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-29T14:57:13.241Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-31":10,"2026-11-01":11,"2026-11-02":4,"2026-11-03":5,"2026-11-04":4,"2026-11-06":8,"2026-11-07":29,"2026-11-08":10,"2026-11-09":6,"2026-11-10":1,"2026-11-11":2,"2026-11-13":9,"2026-11-14":6,"2026-11-15":5},"grade":"A","grade_reasons":["Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 206 times here, and it fetched no path robots.txt disallows"]},{"agent":"Google-Agent","operator":"Google","purpose":"live fetch","user_triggered":true,"conduct":"no_violations_observed","window":{"first_seen":"2026-10-31T20:31:56.218Z","last_seen":"2026-11-13T20:09:07.985Z"},"identity":{"verified_requests":280,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":167,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"not_applicable","robots_txt_fetches":2,"disallowed_fetches":3,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-11-09T19:22:08.454Z","path":"/r/by2qvflvb4","net":"136.121.28.0/24"},{"at":"2026-11-09T19:22:38.694Z","path":"/r/by2qvflvb4","net":"136.121.28.0/24"},{"at":"2026-11-09T19:23:26.441Z","path":"/r/by2qvflvb4","net":"136.121.28.0/24"}],"counted_since":"2026-10-31T20:31:56.218Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":2,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":17,"peak_at":"2026-11-09T19:13","published_limit":null,"measured_since":"2026-10-31T20:31:56.218Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":14,"share_of_verified_since_measured":0.05,"measured_since":"2026-10-31T20:31:56.218Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-31":4,"2026-11-01":1,"2026-11-09":120,"2026-11-13":42},"grade":"A","grade_reasons":["fetches pages a person asked for; its operator states robots.txt does not govern those fetches, and every counted request came from its published addresses"]},{"agent":"DuckDuckBot","operator":"DuckDuckGo","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-12T20:29:50.246Z","last_seen":"2026-11-15T13:23:40.817Z"},"identity":{"verified_requests":168,"forged_claims":0,"forgery_share":0,"signed_requests":29,"failed_signatures":75,"signed_share":0.173,"strongest_proof":"signature","reading":"verified_requests came from the ranges DuckDuckGo publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"robots_txt_not_fetched","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-10T21:21:35.448Z","reason":null,"robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":2,"peak_at":"2026-10-10T21:21","published_limit":null,"measured_since":"2026-10-10T21:21:35.448Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":2,"share_of_verified_since_measured":0.012,"measured_since":"2026-10-10T21:21:35.448Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-02":7,"2026-11-03":6,"2026-11-04":4,"2026-11-05":6,"2026-11-06":6,"2026-11-07":4,"2026-11-08":5,"2026-11-09":3,"2026-11-10":3,"2026-11-11":2,"2026-11-12":7,"2026-11-13":4,"2026-11-14":8,"2026-11-15":4},"grade":"B","grade_reasons":["no disallowed path fetched, but it never fetched robots.txt under its own name here, so it cannot be credited with reading the rules"]},{"agent":"GoogleOther","operator":"Google","purpose":"training","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-10-28T07:22:36.856Z","last_seen":"2026-11-15T22:03:37.650Z"},"identity":{"verified_requests":132,"forged_claims":57,"forgery_share":0.302,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":206,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-28T07:22:36.856Z","reason":"Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 206 times here","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":206,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":5,"peak_at":"2026-11-15T19:39","published_limit":null,"measured_since":"2026-10-28T07:22:36.856Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-28T07:22:36.856Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-02":4,"2026-11-03":2,"2026-11-04":9,"2026-11-05":1,"2026-11-06":10,"2026-11-07":11,"2026-11-08":2,"2026-11-09":24,"2026-11-10":1,"2026-11-11":3,"2026-11-12":4,"2026-11-13":10,"2026-11-14":4,"2026-11-15":31},"grade":"A","grade_reasons":["Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 206 times here, and it fetched no path robots.txt disallows"]},{"agent":"Perplexity-User","operator":"Perplexity","purpose":"live fetch","user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-08-14T01:04:08.513Z","last_seen":"2026-11-09T10:13:29.728Z"},"identity":{"verified_requests":9,"forged_claims":52,"forgery_share":0.852,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Perplexity publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":1,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-11-07T19:14:23.425Z","path":"/r/a7qr2s89eu.md","net":"18.97.43.0/24"}],"counted_since":"2026-11-07T19:14:23.425Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":1,"peak_at":"2026-11-07T19:14","published_limit":null,"measured_since":"2026-11-07T19:14:23.425Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-11-07T19:14:23.425Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":41,"verified":0,"spoofed":41},"daily_verified":{"2026-11-07":1},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-NotebookLM","operator":"Google","purpose":null,"user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-10-09T19:50:34.360Z","last_seen":"2026-11-14T11:56:22.335Z"},"identity":{"verified_requests":8,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":206,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-09T19:50:34.360Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":206,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":4,"peak_at":"2026-10-09T19:50","published_limit":null,"measured_since":"2026-10-09T19:50:34.360Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-09T19:50:34.360Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-14":1},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-Read-Aloud","operator":null,"purpose":null,"user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-11-09T02:47:57.686Z","last_seen":"2026-11-09T02:48:00.569Z"},"identity":{"verified_requests":7,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":206,"disallowed_fetches":2,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-11-09T02:47:57.686Z","path":"/r/j2mipohpuh","net":"66.249.83.0/24"},{"at":"2026-11-09T02:47:58.799Z","path":"/r/j2mipohpuh","net":"66.249.83.0/24"}],"counted_since":"2026-11-09T02:47:57.686Z","reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":206,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":3,"peak_at":"2026-11-09T02:47","published_limit":null,"measured_since":"2026-11-09T02:47:57.686Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-11-09T02:47:57.686Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-11-09":4},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Yeti","operator":"Naver","purpose":"indexing","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-20T06:52:03.909Z","last_seen":"2026-11-10T11:49:02.423Z"},"identity":{"verified_requests":6,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"reverse_dns","reading":"verified_requests passed forward-confirmed reverse DNS to Naver's own domains (naver.com); forged_claims used this agent's name from addresses that did not (impersonators, not the agent). Reverse-DNS checks began 10 October 2026; earlier hits stay unverifiable"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":3,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-20T06:52:03.909Z","reason":null,"robots_txt_own_fetches":3,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":1,"peak_at":"2026-10-20T06:52","published_limit":null,"measured_since":"2026-10-20T06:52:03.909Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-20T06:52:03.909Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-20":1,"2026-10-29":2,"2026-11-04":2,"2026-11-10":1},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"AdsBot-Google","operator":"Google","purpose":"tool","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-27T09:03:19.135Z","last_seen":"2026-11-13T13:13:52.654Z"},"identity":{"verified_requests":5,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":206,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-27T09:03:19.135Z","reason":"its operator states it ignores the * group and obeys only a group that names it; this robots.txt names it in no group, so no rule here binds it and disallowed fetches are reported, not judged","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":206,"reading":"this agent its operator states it ignores the * group and obeys only a group that names it; this robots.txt names it in no group, so no rule here binds it and disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":2,"peak_at":"2026-10-27T09:03","published_limit":null,"measured_since":"2026-10-27T09:03:19.135Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-27T09:03:19.135Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-27":2,"2026-10-30":2,"2026-11-13":1},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-Safety","operator":null,"purpose":null,"user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-29T05:16:06.085Z","last_seen":"2026-11-15T18:57:35.722Z"},"identity":{"verified_requests":4,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":206,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-29T05:16:06.085Z","reason":"its operator states it ignores robots.txt rules (a safety crawler), so disallowed fetches are reported, not judged","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":206,"reading":"this agent its operator states it ignores robots.txt rules (a safety crawler), so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":2,"peak_at":"2026-11-15T18:49","published_limit":null,"measured_since":"2026-10-29T05:16:06.085Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-29T05:16:06.085Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-29":1,"2026-11-15":3},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-Site-Verification","operator":"Google","purpose":"ownership check","user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-10-03T21:54:40.584Z","last_seen":"2026-10-03T21:54:40.980Z"},"identity":{"verified_requests":2,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"signed_share":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":206,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":206,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Claude-SearchBot","operator":"Anthropic","purpose":"indexing","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-08-16T01:38:23.331Z","last_seen":"2026-11-09T10:13:29.211Z"},"identity":{"verified_requests":0,"forged_claims":57,"forgery_share":1,"signed_requests":0,"failed_signatures":0,"signed_share":null,"strongest_proof":"address","reading":"verified_requests came from the ranges Anthropic publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":null,"robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":70,"verified":0,"spoofed":70},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"MistralAI-User","operator":"Mistral","purpose":"live fetch","user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-08-16T01:38:24.858Z","last_seen":"2026-09-12T16:50:37.743Z"},"identity":{"verified_requests":0,"forged_claims":0,"forgery_share":null,"signed_requests":0,"failed_signatures":0,"signed_share":null,"strongest_proof":"address","reading":"verified_requests came from the ranges Mistral publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":0,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"FeedFetcher-Google","operator":"Google","purpose":"feed fetch","user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-09-28T08:19:13.036Z","last_seen":"2026-09-28T08:19:13.036Z"},"identity":{"verified_requests":0,"forged_claims":1,"forgery_share":1,"signed_requests":0,"failed_signatures":0,"signed_share":null,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":206,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":"fetches pages a person asked for; its operator states robots.txt does not govern those fetches","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":206,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-CloudVertexBot","operator":"Google","purpose":"tool","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-30T15:10:18.766Z","last_seen":"2026-10-30T15:10:25.755Z"},"identity":{"verified_requests":0,"forged_claims":3,"forgery_share":1,"signed_requests":0,"failed_signatures":0,"signed_share":null,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":206,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":"Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 206 times here","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":206,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Storebot-Google","operator":"Google","purpose":"indexing","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-31T06:04:36.487Z","last_seen":"2026-11-14T05:57:04.805Z"},"identity":{"verified_requests":0,"forged_claims":8,"forgery_share":1,"signed_requests":0,"failed_signatures":16,"signed_share":null,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":206,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reason":"Google states its crawlers may share one cached robots.txt response; verified Google crawlers fetched robots.txt 206 times here","robots_txt_own_fetches":0,"robots_txt_sibling_fetches":0,"robots_txt_shared_from_google":206,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]}],"not_assessed":[{"agent":"Meta-ExternalAgent","claimed_hits":71993,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"Bytespider","claimed_hits":15983,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"Google-Extended","claimed_hits":49,"why":"a robots.txt control token, not a crawler: its operator sends no requests under this name, so nothing here can be judged as its conduct"},{"agent":"Diffbot","claimed_hits":2,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"SemrushBot","claimed_hits":25202,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"DataForSeoBot","claimed_hits":13652,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"anthropic-ai","claimed_hits":4,"why":"a robots.txt control token, not a crawler: its operator sends no requests under this name, so nothing here can be judged as its conduct"},{"agent":"Applebot-Extended","claimed_hits":1,"why":"a robots.txt control token, not a crawler: its operator sends no requests under this name, so nothing here can be judged as its conduct"},{"agent":"Timpibot","claimed_hits":4,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"YouBot","claimed_hits":2503,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"Scrapy","claimed_hits":18,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"cohere-ai","claimed_hits":51,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"FacebookBot","claimed_hits":1,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"}],"census":"https://crawlcheck.io/api/telemetry","visitor_checks":"https://crawlcheck.io/api/v1/visitor/stats","code":{"version_id":"2b3a6edd-fa27-4966-b48e-b9f31ac87d06","script_sha16":"b56cf27573871964","state":"attested","deployed":{"at":"2026-11-15T20:58:54.055Z","commit":"c317a5a5ac4e","run":"43760948737"},"build_log_entry":2537,"score_version":43,"scoring_policy_sha256":"30c364435ab51c609be32fe5d6851aa48bd0f99ac7fb40a440b70dbb1ffb7d02","decision_rules_sha256":"18dd5636e0896692ab518050cb401744edea2e6f1086dff94fead314959bdc3e","check":"https://crawlcheck.io/api/v1/builds/2b3a6edd-fa27-4966-b48e-b9f31ac87d06"},"sha256":"f276225d4353672a4cff3de5e74c228d05b9df9f5c70a2c8691884744f405bb9","signature":{"v":1,"alg":"Ed25519","kid":"1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps","over":"sha256","message":"crawlcheck-data-v1\nf276225d4353672a4cff3de5e74c228d05b9df9f5c70a2c8691884744f405bb9","sig":"eUlzzc3adjiFB_qdLWFhxqif5ggjBA6icHONxfJMs_tYHmjZhZvjl6CKpSqMpSbz0xEmNW4hty5yhZt6fUYwAA","signed_at":"2026-11-15T23:51:15.355Z","key":{"kid":"1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps","jwk":{"kty":"OKP","crv":"Ed25519","x":"5TmKCOx0TTHepq7HwyXBHvGQpVCSBd2lV8cLzYVqFxQ"},"published_in":"https://crawlcheck.io/.well-known/http-message-signatures-directory"},"canonical":"sha256 over canonical JSON of every field except sha256 and signature"}}