{"kind":"crawlcheck-conduct","v":1,"generated_at":"2026-10-26T00:04:56.866Z","population":"verifiable agents observed on crawlcheck.io","reading":"A census of one property, not a sample of the web: what each agent did HERE, verified by source address against the operator's own published ranges. conduct is no_violations_observed when at least 50 verified requests show no fetch of a path robots.txt disallows; violations_observed when a disallowed path was fetched; insufficient_data below 50. The busiest minute is reported, not judged: this site publishes no crawl rate. site_checks_14d are counts from Visitor Resolve calls other sites made about this agent.","thresholds":{"min_verified_requests":50,"published_rate_limit":null,"disallowed_rules":["/r/","/backup/","/admin-old/"]},"verdicts":{"respected":"robots.txt fetched, and no disallowed path fetched","robots_txt_not_fetched":"no disallowed path fetched, but robots.txt was never fetched under this agent's name here, so it cannot be credited with reading the rules","disallowed_paths_fetched":"a verified fetch of a disallowed path, kept with its time, path and network in disallowed_fetch_records","disallowed_fetch_unrecorded":"disallowed fetches were counted before 2026-10-15, when no per-fetch record was kept; reported, not judged (conduct: not_judged)"},"grades":{"A":"read robots.txt and obeyed it","B":"obeyed, but never fetched robots.txt under its own name here","D":"recorded fetches of disallowed paths, under 0.5% of its verified requests","F":"a fetch of a decoy path named only in robots.txt, or disallowed fetches at 0.5% or more","none":"not enough verified requests, or not judgeable"},"agents":[{"agent":"PerplexityBot","operator":"Perplexity","purpose":"indexing","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-13T02:14:50.805Z","last_seen":"2026-10-25T23:44:38.018Z"},"identity":{"verified_requests":10045,"forged_claims":80,"forgery_share":0.008,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Perplexity publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":119,"disallowed_fetches":11,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-22T08:38:56.498Z","path":"/r/hsem60u0pb","net":"18.97.9.0/24"},{"at":"2026-10-22T15:59:08.322Z","path":"/r/hsem60u0pb","net":"18.97.9.0/24"},{"at":"2026-10-22T15:59:20.457Z","path":"/r/uazt7utoub","net":"18.97.9.0/24"},{"at":"2026-10-24T07:38:31.838Z","path":"/r/hsem60u0pb","net":"18.97.9.0/24"},{"at":"2026-10-24T07:38:34.319Z","path":"/r/phuaa0zazw","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:44.938Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:45.149Z","path":"/r/phuaa0zazw","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:45.922Z","path":"/r/phuaa0zazw","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:46.331Z","path":"/r/phuaa0zazw","net":"18.97.9.0/24"},{"at":"2026-10-25T14:17:47.936Z","path":"/r/cz9abl7ob3","net":"18.97.9.0/24"}],"counted_since":"2026-10-07T04:51:16.683Z","reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":119,"peak_at":"2026-10-11T23:32","published_limit":null,"measured_since":"2026-10-07T04:51:16.683Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T04:51:16.683Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":160,"verified":99,"spoofed":61},"daily_verified":{"2026-10-18":150,"2026-10-19":245,"2026-10-20":62,"2026-10-21":141,"2026-10-22":155,"2026-10-23":298,"2026-10-24":205,"2026-10-25":142},"grade":"D","grade_reasons":["11 recorded fetches of disallowed paths, 0.11% of its 10,045 verified requests"]},{"agent":"Amazonbot","operator":"Amazon","purpose":"indexing","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-13T20:30:33.503Z","last_seen":"2026-10-26T00:04:04.704Z"},"identity":{"verified_requests":8162,"forged_claims":46,"forgery_share":0.006,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Amazon publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":0,"disallowed_fetches":10,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-18T08:45:12.519Z","path":"/r/xufza280d1","net":"18.215.77.0/24"},{"at":"2026-10-18T18:02:03.808Z","path":"/r/xufza280d1","net":"52.205.113.0/24"},{"at":"2026-10-18T18:04:19.333Z","path":"/r/xufza280d1","net":"34.227.156.0/24"},{"at":"2026-10-18T19:28:54.132Z","path":"/r/xufza280d1","net":"54.163.169.0/24"},{"at":"2026-10-18T20:50:07.376Z","path":"/r/xufza280d1","net":"44.220.2.0/24"},{"at":"2026-10-19T23:29:49.080Z","path":"/r/xufza280d1","net":"23.23.137.0/24"},{"at":"2026-10-23T04:17:44.813Z","path":"/r/hsem60u0pb","net":"18.214.186.0/24"},{"at":"2026-10-24T07:16:41.220Z","path":"/r/hsem60u0pb","net":"184.73.68.0/24"},{"at":"2026-10-24T07:16:45.273Z","path":"/r/hsem60u0pb","net":"3.221.156.0/24"},{"at":"2026-10-24T13:42:45.118Z","path":"/r/hsem60u0pb","net":"18.209.137.0/24"}],"counted_since":"2026-10-18T06:50:57.435Z","reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":24,"peak_at":"2026-10-19T05:19","published_limit":null,"measured_since":"2026-10-18T06:50:57.435Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-18T06:50:57.435Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":2,"verified":1,"spoofed":1},"daily_verified":{"2026-10-18":1430,"2026-10-19":1304,"2026-10-20":1042,"2026-10-21":1354,"2026-10-22":1064,"2026-10-23":533,"2026-10-24":648,"2026-10-25":785,"2026-10-26":2},"grade":"D","grade_reasons":["10 recorded fetches of disallowed paths, 0.12% of its 8,162 verified requests"]},{"agent":"Googlebot","operator":"Google","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-13T02:14:51.611Z","last_seen":"2026-10-26T00:00:38.292Z"},"identity":{"verified_requests":7896,"forged_claims":139,"forgery_share":0.017,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":108,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T03:18:09.861Z","reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":13,"peak_at":"2026-10-19T02:10","published_limit":null,"measured_since":"2026-10-07T03:18:09.861Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":505,"share_of_verified_since_measured":0.064,"measured_since":"2026-10-07T03:18:09.861Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":1781,"verified":1135,"spoofed":646},"daily_verified":{"2026-10-18":80,"2026-10-19":155,"2026-10-20":98,"2026-10-21":152,"2026-10-22":152,"2026-10-23":136,"2026-10-24":106,"2026-10-25":129,"2026-10-26":1},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"ClaudeBot","operator":"Anthropic","purpose":"training","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-13T02:14:49.992Z","last_seen":"2026-10-25T23:40:40.346Z"},"identity":{"verified_requests":6951,"forged_claims":87,"forgery_share":0.012,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Anthropic publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":208,"disallowed_fetches":3,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-16T09:27:57.482Z","path":"/r/uazt7utoub","net":"216.73.216.0/24"},{"at":"2026-10-16T09:31:12.061Z","path":"/r/uazt7utoub","net":"216.73.216.0/24"},{"at":"2026-10-16T09:38:15.950Z","path":"/r/uazt7utoub","net":"216.73.216.0/24"}],"counted_since":"2026-10-07T03:48:54.788Z","reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":61,"peak_at":"2026-10-19T01:34","published_limit":null,"measured_since":"2026-10-07T03:48:54.788Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T03:48:54.788Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":241,"verified":176,"spoofed":65},"daily_verified":{"2026-10-18":11,"2026-10-19":1483,"2026-10-20":191,"2026-10-21":32,"2026-10-22":45,"2026-10-23":169,"2026-10-24":158,"2026-10-25":263},"grade":"D","grade_reasons":["3 recorded fetches of disallowed paths, 0.04% of its 6,951 verified requests"]},{"agent":"Applebot","operator":"Apple","purpose":"indexing","user_triggered":false,"conduct":"violations_observed","window":{"first_seen":"2026-08-13T12:32:42.174Z","last_seen":"2026-10-25T22:46:06.583Z"},"identity":{"verified_requests":6030,"forged_claims":84,"forgery_share":0.014,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Apple publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_paths_fetched","robots_txt_fetches":55,"disallowed_fetches":6,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-23T13:32:15.215Z","path":"/r/8dsc0mu5ks","net":"17.166.233.0/24"},{"at":"2026-10-23T13:39:38.180Z","path":"/r/8dsc0mu5ks","net":"17.166.232.0/24"},{"at":"2026-10-24T07:28:50.585Z","path":"/r/8dsc0mu5ks","net":"17.166.235.0/24"},{"at":"2026-10-24T07:32:58.177Z","path":"/r/8dsc0mu5ks","net":"17.166.236.0/24"},{"at":"2026-10-24T16:13:56.254Z","path":"/r/mkx08rapce","net":"17.166.232.0/24"},{"at":"2026-10-24T16:17:40.243Z","path":"/r/mkx08rapce","net":"17.166.235.0/24"}],"counted_since":"2026-10-07T03:15:19.709Z","reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":8,"peak_at":"2026-10-21T17:47","published_limit":null,"measured_since":"2026-10-07T03:15:19.709Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T03:15:19.709Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":565,"verified":538,"spoofed":27},"daily_verified":{"2026-10-18":121,"2026-10-19":204,"2026-10-20":205,"2026-10-21":127,"2026-10-22":243,"2026-10-23":103,"2026-10-24":125,"2026-10-25":114},"grade":"D","grade_reasons":["6 recorded fetches of disallowed paths, 0.1% of its 6,030 verified requests"]},{"agent":"AhrefsBot","operator":"Ahrefs","purpose":"tool","user_triggered":false,"conduct":"not_judged","window":{"first_seen":"2026-08-14T02:11:27.871Z","last_seen":"2026-10-25T23:43:35.289Z"},"identity":{"verified_requests":5916,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":1652,"strongest_proof":"address","reading":"verified_requests came from the ranges Ahrefs publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"disallowed_fetch_unrecorded","robots_txt_fetches":108,"disallowed_fetches":3,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T23:53:27.541Z","reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":8,"peak_at":"2026-10-21T18:15","published_limit":null,"measured_since":"2026-10-07T23:53:27.541Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":5,"share_of_verified_since_measured":0.001,"measured_since":"2026-10-07T23:53:27.541Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-18":118,"2026-10-19":367,"2026-10-20":532,"2026-10-21":97,"2026-10-22":586,"2026-10-23":802,"2026-10-24":345,"2026-10-25":289},"grade":null,"grade_reasons":["a disallowed fetch was counted before fetches were recorded with time and path"]},{"agent":"Bingbot","operator":"Microsoft","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-14T06:25:41.331Z","last_seen":"2026-10-25T23:58:20.712Z"},"identity":{"verified_requests":4498,"forged_claims":47,"forgery_share":0.01,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Microsoft publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":67,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T03:20:08.676Z","reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":5,"peak_at":"2026-10-23T08:28","published_limit":null,"measured_since":"2026-10-07T03:20:08.676Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":100,"share_of_verified_since_measured":0.022,"measured_since":"2026-10-07T03:20:08.676Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":890,"verified":822,"spoofed":68},"daily_verified":{"2026-10-18":136,"2026-10-19":168,"2026-10-20":225,"2026-10-21":214,"2026-10-22":163,"2026-10-23":238,"2026-10-24":238,"2026-10-25":215},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"GPTBot","operator":"OpenAI","purpose":"training","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-12T06:20:42.861Z","last_seen":"2026-10-25T01:51:47.001Z"},"identity":{"verified_requests":4258,"forged_claims":123,"forgery_share":0.028,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges OpenAI publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"robots_txt_not_fetched","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T18:09:36.470Z","reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":83,"peak_at":"2026-10-18T03:37","published_limit":null,"measured_since":"2026-10-07T18:09:36.470Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T18:09:36.470Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":240,"verified":21,"spoofed":216},"daily_verified":{"2026-10-19":8,"2026-10-20":2,"2026-10-21":2,"2026-10-22":2,"2026-10-23":10,"2026-10-24":2,"2026-10-25":2},"grade":"B","grade_reasons":["no disallowed path fetched, but it never fetched robots.txt under its own name here, so it cannot be credited with reading the rules"]},{"agent":"OAI-SearchBot","operator":"OpenAI","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-13T02:14:49.166Z","last_seen":"2026-10-25T13:43:12.767Z"},"identity":{"verified_requests":2388,"forged_claims":87,"forgery_share":0.035,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges OpenAI publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"respected","robots_txt_fetches":199,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T03:30:28.967Z","reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":110,"peak_at":"2026-10-16T01:39","published_limit":null,"measured_since":"2026-10-07T03:30:28.967Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T03:30:28.967Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":647,"verified":602,"spoofed":45},"daily_verified":{"2026-10-18":25,"2026-10-19":34,"2026-10-20":6,"2026-10-21":20,"2026-10-22":42,"2026-10-23":188,"2026-10-24":45,"2026-10-25":5},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"YandexBot","operator":"Yandex","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-19T13:53:12.960Z","last_seen":"2026-10-25T14:21:04.558Z"},"identity":{"verified_requests":1593,"forged_claims":7,"forgery_share":0.004,"signed_requests":0,"failed_signatures":0,"strongest_proof":"reverse_dns","reading":"verified_requests passed forward-confirmed reverse DNS to Yandex's own domains (yandex.ru, yandex.net, yandex.com); forged_claims used this agent's name from addresses that did not (impersonators, not the agent). Reverse-DNS checks began 10 October 2026; earlier hits stay unverifiable"},"robots_txt":{"verdict":"respected","robots_txt_fetches":166,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-10T12:54:23.939Z","reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":25,"peak_at":"2026-10-15T05:43","published_limit":null,"measured_since":"2026-10-10T12:54:23.939Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-10T12:54:23.939Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-18":2,"2026-10-20":22,"2026-10-21":62,"2026-10-22":70,"2026-10-23":17,"2026-10-24":52,"2026-10-25":7},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"ChatGPT-User","operator":"OpenAI","purpose":"live fetch","user_triggered":true,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-13T22:52:23.214Z","last_seen":"2026-10-26T00:02:34.112Z"},"identity":{"verified_requests":859,"forged_claims":80,"forgery_share":0.085,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges OpenAI publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"not_applicable","robots_txt_fetches":1,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T03:26:16.042Z","reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":13,"peak_at":"2026-10-15T19:36","published_limit":null,"measured_since":"2026-10-07T03:26:16.042Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T03:26:16.042Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":213,"verified":200,"spoofed":13},"daily_verified":{"2026-10-18":13,"2026-10-19":20,"2026-10-20":14,"2026-10-21":39,"2026-10-22":28,"2026-10-23":32,"2026-10-24":16,"2026-10-25":21,"2026-10-26":1},"grade":"A","grade_reasons":["fetches pages a person asked for; its operator states robots.txt does not govern those fetches, and every counted request came from its published addresses"]},{"agent":"SeznamBot","operator":"Seznam","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-10-10T13:04:32.776Z","last_seen":"2026-10-25T23:42:00.859Z"},"identity":{"verified_requests":489,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"strongest_proof":"reverse_dns","reading":"verified_requests passed forward-confirmed reverse DNS to Seznam's own domains (seznam.cz); forged_claims used this agent's name from addresses that did not (impersonators, not the agent). Reverse-DNS checks began 10 October 2026; earlier hits stay unverifiable"},"robots_txt":{"verdict":"respected","robots_txt_fetches":12,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-10T13:04:32.776Z","reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":3,"peak_at":"2026-10-10T22:12","published_limit":null,"measured_since":"2026-10-10T13:04:32.776Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":412,"share_of_verified_since_measured":0.843,"measured_since":"2026-10-10T13:04:32.776Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-18":30,"2026-10-19":62,"2026-10-20":32,"2026-10-21":26,"2026-10-22":39,"2026-10-23":24,"2026-10-24":24,"2026-10-25":31},"grade":"A","grade_reasons":["fetched robots.txt and fetched no path it disallows"]},{"agent":"Baiduspider","operator":"Baidu","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-13T21:31:01.445Z","last_seen":"2026-10-25T23:28:05.743Z"},"identity":{"verified_requests":247,"forged_claims":26,"forgery_share":0.095,"signed_requests":0,"failed_signatures":0,"strongest_proof":"reverse_dns","reading":"verified_requests passed forward-confirmed reverse DNS to Baidu's own domains (baidu.com, baidu.jp); forged_claims used this agent's name from addresses that did not (impersonators, not the agent). Reverse-DNS checks began 10 October 2026; earlier hits stay unverifiable"},"robots_txt":{"verdict":"robots_txt_not_fetched","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-10T13:14:25.713Z","reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":4,"peak_at":"2026-10-17T02:09","published_limit":null,"measured_since":"2026-10-10T13:14:25.713Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-10T13:14:25.713Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-18":5,"2026-10-19":18,"2026-10-20":25,"2026-10-21":23,"2026-10-22":7,"2026-10-23":10,"2026-10-24":24,"2026-10-25":14},"grade":"B","grade_reasons":["no disallowed path fetched, but it never fetched robots.txt under its own name here, so it cannot be credited with reading the rules"]},{"agent":"Google-CWS","operator":"Google","purpose":"store listing fetch","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-09-18T16:25:39.959Z","last_seen":"2026-10-25T12:39:38.138Z"},"identity":{"verified_requests":199,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"robots_txt_not_fetched","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-07T11:31:33.296Z","reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":3,"peak_at":"2026-10-07T20:31","published_limit":null,"measured_since":"2026-10-07T11:31:33.296Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T11:31:33.296Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-18":2,"2026-10-20":2,"2026-10-21":11,"2026-10-22":6,"2026-10-23":5,"2026-10-24":5,"2026-10-25":1},"grade":"B","grade_reasons":["no disallowed path fetched, but it never fetched robots.txt under its own name here, so it cannot be credited with reading the rules"]},{"agent":"Claude-User","operator":"Anthropic","purpose":"live fetch","user_triggered":true,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-12T22:57:31.977Z","last_seen":"2026-10-25T20:14:13.546Z"},"identity":{"verified_requests":97,"forged_claims":54,"forgery_share":0.358,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Anthropic publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"not_applicable","robots_txt_fetches":14,"disallowed_fetches":1,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[{"at":"2026-10-17T04:44:31.376Z","path":"/r/zqdnb5b58f","net":"160.79.106.0/24"}],"counted_since":"2026-10-07T11:03:04.563Z","reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"measured","peak_requests_per_minute":3,"peak_at":"2026-10-08T16:11","published_limit":null,"measured_since":"2026-10-07T11:03:04.563Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-07T11:03:04.563Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":20,"verified":4,"spoofed":16},"daily_verified":{"2026-10-25":1},"grade":"A","grade_reasons":["fetches pages a person asked for; its operator states robots.txt does not govern those fetches, and every counted request came from its published addresses"]},{"agent":"DuckDuckBot","operator":"DuckDuckGo","purpose":"indexing","user_triggered":false,"conduct":"no_violations_observed","window":{"first_seen":"2026-08-12T20:29:50.246Z","last_seen":"2026-10-25T09:24:11.731Z"},"identity":{"verified_requests":52,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":12,"strongest_proof":"address","reading":"verified_requests came from the ranges DuckDuckGo publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"robots_txt_not_fetched","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-10T21:21:35.448Z","reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":2,"peak_at":"2026-10-10T21:21","published_limit":null,"measured_since":"2026-10-10T21:21:35.448Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-10T21:21:35.448Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-18":2,"2026-10-19":5,"2026-10-20":5,"2026-10-21":4,"2026-10-22":7,"2026-10-23":4,"2026-10-24":3,"2026-10-25":3},"grade":"B","grade_reasons":["no disallowed path fetched, but it never fetched robots.txt under its own name here, so it cannot be credited with reading the rules"]},{"agent":"Perplexity-User","operator":"Perplexity","purpose":"live fetch","user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-08-14T01:04:08.513Z","last_seen":"2026-10-23T09:41:36.723Z"},"identity":{"verified_requests":8,"forged_claims":51,"forgery_share":0.864,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Perplexity publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":15,"verified":0,"spoofed":15},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-NotebookLM","operator":"Google","purpose":null,"user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-09T19:50:34.360Z","last_seen":"2026-10-13T20:31:29.773Z"},"identity":{"verified_requests":7,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-09T19:50:34.360Z","reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":4,"peak_at":"2026-10-09T19:50","published_limit":null,"measured_since":"2026-10-09T19:50:34.360Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-09T19:50:34.360Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-Site-Verification","operator":"Google","purpose":"ownership check","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-03T21:54:40.584Z","last_seen":"2026-10-03T21:54:40.980Z"},"identity":{"verified_requests":2,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Yeti","operator":"Naver","purpose":"indexing","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-10-20T06:52:03.909Z","last_seen":"2026-10-20T06:52:03.909Z"},"identity":{"verified_requests":1,"forged_claims":0,"forgery_share":0,"signed_requests":0,"failed_signatures":0,"strongest_proof":"reverse_dns","reading":"verified_requests passed forward-confirmed reverse DNS to Naver's own domains (naver.com); forged_claims used this agent's name from addresses that did not (impersonators, not the agent). Reverse-DNS checks began 10 October 2026; earlier hits stay unverifiable"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":1,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":"2026-10-20T06:52:03.909Z","reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"measured","peak_requests_per_minute":1,"peak_at":"2026-10-20T06:52","published_limit":null,"measured_since":"2026-10-20T06:52:03.909Z","reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":0,"measured_since":"2026-10-20T06:52:03.909Z","reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{"2026-10-20":1},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Google-Extended","operator":"Google","purpose":"training","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-08-15T09:11:18.716Z","last_seen":"2026-09-14T23:53:48.097Z"},"identity":{"verified_requests":0,"forged_claims":49,"forgery_share":1,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Google publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Claude-SearchBot","operator":"Anthropic","purpose":"indexing","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-08-16T01:38:23.331Z","last_seen":"2026-10-23T09:41:36.700Z"},"identity":{"verified_requests":0,"forged_claims":53,"forgery_share":1,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Anthropic publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":35,"verified":0,"spoofed":35},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"anthropic-ai","operator":"Anthropic","purpose":"training","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-08-19T11:41:41.601Z","last_seen":"2026-09-24T22:51:40.795Z"},"identity":{"verified_requests":0,"forged_claims":4,"forgery_share":1,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Anthropic publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"Applebot-Extended","operator":"Apple","purpose":"training","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-08-19T11:41:43.412Z","last_seen":"2026-08-19T11:42:02.678Z"},"identity":{"verified_requests":0,"forged_claims":1,"forgery_share":1,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Apple publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"MistralAI-User","operator":"Mistral","purpose":"live fetch","user_triggered":true,"conduct":"insufficient_data","window":{"first_seen":"2026-08-16T01:38:24.858Z","last_seen":"2026-09-12T16:50:37.743Z"},"identity":{"verified_requests":0,"forged_claims":0,"forgery_share":null,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges Mistral publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reading":"this agent fetches pages a user asked for; its operator states robots.txt does not govern those fetches, so disallowed fetches are reported, not judged"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":1,"verified":0,"spoofed":1},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]},{"agent":"FeedFetcher-Google","operator":"Google","purpose":"feed fetch","user_triggered":false,"conduct":"insufficient_data","window":{"first_seen":"2026-09-28T08:19:13.036Z","last_seen":"2026-09-28T08:19:13.036Z"},"identity":{"verified_requests":0,"forged_claims":1,"forgery_share":1,"signed_requests":0,"failed_signatures":0,"strongest_proof":"address","reading":"verified_requests came from the ranges the operator publishes; forged_claims used this agent's name from outside them (impersonators, not the agent)"},"robots_txt":{"verdict":"insufficient_data","robots_txt_fetches":0,"disallowed_fetches":0,"disallowed_rules":["/r/","/backup/","/admin-old/"],"disallowed_fetch_records":[],"counted_since":null,"reading":"verified requests to paths this site's robots.txt disallows for every agent, including a group that names this one; a lower bound before 7 Oct 2026 (per-path map capped at 40 paths)"},"rate":{"verdict":"not_yet_measured","peak_requests_per_minute":null,"peak_at":null,"published_limit":null,"measured_since":null,"reading":"the busiest minute of verified requests observed here since the counter started, all of the operator's addresses combined; a lower bound. This site publishes no crawl rate (its robots.txt has no Crawl-delay), so the rate is reported, never judged"},"revalidation":{"conditional_requests":0,"share_of_verified_since_measured":null,"measured_since":null,"reading":"requests carrying If-None-Match or If-Modified-Since: an agent that revalidates instead of refetching"},"site_checks_14d":{"checked":0,"verified":0,"spoofed":0},"daily_verified":{},"grade":null,"grade_reasons":["fewer than 50 verified requests so far"]}],"not_assessed":[{"agent":"Meta-ExternalAgent","claimed_hits":42952,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"Bytespider","claimed_hits":308,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"CCBot","claimed_hits":1115,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"Diffbot","claimed_hits":1,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"SemrushBot","claimed_hits":12534,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"DataForSeoBot","claimed_hits":7206,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"Timpibot","claimed_hits":4,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"YouBot","claimed_hits":741,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"Scrapy","claimed_hits":13,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"cohere-ai","claimed_hits":40,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"},{"agent":"FacebookBot","claimed_hits":1,"why":"no address-range file this record reads, so no request can be tied to the operator; nothing it did here can be told from an impersonator"}],"census":"https://crawlcheck.io/api/telemetry","visitor_checks":"https://crawlcheck.io/api/v1/visitor/stats","code":{"version_id":"4e7ce319-9f4e-482a-a802-8af4f8a75220","script_sha16":null,"state":"no_receipt","deployed":null,"build_log_entry":null,"score_version":34,"scoring_policy_sha256":"76ee7526c402d711f2c976ad88ae13d4d3411aaa9d5ca3dc87ab3102396b1370","decision_rules_sha256":"18dd5636e0896692ab518050cb401744edea2e6f1086dff94fead314959bdc3e","check":"https://crawlcheck.io/api/v1/builds/4e7ce319-9f4e-482a-a802-8af4f8a75220"},"sha256":"31d8bcabe6c0d0ca92b9aa0a7331f5525e6c898d715e6364dd1fe9689f40db48","signature":{"v":1,"alg":"Ed25519","kid":"1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps","over":"sha256","message":"crawlcheck-data-v1\n31d8bcabe6c0d0ca92b9aa0a7331f5525e6c898d715e6364dd1fe9689f40db48","sig":"Xr38mG-XJVxXyeHjdeTRVz9zJMwjfb28dCQqFXEobsYwrbjSlbIhLv7kv0J26qmG4mLGjai4Jjr-9yYU5omGAw","signed_at":"2026-10-26T00:04:57.318Z","key":{"kid":"1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps","jwk":{"kty":"OKP","crv":"Ed25519","x":"5TmKCOx0TTHepq7HwyXBHvGQpVCSBd2lV8cLzYVqFxQ"},"published_in":"https://crawlcheck.io/.well-known/http-message-signatures-directory"},"canonical":"sha256 over canonical JSON of every field except sha256 and signature"}}