{"ok":true,"kind":"crawlcheck-agentic-web-census","v":1,"at":"2026-11-17T11:18:08.542Z","frame_id":"top10k","frame_label":"top 10,000, daily","every_hours":24,"anchor":"census-10k","frames":[{"id":"top1k","label":"top 1,000, daily","summary":"https://crawlcheck.io/api/census"},{"id":"top10k","label":"top 10,000, daily","summary":"https://crawlcheck.io/api/census?frame=top10k"},{"id":"top100k","label":"top 100,000, monthly","summary":"https://crawlcheck.io/api/census?frame=top100k"}],"probe_note":null,"frame":{"source":"Tranco","list_id":"Y83YG","list_url":"https://tranco-list.eu/list/Y83YG/10000","list_created_on":"2026-10-01T22:00:02.056039","requested":10000,"members":9776,"excluded":{"publish_filter":223,"opted_out":0},"frozen_at":"2026-10-03T14:32:49.676Z","rule":"The top 10000 registrable domains of one Tranco list (a public ranking combining CrUX, Farsight, Majestic, Cloudflare Radar and Umbrella), frozen at the first run so every run measures the same sites. Hosts the publish filter refuses (adult, gambling) and opted-out domains leave the frame and are counted, never named. Infrastructure domains in the list (CDNs, API hosts) stay in: one that serves no homepage is counted as unreachable."},"shards":8,"runs":[{"run":1,"started_at":"2026-10-03T14:32:50.127Z","finished_at":"2026-10-03T21:53:40.747Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=1"},{"run":2,"started_at":"2026-10-10T14:35:35.291Z","finished_at":"2026-10-10T23:13:15.648Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=2"},{"run":3,"started_at":"2026-10-17T14:43:10.267Z","finished_at":"2026-10-17T18:23:49.920Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=3"},{"run":4,"started_at":"2026-10-24T14:52:06.331Z","finished_at":"2026-10-24T23:42:20.214Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=4"},{"run":5,"started_at":"2026-10-31T14:52:13.341Z","finished_at":"2026-10-31T21:02:55.469Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=5"},{"run":6,"started_at":"2026-11-01T17:43:03.642Z","finished_at":"2026-11-02T02:35:41.703Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=6"},{"run":7,"started_at":"2026-11-02T17:46:34.069Z","finished_at":"2026-11-03T05:19:47.957Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=7"},{"run":8,"started_at":"2026-11-03T17:46:36.711Z","finished_at":"2026-11-04T03:45:48.281Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=8"},{"run":9,"started_at":"2026-11-04T17:46:37.516Z","finished_at":"2026-11-05T00:34:35.201Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=9"},{"run":10,"started_at":"2026-11-05T17:56:16.828Z","finished_at":"2026-11-05T22:35:16.648Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=10"},{"run":11,"started_at":"2026-11-06T17:56:55.730Z","finished_at":"2026-11-07T00:09:58.632Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=11"},{"run":12,"started_at":"2026-11-07T18:00:54.641Z","finished_at":"2026-11-08T00:06:35.271Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=12"},{"run":13,"started_at":"2026-11-08T18:10:23.240Z","finished_at":"2026-11-09T00:36:10.944Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=13"},{"run":14,"started_at":"2026-11-09T18:19:22.789Z","finished_at":"2026-11-10T01:07:21.108Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=14"},{"run":15,"started_at":"2026-11-10T18:28:51.048Z","finished_at":"2026-11-11T00:52:39.562Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=15"},{"run":16,"started_at":"2026-11-11T18:37:33.607Z","finished_at":"2026-11-12T00:35:45.080Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=16"},{"run":17,"started_at":"2026-11-12T18:37:34.143Z","finished_at":"2026-11-13T00:53:51.813Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=17"},{"run":18,"started_at":"2026-11-13T18:46:25.970Z","finished_at":"2026-11-14T01:34:48.953Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=18"},{"run":19,"started_at":"2026-11-14T18:56:54.746Z","finished_at":"2026-11-15T03:08:26.484Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=19"},{"run":20,"started_at":"2026-11-15T18:56:54.910Z","finished_at":"2026-11-16T03:17:00.502Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=20"},{"run":21,"started_at":"2026-11-16T18:56:55.185Z","finished_at":"2026-11-17T02:36:01.244Z","measured":9776,"of":9776,"shards":8,"records":"https://crawlcheck.io/api/census/records?frame=top10k&run=21"}],"running":null,"latest":{"run":21,"started_at":"2026-11-16T18:56:55.185Z","finished_at":"2026-11-17T02:36:01.244Z","sites":9776,"figures":[{"id":"reachable","group":"Reach","label":"Homepage answers CrawlCheck's census fetch (2xx after redirects)","count":5636,"of":9776,"pct":57.7},{"id":"refused_us","group":"Reach","label":"Homepage refuses CrawlCheck's census fetch (400, 401, 403 or 429): blocked or rate-limited, not down","count":1365,"of":9776,"pct":14},{"id":"no_site","group":"Reach","label":"No website answers at that name (5xx, 530, timeout or no answer)","count":2574,"of":9776,"pct":26.3},{"id":"robots_served","group":"Crawler policy","label":"Serves a robots.txt (200, not HTML)","count":4614,"of":5636,"pct":81.9},{"id":"robots_sitemap","group":"Crawler policy","label":"robots.txt names a sitemap","count":3248,"of":4614,"pct":70.4},{"id":"content_signal","group":"Crawler policy","label":"robots.txt carries a Content-Signal line","count":300,"of":4614,"pct":6.5},{"id":"cs_ai_train_no","group":"Crawler policy","label":"Content-Signal says ai-train=no","count":229,"of":4614,"pct":5},{"id":"cs_ai_input_no","group":"Crawler policy","label":"Content-Signal says ai-input=no or search=no","count":22,"of":4614,"pct":0.5},{"id":"aipref_train","group":"Crawler policy","label":"robots.txt names the AI-training policy token (__usage_ai-training, IETF AIPREF draft)","count":22,"of":4610,"pct":0.5},{"id":"aipref_train_no","group":"Crawler policy","label":"robots.txt states a preference against AI training on / with that token","count":22,"of":4610,"pct":0.5},{"id":"blocks_any_ai","group":"Crawler policy","label":"robots.txt closes / to at least one of the 12 AI agents","count":1012,"of":4614,"pct":21.9},{"id":"blocks_GPTBot","group":"Crawler policy","label":"robots.txt closes / to GPTBot","count":739,"of":4614,"pct":16},{"id":"blocks_OAI-SearchBot","group":"Crawler policy","label":"robots.txt closes / to OAI-SearchBot","count":278,"of":4614,"pct":6},{"id":"blocks_ChatGPT-User","group":"Crawler policy","label":"robots.txt closes / to ChatGPT-User","count":363,"of":4614,"pct":7.9},{"id":"blocks_ClaudeBot","group":"Crawler policy","label":"robots.txt closes / to ClaudeBot","count":686,"of":4614,"pct":14.9},{"id":"blocks_Claude-SearchBot","group":"Crawler policy","label":"robots.txt closes / to Claude-SearchBot","count":264,"of":4614,"pct":5.7},{"id":"blocks_PerplexityBot","group":"Crawler policy","label":"robots.txt closes / to PerplexityBot","count":404,"of":4614,"pct":8.8},{"id":"blocks_Google-Extended","group":"Crawler policy","label":"robots.txt closes / to Google-Extended","count":614,"of":4614,"pct":13.3},{"id":"blocks_Applebot-Extended","group":"Crawler policy","label":"robots.txt closes / to Applebot-Extended","count":603,"of":4614,"pct":13.1},{"id":"blocks_CCBot","group":"Crawler policy","label":"robots.txt closes / to CCBot","count":788,"of":4614,"pct":17.1},{"id":"blocks_Bytespider","group":"Crawler policy","label":"robots.txt closes / to Bytespider","count":787,"of":4614,"pct":17.1},{"id":"blocks_Meta-ExternalAgent","group":"Crawler policy","label":"robots.txt closes / to Meta-ExternalAgent","count":639,"of":4614,"pct":13.8},{"id":"blocks_Amazonbot","group":"Crawler policy","label":"robots.txt closes / to Amazonbot","count":620,"of":4614,"pct":13.4},{"id":"llms_txt","group":"Machine files","label":"Serves /llms.txt","count":963,"of":5636,"pct":17.1},{"id":"agents_md","group":"Machine files","label":"Serves /agents.md","count":83,"of":5636,"pct":1.5},{"id":"agent_skills","group":"Capabilities","label":"Publishes an agent-skills index","count":60,"of":5636,"pct":1.1},{"id":"mcp_card","group":"Capabilities","label":"Publishes an MCP server card","count":67,"of":5636,"pct":1.2},{"id":"api_catalog","group":"Capabilities","label":"Publishes an API catalog (RFC 9727)","count":73,"of":5636,"pct":1.3},{"id":"a2a_card","group":"Capabilities","label":"Publishes an A2A agent card","count":17,"of":5636,"pct":0.3},{"id":"any_capability","group":"Capabilities","label":"Declares any agent capability (any of the four above)","count":88,"of":5636,"pct":1.6},{"id":"ai_refused_any","group":"Delivery parity","label":"At least one AI identity is refused (non-2xx) where the browser got 2xx","count":1272,"of":5636,"pct":22.6},{"id":"ai_thin_any","group":"Delivery parity","label":"At least one AI identity gets under 70% of the browser's bytes (both 2xx)","count":125,"of":5636,"pct":2.2},{"id":"mcp_declared","group":"Endpoint rings","label":"Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one)","count":331,"of":5636,"pct":5.9},{"id":"mcp_live","group":"Endpoint rings","label":"An MCP endpoint completes the handshake (initialize and tools/list; no tool called)","count":52,"of":5636,"pct":0.9},{"id":"mcp_live_of_declared","group":"Endpoint rings","label":"Of sites naming an MCP endpoint, one completes the handshake or asks for credentials","count":260,"of":331,"pct":78.5},{"id":"a2a_declared","group":"Endpoint rings","label":"The A2A agent card names an endpoint url","count":9,"of":5636,"pct":0.2},{"id":"a2a_answers","group":"Endpoint rings","label":"Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page)","count":4,"of":9,"pct":44.4},{"id":"webmcp","group":"Endpoint rings","label":"Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block)","count":5,"of":5636,"pct":0.1},{"id":"sb_listed","group":"Safety","label":"Listed by Google Safe Browsing (all or some pages): browsers show a red warning","count":40,"of":4374,"pct":0.9},{"id":"cf_listed","group":"Safety","label":"Listed as malware or phishing by a public malware-blocking DNS resolver","count":16,"of":5657,"pct":0.3},{"id":"hsts_preloaded","group":"Safety","label":"On the browsers' HSTS preload list (HTTPS-only from the first visit)","count":451,"of":5651,"pct":8},{"id":"outside_scripts","group":"Safety","label":"Homepage loads scripts or styles from another domain","count":3813,"of":5657,"pct":67.4},{"id":"outside_5plus","group":"Safety","label":"Homepage loads scripts or styles from 5 or more other domains","count":608,"of":5657,"pct":10.7},{"id":"any_integrity","group":"Safety","label":"Pins at least one outside script or style with integrity=","count":376,"of":3813,"pct":9.9},{"id":"unpinned_cdn","group":"Safety","label":"Loads a library from a public CDN without integrity=","count":750,"of":5657,"pct":13.3},{"id":"hijacked_cdn","group":"Safety","label":"Still loads scripts from a CDN taken over in 2024 (polyfill.io and its sister CDNs)","count":4,"of":5657,"pct":0.1},{"id":"hack_signs","group":"Safety","label":"Homepage shows hidden spam links or cryptominer code","count":1,"of":5657,"pct":0},{"id":"refused_gptbot","group":"Delivery parity","label":"GPTBot refused where the browser got 2xx","count":1084,"of":5636,"pct":19.2},{"id":"refused_claudebot","group":"Delivery parity","label":"ClaudeBot refused where the browser got 2xx","count":1122,"of":5636,"pct":19.9},{"id":"refused_oaisearch","group":"Delivery parity","label":"OAI-SearchBot refused where the browser got 2xx","count":714,"of":5636,"pct":12.7},{"id":"refused_perplexity","group":"Delivery parity","label":"PerplexityBot refused where the browser got 2xx","count":931,"of":5636,"pct":16.5}]},"change_basis":"Change compares run 21 with run 1, the first run measured with the current probe. Runs before 1 are excluded from the comparison: a defect in our own probe produced false timeouts in them. points_paired (shown on the page) is like-for-like: (sites gained - sites lost) / sites compared, among sites measured in both runs and inside the figure's denominator both times; points is the plain difference of the two percentages and moves when sites leave a denominator.","first":{"run":1,"started_at":"2026-10-03T14:32:50.127Z","finished_at":"2026-10-03T21:53:40.747Z","sites":9776,"figures":[{"id":"reachable","group":"Reach","label":"Homepage answers CrawlCheck's census fetch (2xx after redirects)","count":5643,"of":9776,"pct":57.7},{"id":"refused_us","group":"Reach","label":"Homepage refuses CrawlCheck's census fetch (400, 401, 403 or 429): blocked or rate-limited, not down","count":0,"of":0,"pct":null},{"id":"no_site","group":"Reach","label":"No website answers at that name (5xx, 530, timeout or no answer)","count":0,"of":0,"pct":null},{"id":"robots_served","group":"Crawler policy","label":"Serves a robots.txt (200, not HTML)","count":4562,"of":5643,"pct":80.8},{"id":"robots_sitemap","group":"Crawler policy","label":"robots.txt names a sitemap","count":3180,"of":4562,"pct":69.7},{"id":"content_signal","group":"Crawler policy","label":"robots.txt carries a Content-Signal line","count":146,"of":4562,"pct":3.2},{"id":"cs_ai_train_no","group":"Crawler policy","label":"Content-Signal says ai-train=no","count":89,"of":4562,"pct":2},{"id":"cs_ai_input_no","group":"Crawler policy","label":"Content-Signal says ai-input=no or search=no","count":22,"of":4562,"pct":0.5},{"id":"aipref_train","group":"Crawler policy","label":"robots.txt names the AI-training policy token (__usage_ai-training, IETF AIPREF draft)","count":0,"of":0,"pct":null},{"id":"aipref_train_no","group":"Crawler policy","label":"robots.txt states a preference against AI training on / with that token","count":0,"of":0,"pct":null},{"id":"blocks_any_ai","group":"Crawler policy","label":"robots.txt closes / to at least one of the 12 AI agents","count":873,"of":4562,"pct":19.1},{"id":"blocks_GPTBot","group":"Crawler policy","label":"robots.txt closes / to GPTBot","count":626,"of":4562,"pct":13.7},{"id":"blocks_OAI-SearchBot","group":"Crawler policy","label":"robots.txt closes / to OAI-SearchBot","count":292,"of":4562,"pct":6.4},{"id":"blocks_ChatGPT-User","group":"Crawler policy","label":"robots.txt closes / to ChatGPT-User","count":382,"of":4562,"pct":8.4},{"id":"blocks_ClaudeBot","group":"Crawler policy","label":"robots.txt closes / to ClaudeBot","count":563,"of":4562,"pct":12.3},{"id":"blocks_Claude-SearchBot","group":"Crawler policy","label":"robots.txt closes / to Claude-SearchBot","count":267,"of":4562,"pct":5.9},{"id":"blocks_PerplexityBot","group":"Crawler policy","label":"robots.txt closes / to PerplexityBot","count":431,"of":4562,"pct":9.4},{"id":"blocks_Google-Extended","group":"Crawler policy","label":"robots.txt closes / to Google-Extended","count":476,"of":4562,"pct":10.4},{"id":"blocks_Applebot-Extended","group":"Crawler policy","label":"robots.txt closes / to Applebot-Extended","count":460,"of":4562,"pct":10.1},{"id":"blocks_CCBot","group":"Crawler policy","label":"robots.txt closes / to CCBot","count":678,"of":4562,"pct":14.9},{"id":"blocks_Bytespider","group":"Crawler policy","label":"robots.txt closes / to Bytespider","count":640,"of":4562,"pct":14},{"id":"blocks_Meta-ExternalAgent","group":"Crawler policy","label":"robots.txt closes / to Meta-ExternalAgent","count":508,"of":4562,"pct":11.1},{"id":"blocks_Amazonbot","group":"Crawler policy","label":"robots.txt closes / to Amazonbot","count":466,"of":4562,"pct":10.2},{"id":"llms_txt","group":"Machine files","label":"Serves /llms.txt","count":871,"of":5643,"pct":15.4},{"id":"agents_md","group":"Machine files","label":"Serves /agents.md","count":74,"of":5643,"pct":1.3},{"id":"agent_skills","group":"Capabilities","label":"Publishes an agent-skills index","count":51,"of":5643,"pct":0.9},{"id":"mcp_card","group":"Capabilities","label":"Publishes an MCP server card","count":53,"of":5643,"pct":0.9},{"id":"api_catalog","group":"Capabilities","label":"Publishes an API catalog (RFC 9727)","count":62,"of":5643,"pct":1.1},{"id":"a2a_card","group":"Capabilities","label":"Publishes an A2A agent card","count":16,"of":5643,"pct":0.3},{"id":"any_capability","group":"Capabilities","label":"Declares any agent capability (any of the four above)","count":75,"of":5643,"pct":1.3},{"id":"ai_refused_any","group":"Delivery parity","label":"At least one AI identity is refused (non-2xx) where the browser got 2xx","count":1294,"of":5643,"pct":22.9},{"id":"ai_thin_any","group":"Delivery parity","label":"At least one AI identity gets under 70% of the browser's bytes (both 2xx)","count":138,"of":5643,"pct":2.4},{"id":"mcp_declared","group":"Endpoint rings","label":"Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one)","count":0,"of":0,"pct":null},{"id":"mcp_live","group":"Endpoint rings","label":"An MCP endpoint completes the handshake (initialize and tools/list; no tool called)","count":0,"of":0,"pct":null},{"id":"mcp_live_of_declared","group":"Endpoint rings","label":"Of sites naming an MCP endpoint, one completes the handshake or asks for credentials","count":0,"of":0,"pct":null},{"id":"a2a_declared","group":"Endpoint rings","label":"The A2A agent card names an endpoint url","count":0,"of":0,"pct":null},{"id":"a2a_answers","group":"Endpoint rings","label":"Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page)","count":0,"of":0,"pct":null},{"id":"webmcp","group":"Endpoint rings","label":"Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block)","count":0,"of":0,"pct":null},{"id":"sb_listed","group":"Safety","label":"Listed by Google Safe Browsing (all or some pages): browsers show a red warning","count":0,"of":0,"pct":null},{"id":"cf_listed","group":"Safety","label":"Listed as malware or phishing by a public malware-blocking DNS resolver","count":0,"of":0,"pct":null},{"id":"hsts_preloaded","group":"Safety","label":"On the browsers' HSTS preload list (HTTPS-only from the first visit)","count":0,"of":0,"pct":null},{"id":"outside_scripts","group":"Safety","label":"Homepage loads scripts or styles from another domain","count":0,"of":0,"pct":null},{"id":"outside_5plus","group":"Safety","label":"Homepage loads scripts or styles from 5 or more other domains","count":0,"of":0,"pct":null},{"id":"any_integrity","group":"Safety","label":"Pins at least one outside script or style with integrity=","count":0,"of":0,"pct":null},{"id":"unpinned_cdn","group":"Safety","label":"Loads a library from a public CDN without integrity=","count":0,"of":0,"pct":null},{"id":"hijacked_cdn","group":"Safety","label":"Still loads scripts from a CDN taken over in 2024 (polyfill.io and its sister CDNs)","count":0,"of":0,"pct":null},{"id":"hack_signs","group":"Safety","label":"Homepage shows hidden spam links or cryptominer code","count":0,"of":0,"pct":null},{"id":"refused_gptbot","group":"Delivery parity","label":"GPTBot refused where the browser got 2xx","count":1105,"of":5643,"pct":19.6},{"id":"refused_claudebot","group":"Delivery parity","label":"ClaudeBot refused where the browser got 2xx","count":1117,"of":5643,"pct":19.8},{"id":"refused_oaisearch","group":"Delivery parity","label":"OAI-SearchBot refused where the browser got 2xx","count":694,"of":5643,"pct":12.3},{"id":"refused_perplexity","group":"Delivery parity","label":"PerplexityBot refused where the browser got 2xx","count":718,"of":5643,"pct":12.7}]},"change":[{"id":"reachable","pct_now":57.7,"pct_first":57.7,"points":0,"points_paired":-0.1,"sites_gained":267,"sites_lost":274,"compared":9776},{"id":"refused_us","pct_now":14,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"no_site","pct_now":26.3,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"robots_served","pct_now":81.9,"pct_first":80.8,"points":1.1,"points_paired":1.1,"sites_gained":104,"sites_lost":47,"compared":5369},{"id":"robots_sitemap","pct_now":70.4,"pct_first":69.7,"points":0.7,"points_paired":1,"sites_gained":62,"sites_lost":21,"compared":4299},{"id":"content_signal","pct_now":6.5,"pct_first":3.2,"points":3.3,"points_paired":2.8,"sites_gained":123,"sites_lost":2,"compared":4299},{"id":"cs_ai_train_no","pct_now":5,"pct_first":2,"points":3,"points_paired":2.5,"sites_gained":113,"sites_lost":4,"compared":4299},{"id":"cs_ai_input_no","pct_now":0.5,"pct_first":0.5,"points":0,"points_paired":0,"sites_gained":1,"sites_lost":1,"compared":4299},{"id":"aipref_train","pct_now":0.5,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"aipref_train_no","pct_now":0.5,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"blocks_any_ai","pct_now":21.9,"pct_first":19.1,"points":2.8,"points_paired":2.8,"sites_gained":130,"sites_lost":8,"compared":4299},{"id":"blocks_GPTBot","pct_now":16,"pct_first":13.7,"points":2.3,"points_paired":2.2,"sites_gained":111,"sites_lost":16,"compared":4299},{"id":"blocks_OAI-SearchBot","pct_now":6,"pct_first":6.4,"points":-0.4,"points_paired":-0.2,"sites_gained":8,"sites_lost":18,"compared":4299},{"id":"blocks_ChatGPT-User","pct_now":7.9,"pct_first":8.4,"points":-0.5,"points_paired":-0.3,"sites_gained":14,"sites_lost":28,"compared":4299},{"id":"blocks_ClaudeBot","pct_now":14.9,"pct_first":12.3,"points":2.6,"points_paired":2.4,"sites_gained":118,"sites_lost":14,"compared":4299},{"id":"blocks_Claude-SearchBot","pct_now":5.7,"pct_first":5.9,"points":-0.2,"points_paired":-0.1,"sites_gained":10,"sites_lost":13,"compared":4299},{"id":"blocks_PerplexityBot","pct_now":8.8,"pct_first":9.4,"points":-0.6,"points_paired":-0.3,"sites_gained":9,"sites_lost":24,"compared":4299},{"id":"blocks_Google-Extended","pct_now":13.3,"pct_first":10.4,"points":2.9,"points_paired":2.4,"sites_gained":128,"sites_lost":24,"compared":4299},{"id":"blocks_Applebot-Extended","pct_now":13.1,"pct_first":10.1,"points":3,"points_paired":2.7,"sites_gained":127,"sites_lost":10,"compared":4299},{"id":"blocks_CCBot","pct_now":17.1,"pct_first":14.9,"points":2.2,"points_paired":2.1,"sites_gained":124,"sites_lost":34,"compared":4299},{"id":"blocks_Bytespider","pct_now":17.1,"pct_first":14,"points":3.1,"points_paired":2.9,"sites_gained":131,"sites_lost":6,"compared":4299},{"id":"blocks_Meta-ExternalAgent","pct_now":13.8,"pct_first":11.1,"points":2.7,"points_paired":2.5,"sites_gained":135,"sites_lost":28,"compared":4299},{"id":"blocks_Amazonbot","pct_now":13.4,"pct_first":10.2,"points":3.2,"points_paired":2.9,"sites_gained":135,"sites_lost":11,"compared":4299},{"id":"llms_txt","pct_now":17.1,"pct_first":15.4,"points":1.7,"points_paired":1.6,"sites_gained":111,"sites_lost":23,"compared":5369},{"id":"agents_md","pct_now":1.5,"pct_first":1.3,"points":0.2,"points_paired":0.1,"sites_gained":11,"sites_lost":5,"compared":5369},{"id":"agent_skills","pct_now":1.1,"pct_first":0.9,"points":0.2,"points_paired":0.1,"sites_gained":11,"sites_lost":3,"compared":5369},{"id":"mcp_card","pct_now":1.2,"pct_first":0.9,"points":0.3,"points_paired":0.3,"sites_gained":17,"sites_lost":3,"compared":5369},{"id":"api_catalog","pct_now":1.3,"pct_first":1.1,"points":0.2,"points_paired":0.2,"sites_gained":16,"sites_lost":4,"compared":5369},{"id":"a2a_card","pct_now":0.3,"pct_first":0.3,"points":0,"points_paired":0,"sites_gained":3,"sites_lost":2,"compared":5369},{"id":"any_capability","pct_now":1.6,"pct_first":1.3,"points":0.3,"points_paired":0.2,"sites_gained":17,"sites_lost":4,"compared":5369},{"id":"ai_refused_any","pct_now":22.6,"pct_first":22.9,"points":-0.3,"points_paired":-0.1,"sites_gained":98,"sites_lost":103,"compared":5369},{"id":"ai_thin_any","pct_now":2.2,"pct_first":2.4,"points":-0.2,"points_paired":-0.5,"sites_gained":25,"sites_lost":51,"compared":5369},{"id":"mcp_declared","pct_now":5.9,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"mcp_live","pct_now":0.9,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"mcp_live_of_declared","pct_now":78.5,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"a2a_declared","pct_now":0.2,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"a2a_answers","pct_now":44.4,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"webmcp","pct_now":0.1,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"sb_listed","pct_now":0.9,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"cf_listed","pct_now":0.3,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"hsts_preloaded","pct_now":8,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"outside_scripts","pct_now":67.4,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"outside_5plus","pct_now":10.7,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"any_integrity","pct_now":9.9,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"unpinned_cdn","pct_now":13.3,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"hijacked_cdn","pct_now":0.1,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"hack_signs","pct_now":0,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"refused_gptbot","pct_now":19.2,"pct_first":19.6,"points":-0.4,"points_paired":-0.1,"sites_gained":76,"sites_lost":80,"compared":5369},{"id":"refused_claudebot","pct_now":19.9,"pct_first":19.8,"points":0.1,"points_paired":0.4,"sites_gained":104,"sites_lost":81,"compared":5369},{"id":"refused_oaisearch","pct_now":12.7,"pct_first":12.3,"points":0.4,"points_paired":0.7,"sites_gained":84,"sites_lost":45,"compared":5369},{"id":"refused_perplexity","pct_now":16.5,"pct_first":12.7,"points":3.8,"points_paired":4.2,"sites_gained":269,"sites_lost":42,"compared":5369}],"method":{"identities":"Homepage fetched once each as a mobile browser and as GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot, from CrawlCheck's network, following redirects. A refusal of a named crawler from this network may be identity verification (the crawler's real addresses are not ours), so delivery figures read 'refused from this vantage', not 'blocked'.","robots":"robots.txt read once; each of 12 AI agents' access to / is decided per RFC 9309 (the most specific matching group; longest-match rules). The text is kept in the record.","files":"A file counts as present when it answers 200, is not an HTML page, is not empty, and (for the JSON ones) parses as a JSON object.","rings":"Endpoint rings (from run derive_v 3): MCP endpoints named by the server card, llms.txt, agents.md or a homepage link, plus GET /mcp on the site and on mcp.<domain> when the answer is the shape a streamable-HTTP server gives (405/406/415, an SSE stream, a JSON-RPC error, a Bearer challenge; never an HTML page or a 404). Up to three candidates get the MCP handshake - initialize and tools/list, never a tool call. The A2A card's url gets one GET. WebMCP is navigator.modelContext, a WebMCP bridge script or an application/mcp+json block in the homepage HTML.","denominators":"Each figure states its own denominator: reach is over every frame member; machine files, capabilities and delivery are over sites a browser reached; robots figures are over sites serving a robots.txt.","recompute":"Every figure is a count over the derived fields of the run's records. Page https://crawlcheck.io/api/census/records?frame=top10k&run=N and count (or ask for one site with &domain=); each record also carries the raw readings (statuses, sizes, SHA-256 of what was served, the robots.txt text) the derived fields come from."},"api":{"weekly":"https://crawlcheck.io/api/v1/census/weekly","summary":"https://crawlcheck.io/api/census?frame=top10k","records":"https://crawlcheck.io/api/census/records?frame=top10k&run={run}&offset=0&limit=200","record":"https://crawlcheck.io/api/census/records?frame=top10k&run={run}&domain={domain}","frame":"https://crawlcheck.io/api/census/frame?frame=top10k"}}