{"ok":true,"kind":"crawlcheck-agentic-web-census","v":1,"at":"2026-10-13T08:40:20.777Z","frame_id":"top1k","frame_label":"top 1,000, daily","every_hours":24,"anchor":"census","frames":[{"id":"top1k","label":"top 1,000, daily","summary":"https://crawlcheck.io/api/census"},{"id":"top10k","label":"top 10,000, weekly","summary":"https://crawlcheck.io/api/census?frame=top10k"},{"id":"top100k","label":"top 100,000, monthly","summary":"https://crawlcheck.io/api/census?frame=top100k"}],"probe_note":null,"frame":{"source":"Tranco","list_id":"Y83YG","list_url":"https://tranco-list.eu/list/Y83YG/1000","list_created_on":"2026-10-01T22:00:02.056039","requested":1000,"members":991,"excluded":{"publish_filter":9,"opted_out":0},"frozen_at":"2026-10-02T04:32:37.816Z","rule":"The top 1000 registrable domains of one Tranco list (a public ranking combining CrUX, Farsight, Majestic, Cloudflare Radar and Umbrella), frozen at the first run so every run measures the same sites. Hosts the publish filter refuses (adult, gambling) and opted-out domains leave the frame and are counted, never named. Infrastructure domains in the list (CDNs, API hosts) stay in: one that serves no homepage is counted as unreachable."},"shards":8,"runs":[{"run":1,"started_at":"2026-10-02T04:32:37.868Z","finished_at":"2026-10-02T12:32:46.592Z","measured":991,"of":991,"shards":1,"records":"https://crawlcheck.io/api/census/records?run=1"},{"run":2,"started_at":"2026-10-03T04:32:52.846Z","finished_at":"2026-10-03T12:48:12.613Z","measured":991,"of":991,"shards":1,"records":"https://crawlcheck.io/api/census/records?run=2"},{"run":3,"started_at":"2026-10-04T04:35:29.982Z","finished_at":"2026-10-04T05:00:02.620Z","measured":991,"of":991,"shards":8,"records":"https://crawlcheck.io/api/census/records?run=3"},{"run":4,"started_at":"2026-10-05T04:35:38.310Z","finished_at":"2026-10-05T05:08:56.321Z","measured":991,"of":991,"shards":8,"records":"https://crawlcheck.io/api/census/records?run=4"},{"run":5,"started_at":"2026-10-06T04:43:38.280Z","finished_at":"2026-10-06T05:16:57.012Z","measured":991,"of":991,"shards":8,"records":"https://crawlcheck.io/api/census/records?run=5"},{"run":6,"started_at":"2026-10-07T04:43:43.665Z","finished_at":"2026-10-07T05:08:57.280Z","measured":991,"of":991,"shards":8,"records":"https://crawlcheck.io/api/census/records?run=6"},{"run":7,"started_at":"2026-10-08T04:52:39.431Z","finished_at":"2026-10-08T05:17:14.418Z","measured":991,"of":991,"shards":8,"records":"https://crawlcheck.io/api/census/records?run=7"},{"run":8,"started_at":"2026-10-09T04:52:59.028Z","finished_at":"2026-10-09T05:17:30.836Z","measured":991,"of":991,"shards":8,"records":"https://crawlcheck.io/api/census/records?run=8"},{"run":9,"started_at":"2026-10-10T05:01:23.847Z","finished_at":"2026-10-10T05:26:41.997Z","measured":991,"of":991,"shards":8,"records":"https://crawlcheck.io/api/census/records?run=9"},{"run":10,"started_at":"2026-10-11T05:09:17.389Z","finished_at":"2026-10-11T06:04:35.685Z","measured":991,"of":991,"shards":8,"records":"https://crawlcheck.io/api/census/records?run=10"},{"run":11,"started_at":"2026-10-12T05:18:18.275Z","finished_at":"2026-10-12T06:12:04.839Z","measured":991,"of":991,"shards":8,"records":"https://crawlcheck.io/api/census/records?run=11"},{"run":12,"started_at":"2026-10-13T05:18:19.873Z","finished_at":"2026-10-13T06:17:07.201Z","measured":991,"of":991,"shards":8,"records":"https://crawlcheck.io/api/census/records?run=12"}],"running":null,"latest":{"run":12,"started_at":"2026-10-13T05:18:19.873Z","finished_at":"2026-10-13T06:17:07.201Z","sites":991,"figures":[{"id":"reachable","group":"Reach","label":"Homepage answers a browser (2xx after redirects)","count":513,"of":991,"pct":51.8},{"id":"robots_served","group":"Crawler policy","label":"Serves a robots.txt (200, not HTML)","count":410,"of":513,"pct":79.9},{"id":"robots_sitemap","group":"Crawler policy","label":"robots.txt names a sitemap","count":289,"of":410,"pct":70.5},{"id":"content_signal","group":"Crawler policy","label":"robots.txt carries a Content-Signal line","count":15,"of":410,"pct":3.7},{"id":"cs_ai_train_no","group":"Crawler policy","label":"Content-Signal says ai-train=no","count":9,"of":410,"pct":2.2},{"id":"cs_ai_input_no","group":"Crawler policy","label":"Content-Signal says ai-input=no or search=no","count":4,"of":410,"pct":1},{"id":"blocks_any_ai","group":"Crawler policy","label":"robots.txt closes / to at least one of the 12 AI agents","count":97,"of":410,"pct":23.7},{"id":"blocks_GPTBot","group":"Crawler policy","label":"robots.txt closes / to GPTBot","count":63,"of":410,"pct":15.4},{"id":"blocks_OAI-SearchBot","group":"Crawler policy","label":"robots.txt closes / to OAI-SearchBot","count":29,"of":410,"pct":7.1},{"id":"blocks_ChatGPT-User","group":"Crawler policy","label":"robots.txt closes / to ChatGPT-User","count":38,"of":410,"pct":9.3},{"id":"blocks_ClaudeBot","group":"Crawler policy","label":"robots.txt closes / to ClaudeBot","count":59,"of":410,"pct":14.4},{"id":"blocks_Claude-SearchBot","group":"Crawler policy","label":"robots.txt closes / to Claude-SearchBot","count":31,"of":410,"pct":7.6},{"id":"blocks_PerplexityBot","group":"Crawler policy","label":"robots.txt closes / to PerplexityBot","count":45,"of":410,"pct":11},{"id":"blocks_Google-Extended","group":"Crawler policy","label":"robots.txt closes / to Google-Extended","count":51,"of":410,"pct":12.4},{"id":"blocks_Applebot-Extended","group":"Crawler policy","label":"robots.txt closes / to Applebot-Extended","count":48,"of":410,"pct":11.7},{"id":"blocks_CCBot","group":"Crawler policy","label":"robots.txt closes / to CCBot","count":76,"of":410,"pct":18.5},{"id":"blocks_Bytespider","group":"Crawler policy","label":"robots.txt closes / to Bytespider","count":73,"of":410,"pct":17.8},{"id":"blocks_Meta-ExternalAgent","group":"Crawler policy","label":"robots.txt closes / to Meta-ExternalAgent","count":58,"of":410,"pct":14.1},{"id":"blocks_Amazonbot","group":"Crawler policy","label":"robots.txt closes / to Amazonbot","count":53,"of":410,"pct":12.9},{"id":"llms_txt","group":"Machine files","label":"Serves /llms.txt","count":109,"of":513,"pct":21.2},{"id":"agents_md","group":"Machine files","label":"Serves /agents.md","count":14,"of":513,"pct":2.7},{"id":"agent_skills","group":"Capabilities","label":"Publishes an agent-skills index","count":13,"of":513,"pct":2.5},{"id":"mcp_card","group":"Capabilities","label":"Publishes an MCP server card","count":13,"of":513,"pct":2.5},{"id":"api_catalog","group":"Capabilities","label":"Publishes an API catalog (RFC 9727)","count":11,"of":513,"pct":2.1},{"id":"a2a_card","group":"Capabilities","label":"Publishes an A2A agent card","count":3,"of":513,"pct":0.6},{"id":"any_capability","group":"Capabilities","label":"Declares any agent capability (any of the four above)","count":16,"of":513,"pct":3.1},{"id":"ai_refused_any","group":"Delivery parity","label":"At least one AI identity is refused (non-2xx) where the browser got 2xx","count":97,"of":513,"pct":18.9},{"id":"ai_thin_any","group":"Delivery parity","label":"At least one AI identity gets under 70% of the browser's bytes (both 2xx)","count":20,"of":513,"pct":3.9},{"id":"mcp_declared","group":"Endpoint rings","label":"Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one)","count":58,"of":513,"pct":11.3},{"id":"mcp_live","group":"Endpoint rings","label":"An MCP endpoint completes the handshake (initialize and tools/list; no tool called)","count":8,"of":513,"pct":1.6},{"id":"mcp_live_of_declared","group":"Endpoint rings","label":"Of sites naming an MCP endpoint, one completes the handshake or asks for credentials","count":40,"of":58,"pct":69},{"id":"a2a_declared","group":"Endpoint rings","label":"The A2A agent card names an endpoint url","count":1,"of":513,"pct":0.2},{"id":"a2a_answers","group":"Endpoint rings","label":"Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page)","count":0,"of":1,"pct":0},{"id":"webmcp","group":"Endpoint rings","label":"Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block)","count":1,"of":513,"pct":0.2},{"id":"refused_gptbot","group":"Delivery parity","label":"GPTBot refused where the browser got 2xx","count":87,"of":513,"pct":17},{"id":"refused_claudebot","group":"Delivery parity","label":"ClaudeBot refused where the browser got 2xx","count":77,"of":513,"pct":15},{"id":"refused_oaisearch","group":"Delivery parity","label":"OAI-SearchBot refused where the browser got 2xx","count":56,"of":513,"pct":10.9},{"id":"refused_perplexity","group":"Delivery parity","label":"PerplexityBot refused where the browser got 2xx","count":58,"of":513,"pct":11.3}]},"change_basis":"Change compares run 12 with run 3, the first run measured with the current probe. Runs before 3 are excluded from the comparison: a defect in our own probe produced false timeouts in them. points_paired (shown on the page) is like-for-like: (sites gained - sites lost) / sites compared, among sites measured in both runs and inside the figure's denominator both times; points is the plain difference of the two percentages and moves when sites leave a denominator.","first":{"run":3,"started_at":"2026-10-04T04:35:29.982Z","finished_at":"2026-10-04T05:00:02.620Z","sites":991,"figures":[{"id":"reachable","group":"Reach","label":"Homepage answers a browser (2xx after redirects)","count":510,"of":991,"pct":51.5},{"id":"robots_served","group":"Crawler policy","label":"Serves a robots.txt (200, not HTML)","count":403,"of":510,"pct":79},{"id":"robots_sitemap","group":"Crawler policy","label":"robots.txt names a sitemap","count":281,"of":403,"pct":69.7},{"id":"content_signal","group":"Crawler policy","label":"robots.txt carries a Content-Signal line","count":15,"of":403,"pct":3.7},{"id":"cs_ai_train_no","group":"Crawler policy","label":"Content-Signal says ai-train=no","count":9,"of":403,"pct":2.2},{"id":"cs_ai_input_no","group":"Crawler policy","label":"Content-Signal says ai-input=no or search=no","count":4,"of":403,"pct":1},{"id":"blocks_any_ai","group":"Crawler policy","label":"robots.txt closes / to at least one of the 12 AI agents","count":99,"of":403,"pct":24.6},{"id":"blocks_GPTBot","group":"Crawler policy","label":"robots.txt closes / to GPTBot","count":67,"of":403,"pct":16.6},{"id":"blocks_OAI-SearchBot","group":"Crawler policy","label":"robots.txt closes / to OAI-SearchBot","count":33,"of":403,"pct":8.2},{"id":"blocks_ChatGPT-User","group":"Crawler policy","label":"robots.txt closes / to ChatGPT-User","count":42,"of":403,"pct":10.4},{"id":"blocks_ClaudeBot","group":"Crawler policy","label":"robots.txt closes / to ClaudeBot","count":62,"of":403,"pct":15.4},{"id":"blocks_Claude-SearchBot","group":"Crawler policy","label":"robots.txt closes / to Claude-SearchBot","count":34,"of":403,"pct":8.4},{"id":"blocks_PerplexityBot","group":"Crawler policy","label":"robots.txt closes / to PerplexityBot","count":50,"of":403,"pct":12.4},{"id":"blocks_Google-Extended","group":"Crawler policy","label":"robots.txt closes / to Google-Extended","count":56,"of":403,"pct":13.9},{"id":"blocks_Applebot-Extended","group":"Crawler policy","label":"robots.txt closes / to Applebot-Extended","count":48,"of":403,"pct":11.9},{"id":"blocks_CCBot","group":"Crawler policy","label":"robots.txt closes / to CCBot","count":79,"of":403,"pct":19.6},{"id":"blocks_Bytespider","group":"Crawler policy","label":"robots.txt closes / to Bytespider","count":76,"of":403,"pct":18.9},{"id":"blocks_Meta-ExternalAgent","group":"Crawler policy","label":"robots.txt closes / to Meta-ExternalAgent","count":61,"of":403,"pct":15.1},{"id":"blocks_Amazonbot","group":"Crawler policy","label":"robots.txt closes / to Amazonbot","count":52,"of":403,"pct":12.9},{"id":"llms_txt","group":"Machine files","label":"Serves /llms.txt","count":105,"of":510,"pct":20.6},{"id":"agents_md","group":"Machine files","label":"Serves /agents.md","count":14,"of":510,"pct":2.7},{"id":"agent_skills","group":"Capabilities","label":"Publishes an agent-skills index","count":10,"of":510,"pct":2},{"id":"mcp_card","group":"Capabilities","label":"Publishes an MCP server card","count":12,"of":510,"pct":2.4},{"id":"api_catalog","group":"Capabilities","label":"Publishes an API catalog (RFC 9727)","count":11,"of":510,"pct":2.2},{"id":"a2a_card","group":"Capabilities","label":"Publishes an A2A agent card","count":2,"of":510,"pct":0.4},{"id":"any_capability","group":"Capabilities","label":"Declares any agent capability (any of the four above)","count":14,"of":510,"pct":2.7},{"id":"ai_refused_any","group":"Delivery parity","label":"At least one AI identity is refused (non-2xx) where the browser got 2xx","count":87,"of":510,"pct":17.1},{"id":"ai_thin_any","group":"Delivery parity","label":"At least one AI identity gets under 70% of the browser's bytes (both 2xx)","count":19,"of":510,"pct":3.7},{"id":"mcp_declared","group":"Endpoint rings","label":"Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one)","count":0,"of":0,"pct":null},{"id":"mcp_live","group":"Endpoint rings","label":"An MCP endpoint completes the handshake (initialize and tools/list; no tool called)","count":0,"of":0,"pct":null},{"id":"mcp_live_of_declared","group":"Endpoint rings","label":"Of sites naming an MCP endpoint, one completes the handshake or asks for credentials","count":0,"of":0,"pct":null},{"id":"a2a_declared","group":"Endpoint rings","label":"The A2A agent card names an endpoint url","count":0,"of":0,"pct":null},{"id":"a2a_answers","group":"Endpoint rings","label":"Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page)","count":0,"of":0,"pct":null},{"id":"webmcp","group":"Endpoint rings","label":"Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block)","count":0,"of":0,"pct":null},{"id":"refused_gptbot","group":"Delivery parity","label":"GPTBot refused where the browser got 2xx","count":75,"of":510,"pct":14.7},{"id":"refused_claudebot","group":"Delivery parity","label":"ClaudeBot refused where the browser got 2xx","count":71,"of":510,"pct":13.9},{"id":"refused_oaisearch","group":"Delivery parity","label":"OAI-SearchBot refused where the browser got 2xx","count":48,"of":510,"pct":9.4},{"id":"refused_perplexity","group":"Delivery parity","label":"PerplexityBot refused where the browser got 2xx","count":51,"of":510,"pct":10}]},"change":[{"id":"reachable","pct_now":51.8,"pct_first":51.5,"points":0.3,"points_paired":0.3,"sites_gained":15,"sites_lost":12,"compared":991},{"id":"robots_served","pct_now":79.9,"pct_first":79,"points":0.9,"points_paired":0.8,"sites_gained":5,"sites_lost":1,"compared":498},{"id":"robots_sitemap","pct_now":70.5,"pct_first":69.7,"points":0.8,"points_paired":-0.3,"sites_gained":0,"sites_lost":1,"compared":392},{"id":"content_signal","pct_now":3.7,"pct_first":3.7,"points":0,"points_paired":0,"sites_gained":0,"sites_lost":0,"compared":392},{"id":"cs_ai_train_no","pct_now":2.2,"pct_first":2.2,"points":0,"points_paired":0,"sites_gained":0,"sites_lost":0,"compared":392},{"id":"cs_ai_input_no","pct_now":1,"pct_first":1,"points":0,"points_paired":0,"sites_gained":0,"sites_lost":0,"compared":392},{"id":"blocks_any_ai","pct_now":23.7,"pct_first":24.6,"points":-0.9,"points_paired":0,"sites_gained":0,"sites_lost":0,"compared":392},{"id":"blocks_GPTBot","pct_now":15.4,"pct_first":16.6,"points":-1.2,"points_paired":-0.3,"sites_gained":0,"sites_lost":1,"compared":392},{"id":"blocks_OAI-SearchBot","pct_now":7.1,"pct_first":8.2,"points":-1.1,"points_paired":0,"sites_gained":0,"sites_lost":0,"compared":392},{"id":"blocks_ChatGPT-User","pct_now":9.3,"pct_first":10.4,"points":-1.1,"points_paired":0,"sites_gained":0,"sites_lost":0,"compared":392},{"id":"blocks_ClaudeBot","pct_now":14.4,"pct_first":15.4,"points":-1,"points_paired":-0.3,"sites_gained":0,"sites_lost":1,"compared":392},{"id":"blocks_Claude-SearchBot","pct_now":7.6,"pct_first":8.4,"points":-0.8,"points_paired":0,"sites_gained":0,"sites_lost":0,"compared":392},{"id":"blocks_PerplexityBot","pct_now":11,"pct_first":12.4,"points":-1.4,"points_paired":-0.3,"sites_gained":0,"sites_lost":1,"compared":392},{"id":"blocks_Google-Extended","pct_now":12.4,"pct_first":13.9,"points":-1.5,"points_paired":-0.3,"sites_gained":0,"sites_lost":1,"compared":392},{"id":"blocks_Applebot-Extended","pct_now":11.7,"pct_first":11.9,"points":-0.2,"points_paired":-0.3,"sites_gained":0,"sites_lost":1,"compared":392},{"id":"blocks_CCBot","pct_now":18.5,"pct_first":19.6,"points":-1.1,"points_paired":0,"sites_gained":0,"sites_lost":0,"compared":392},{"id":"blocks_Bytespider","pct_now":17.8,"pct_first":18.9,"points":-1.1,"points_paired":0,"sites_gained":0,"sites_lost":0,"compared":392},{"id":"blocks_Meta-ExternalAgent","pct_now":14.1,"pct_first":15.1,"points":-1,"points_paired":0,"sites_gained":0,"sites_lost":0,"compared":392},{"id":"blocks_Amazonbot","pct_now":12.9,"pct_first":12.9,"points":0,"points_paired":-0.3,"sites_gained":0,"sites_lost":1,"compared":392},{"id":"llms_txt","pct_now":21.2,"pct_first":20.6,"points":0.6,"points_paired":0.6,"sites_gained":3,"sites_lost":0,"compared":498},{"id":"agents_md","pct_now":2.7,"pct_first":2.7,"points":0,"points_paired":0,"sites_gained":0,"sites_lost":0,"compared":498},{"id":"agent_skills","pct_now":2.5,"pct_first":2,"points":0.5,"points_paired":0.4,"sites_gained":2,"sites_lost":0,"compared":498},{"id":"mcp_card","pct_now":2.5,"pct_first":2.4,"points":0.1,"points_paired":0.2,"sites_gained":1,"sites_lost":0,"compared":498},{"id":"api_catalog","pct_now":2.1,"pct_first":2.2,"points":-0.1,"points_paired":0,"sites_gained":0,"sites_lost":0,"compared":498},{"id":"a2a_card","pct_now":0.6,"pct_first":0.4,"points":0.2,"points_paired":0.2,"sites_gained":1,"sites_lost":0,"compared":498},{"id":"any_capability","pct_now":3.1,"pct_first":2.7,"points":0.4,"points_paired":0.2,"sites_gained":1,"sites_lost":0,"compared":498},{"id":"ai_refused_any","pct_now":18.9,"pct_first":17.1,"points":1.8,"points_paired":0.4,"sites_gained":5,"sites_lost":3,"compared":498},{"id":"ai_thin_any","pct_now":3.9,"pct_first":3.7,"points":0.2,"points_paired":0,"sites_gained":0,"sites_lost":0,"compared":498},{"id":"mcp_declared","pct_now":11.3,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"mcp_live","pct_now":1.6,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"mcp_live_of_declared","pct_now":69,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"a2a_declared","pct_now":0.2,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"a2a_answers","pct_now":0,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"webmcp","pct_now":0.2,"pct_first":null,"points":null,"points_paired":null,"sites_gained":null,"sites_lost":null,"compared":0},{"id":"refused_gptbot","pct_now":17,"pct_first":14.7,"points":2.3,"points_paired":0.8,"sites_gained":5,"sites_lost":1,"compared":498},{"id":"refused_claudebot","pct_now":15,"pct_first":13.9,"points":1.1,"points_paired":-0.2,"sites_gained":3,"sites_lost":4,"compared":498},{"id":"refused_oaisearch","pct_now":10.9,"pct_first":9.4,"points":1.5,"points_paired":0.2,"sites_gained":2,"sites_lost":1,"compared":498},{"id":"refused_perplexity","pct_now":11.3,"pct_first":10,"points":1.3,"points_paired":0.2,"sites_gained":3,"sites_lost":2,"compared":498}],"method":{"identities":"Homepage fetched once each as a mobile browser and as GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot, from CrawlCheck's network, following redirects. A refusal of a named crawler from this network may be identity verification (the crawler's real addresses are not ours), so delivery figures read 'refused from this vantage', not 'blocked'.","robots":"robots.txt read once; each of 12 AI agents' access to / is decided per RFC 9309 (the most specific matching group; longest-match rules). The text is kept in the record.","files":"A file counts as present when it answers 200, is not an HTML page, is not empty, and (for the JSON ones) parses as a JSON object.","rings":"Endpoint rings (from run derive_v 3): MCP endpoints named by the server card, llms.txt, agents.md or a homepage link, plus GET /mcp on the site and on mcp.<domain> when the answer is the shape a streamable-HTTP server gives (405/406/415, an SSE stream, a JSON-RPC error, a Bearer challenge; never an HTML page or a 404). Up to three candidates get the MCP handshake - initialize and tools/list, never a tool call. The A2A card's url gets one GET. WebMCP is navigator.modelContext, a WebMCP bridge script or an application/mcp+json block in the homepage HTML.","denominators":"Each figure states its own denominator: reach is over every frame member; machine files, capabilities and delivery are over sites a browser reached; robots figures are over sites serving a robots.txt.","recompute":"Every figure is a count over the derived fields of the run's records. Page https://crawlcheck.io/api/census/records?run=N and count (or ask for one site with &domain=); each record also carries the raw readings (statuses, sizes, SHA-256 of what was served, the robots.txt text) the derived fields come from."},"api":{"weekly":"https://crawlcheck.io/api/v1/census/weekly","summary":"https://crawlcheck.io/api/census","records":"https://crawlcheck.io/api/census/records?run={run}&offset=0&limit=200","record":"https://crawlcheck.io/api/census/records?run={run}&domain={domain}","frame":"https://crawlcheck.io/api/census/frame"}}