Agentic web census, week 2026-W42
Sealed 2026-10-20. The same frozen sets of the most-visited sites, measured on a schedule from outside: what they tell and serve AI agents this week against last week. No site is named.
top 1,000, daily
Run 17 (finished 2026-10-18), 991 sites, compared with run 10 (2026-10-11).
Biggest moves: Homepage answers a browser (2xx after redirects) +2.3 pt (40 gained, 17 lost); PerplexityBot refused where the browser got 2xx +1 pt (5 gained, 0 lost); ClaudeBot refused where the browser got 2xx +0.8 pt (8 gained, 4 lost); Serves /llms.txt +0.8 pt (5 gained, 1 lost); At least one AI identity is refused (non-2xx) where the browser got 2xx +0.6 pt (7 gained, 4 lost).
| Figure | This week | Change, same sites | Sites +/− |
|---|---|---|---|
| Homepage answers a browser (2xx after redirects) of 991 | 54.3% | +2.3 pt | +40 / −17 |
| Serves a robots.txt (200, not HTML) of 538 | 77.7% | -0.2 pt | +0 / −1 |
| robots.txt names a sitemap of 418 | 70.8% | +0.5 pt | +2 / −0 |
| robots.txt carries a Content-Signal line of 418 | 3.8% | +0.3 pt | +1 / −0 |
| Content-Signal says ai-train=no of 418 | 2.4% | +0.3 pt | +1 / −0 |
| Content-Signal says ai-input=no or search=no of 418 | 1% | 0 pt | +0 / −0 |
| robots.txt closes / to at least one of the 12 AI agents of 418 | 23.2% | +0.3 pt | +1 / −0 |
| robots.txt closes / to GPTBot of 418 | 14.8% | +0.3 pt | +1 / −0 |
| robots.txt closes / to OAI-SearchBot of 418 | 7.2% | 0 pt | +0 / −0 |
| robots.txt closes / to ChatGPT-User of 418 | 9.3% | 0 pt | +0 / −0 |
| robots.txt closes / to ClaudeBot of 418 | 14.6% | +0.3 pt | +1 / −0 |
| robots.txt closes / to Claude-SearchBot of 418 | 7.7% | 0 pt | +0 / −0 |
| robots.txt closes / to PerplexityBot of 418 | 11.2% | 0 pt | +0 / −0 |
| robots.txt closes / to Google-Extended of 418 | 12.2% | +0.3 pt | +2 / −1 |
| robots.txt closes / to Applebot-Extended of 418 | 11.2% | 0 pt | +0 / −0 |
| robots.txt closes / to CCBot of 418 | 18.4% | +0.3 pt | +1 / −0 |
| robots.txt closes / to Bytespider of 418 | 17.9% | +0.3 pt | +1 / −0 |
| robots.txt closes / to Meta-ExternalAgent of 418 | 14.1% | 0 pt | +0 / −0 |
| robots.txt closes / to Amazonbot of 418 | 13.2% | +0.3 pt | +1 / −0 |
| Serves /llms.txt of 538 | 21.2% | +0.8 pt | +5 / −1 |
| Serves /agents.md of 538 | 2.6% | 0 pt | +0 / −0 |
| Publishes an agent-skills index of 538 | 2.4% | +0.2 pt | +1 / −0 |
| Publishes an MCP server card of 538 | 2.4% | +0.2 pt | +1 / −0 |
| Publishes an API catalog (RFC 9727) of 538 | 2.2% | +0.2 pt | +1 / −0 |
| Publishes an A2A agent card of 538 | 0.6% | +0.2 pt | +1 / −0 |
| Declares any agent capability (any of the four above) of 538 | 3% | +0.2 pt | +1 / −0 |
| At least one AI identity is refused (non-2xx) where the browser got 2xx of 538 | 22.3% | +0.6 pt | +7 / −4 |
| At least one AI identity gets under 70% of the browser's bytes (both 2xx) of 538 | 4.5% | +0.4 pt | +3 / −1 |
| Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one) of 538 | 10.2% | +0.4 pt | +3 / −1 |
| An MCP endpoint completes the handshake (initialize and tools/list; no tool called) of 538 | 1.7% | +0.2 pt | +1 / −0 |
| Of sites naming an MCP endpoint, one completes the handshake or asks for credentials of 55 | 72.7% | 0 pt | +0 / −0 |
| The A2A agent card names an endpoint url of 538 | 0.2% | 0 pt | +0 / −0 |
| Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page) of 1 | 0% | 0 pt | +0 / −0 |
| Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block) of 538 | 0.2% | 0 pt | +0 / −0 |
| GPTBot refused where the browser got 2xx of 538 | 19.1% | 0 pt | +2 / −2 |
| ClaudeBot refused where the browser got 2xx of 538 | 18.6% | +0.8 pt | +8 / −4 |
| OAI-SearchBot refused where the browser got 2xx of 538 | 13.9% | -0.2 pt | +1 / −2 |
| PerplexityBot refused where the browser got 2xx of 538 | 15.2% | +1 pt | +5 / −0 |
top 10,000, weekly
Run 3 (finished 2026-10-17), 9776 sites, compared with run 2 (2026-10-10).
Biggest moves: Homepage answers a browser (2xx after redirects) +2.4 pt (425 gained, 195 lost); robots.txt closes / to Amazonbot +0.4 pt (22 gained, 5 lost); robots.txt closes / to Applebot-Extended +0.4 pt (21 gained, 5 lost); robots.txt closes / to Meta-ExternalAgent +0.4 pt (21 gained, 4 lost); Of sites naming an MCP endpoint, one completes the handshake or asks for credentials +0.4 pt (1 gained, 0 lost).
| Figure | This week | Change, same sites | Sites +/− |
|---|---|---|---|
| Homepage answers a browser (2xx after redirects) of 9776 | 59.2% | +2.4 pt | +425 / −195 |
| Serves a robots.txt (200, not HTML) of 5789 | 79.3% | -0.1 pt | +34 / −37 |
| robots.txt names a sitemap of 4588 | 69.8% | +0.2 pt | +13 / −3 |
| robots.txt carries a Content-Signal line of 4588 | 3.5% | +0.2 pt | +9 / −2 |
| Content-Signal says ai-train=no of 4588 | 2.2% | +0.1 pt | +8 / −2 |
| Content-Signal says ai-input=no or search=no of 4588 | 0.5% | 0 pt | +0 / −0 |
| robots.txt closes / to at least one of the 12 AI agents of 4588 | 19.6% | +0.2 pt | +14 / −5 |
| robots.txt closes / to GPTBot of 4588 | 13.8% | +0.1 pt | +10 / −5 |
| robots.txt closes / to OAI-SearchBot of 4588 | 6.4% | 0 pt | +6 / −4 |
| robots.txt closes / to ChatGPT-User of 4588 | 8.1% | -0.2 pt | +4 / −11 |
| robots.txt closes / to ClaudeBot of 4588 | 12.5% | +0.1 pt | +10 / −5 |
| robots.txt closes / to Claude-SearchBot of 4588 | 5.9% | +0.1 pt | +5 / −1 |
| robots.txt closes / to PerplexityBot of 4588 | 9.2% | 0 pt | +6 / −4 |
| robots.txt closes / to Google-Extended of 4588 | 10.8% | +0.3 pt | +20 / −6 |
| robots.txt closes / to Applebot-Extended of 4588 | 10.4% | +0.4 pt | +21 / −5 |
| robots.txt closes / to CCBot of 4588 | 14.6% | +0.2 pt | +11 / −4 |
| robots.txt closes / to Bytespider of 4588 | 14.2% | +0.2 pt | +11 / −4 |
| robots.txt closes / to Meta-ExternalAgent of 4588 | 11.2% | +0.4 pt | +21 / −4 |
| robots.txt closes / to Amazonbot of 4588 | 10.7% | +0.4 pt | +22 / −5 |
| Serves /llms.txt of 5789 | 15.6% | +0.2 pt | +21 / −8 |
| Serves /agents.md of 5789 | 1.3% | 0 pt | +4 / −3 |
| Publishes an agent-skills index of 5789 | 0.9% | 0 pt | +1 / −1 |
| Publishes an MCP server card of 5789 | 0.9% | 0 pt | +2 / −1 |
| Publishes an API catalog (RFC 9727) of 5789 | 1.1% | 0 pt | +1 / −1 |
| Publishes an A2A agent card of 5789 | 0.3% | 0 pt | +0 / −2 |
| Declares any agent capability (any of the four above) of 5789 | 1.3% | 0 pt | +3 / −1 |
| At least one AI identity is refused (non-2xx) where the browser got 2xx of 5789 | 25.3% | +0.1 pt | +73 / −70 |
| At least one AI identity gets under 70% of the browser's bytes (both 2xx) of 5789 | 2.5% | 0 pt | +14 / −15 |
| Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one) of 5789 | 5.3% | +0.1 pt | +9 / −3 |
| An MCP endpoint completes the handshake (initialize and tools/list; no tool called) of 5789 | 0.9% | 0 pt | +2 / −0 |
| Of sites naming an MCP endpoint, one completes the handshake or asks for credentials of 305 | 77.7% | +0.4 pt | +1 / −0 |
| The A2A agent card names an endpoint url of 5789 | 0.1% | 0 pt | +0 / −0 |
| Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page) of 8 | 37.5% | 0 pt | +0 / −0 |
| Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block) of 5789 | 0.1% | 0 pt | +0 / −0 |
| GPTBot refused where the browser got 2xx of 5789 | 21.7% | 0 pt | +39 / −39 |
| ClaudeBot refused where the browser got 2xx of 5789 | 22.2% | +0.2 pt | +54 / −45 |
| OAI-SearchBot refused where the browser got 2xx of 5789 | 14.9% | -0.1 pt | +33 / −37 |
| PerplexityBot refused where the browser got 2xx of 5789 | 15.4% | -0.1 pt | +38 / −41 |
Each frame compares the last run that finished in this ISO week (UTC) with the last run that finished before it. Change is like-for-like: points = (sites gained - sites lost) / sites compared, counting only sites measured in both runs and inside the figure's denominator both times, so a site that did not answer in one run cannot move a figure. points_all is the plain difference of the two percentages, and denominator_entered / denominator_left count the sites that moved in or out of a figure's base (often a site that did not answer one run). No site is named; per-site records: https://crawlcheck.io/api/census/records.
Signed JSON: /api/v1/census/weekly?week=2026-W42 · feed: Atom · method: /api/census. Editions: 2026-W46 · 2026-W45 · 2026-W44 · 2026-W43 · 2026-W42 · 2026-W41 · 2026-W40.
Check it yourself
Every number here comes from the signed JSON at /api/v1/census/weekly; each copy carries its own sha256 and an Ed25519 signature by key 1hHmE3fo1rg5…. Verify it with no call to us:
node crawlcheck-verify.mjs --online census-weekly.json
The verifier: /crawlcheck-verify.mjs, zero dependencies.