Agentic web census, week 2026-W42
Week to date; sealed after Sunday, midnight UTC. The same frozen sets of the most-visited sites, measured on a schedule from outside: what they tell and serve AI agents this week against last week. No site is named.
top 1,000, daily
Run 12 (finished 2026-10-13), 991 sites, compared with run 10 (2026-10-11).
Biggest moves: Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one) +0.4 pt (2 gained, 0 lost); Homepage answers a browser (2xx after redirects) -0.2 pt (5 gained, 7 lost); ClaudeBot refused where the browser got 2xx +0.2 pt (4 gained, 3 lost); GPTBot refused where the browser got 2xx +0.2 pt (3 gained, 2 lost); At least one AI identity gets under 70% of the browser's bytes (both 2xx) -0.2 pt (1 gained, 2 lost).
| Figure | This week | Change, same sites | Sites +/− |
|---|---|---|---|
| Homepage answers a browser (2xx after redirects) of 991 | 51.8% | -0.2 pt | +5 / −7 |
| Serves a robots.txt (200, not HTML) of 513 | 79.9% | -0.2 pt | +0 / −1 |
| robots.txt names a sitemap of 410 | 70.5% | 0 pt | +0 / −0 |
| robots.txt carries a Content-Signal line of 410 | 3.7% | 0 pt | +0 / −0 |
| Content-Signal says ai-train=no of 410 | 2.2% | 0 pt | +0 / −0 |
| Content-Signal says ai-input=no or search=no of 410 | 1% | 0 pt | +0 / −0 |
| robots.txt closes / to at least one of the 12 AI agents of 410 | 23.7% | 0 pt | +0 / −0 |
| robots.txt closes / to GPTBot of 410 | 15.4% | 0 pt | +0 / −0 |
| robots.txt closes / to OAI-SearchBot of 410 | 7.1% | 0 pt | +0 / −0 |
| robots.txt closes / to ChatGPT-User of 410 | 9.3% | 0 pt | +0 / −0 |
| robots.txt closes / to ClaudeBot of 410 | 14.4% | 0 pt | +0 / −0 |
| robots.txt closes / to Claude-SearchBot of 410 | 7.6% | 0 pt | +0 / −0 |
| robots.txt closes / to PerplexityBot of 410 | 11% | 0 pt | +0 / −0 |
| robots.txt closes / to Google-Extended of 410 | 12.4% | 0 pt | +0 / −0 |
| robots.txt closes / to Applebot-Extended of 410 | 11.7% | 0 pt | +0 / −0 |
| robots.txt closes / to CCBot of 410 | 18.5% | 0 pt | +0 / −0 |
| robots.txt closes / to Bytespider of 410 | 17.8% | 0 pt | +0 / −0 |
| robots.txt closes / to Meta-ExternalAgent of 410 | 14.1% | 0 pt | +0 / −0 |
| robots.txt closes / to Amazonbot of 410 | 12.9% | 0 pt | +0 / −0 |
| Serves /llms.txt of 513 | 21.2% | +0.2 pt | +1 / −0 |
| Serves /agents.md of 513 | 2.7% | 0 pt | +0 / −0 |
| Publishes an agent-skills index of 513 | 2.5% | +0.2 pt | +1 / −0 |
| Publishes an MCP server card of 513 | 2.5% | +0.2 pt | +1 / −0 |
| Publishes an API catalog (RFC 9727) of 513 | 2.1% | 0 pt | +0 / −0 |
| Publishes an A2A agent card of 513 | 0.6% | +0.2 pt | +1 / −0 |
| Declares any agent capability (any of the four above) of 513 | 3.1% | +0.2 pt | +1 / −0 |
| At least one AI identity is refused (non-2xx) where the browser got 2xx of 513 | 18.9% | 0 pt | +4 / −4 |
| At least one AI identity gets under 70% of the browser's bytes (both 2xx) of 513 | 3.9% | -0.2 pt | +1 / −2 |
| Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one) of 513 | 11.3% | +0.4 pt | +2 / −0 |
| An MCP endpoint completes the handshake (initialize and tools/list; no tool called) of 513 | 1.6% | +0.2 pt | +1 / −0 |
| Of sites naming an MCP endpoint, one completes the handshake or asks for credentials of 58 | 69% | 0 pt | +0 / −0 |
| The A2A agent card names an endpoint url of 513 | 0.2% | 0 pt | +0 / −0 |
| Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page) of 1 | 0% | 0 pt | +0 / −0 |
| Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block) of 513 | 0.2% | 0 pt | +0 / −0 |
| GPTBot refused where the browser got 2xx of 513 | 17% | +0.2 pt | +3 / −2 |
| ClaudeBot refused where the browser got 2xx of 513 | 15% | +0.2 pt | +4 / −3 |
| OAI-SearchBot refused where the browser got 2xx of 513 | 10.9% | +0.2 pt | +1 / −0 |
| PerplexityBot refused where the browser got 2xx of 513 | 11.3% | +0.2 pt | +1 / −0 |
top 10,000, weekly
no run of this frame finished in the week.
Each frame compares the last run that finished in this ISO week (UTC) with the last run that finished before it. Change is like-for-like: points = (sites gained - sites lost) / sites compared, counting only sites measured in both runs and inside the figure's denominator both times, so a site that did not answer in one run cannot move a figure. points_all is the plain difference of the two percentages, and denominator_entered / denominator_left count the sites that moved in or out of a figure's base (often a site that did not answer one run). No site is named; per-site records: https://crawlcheck.io/api/census/records.
Signed JSON: /api/v1/census/weekly?week=2026-W42 · feed: Atom · method: /api/census. Editions: 2026-W41 · 2026-W40.