Agentic web census, week 2026-W45
Sealed 2026-11-09. The same frozen sets of the most-visited sites, measured on a schedule from outside: what they tell and serve AI agents this week against last week. No site is named.
top 1,000, daily
Run 38 (finished 2026-11-08), 991 sites, compared with run 31 (2026-11-01).
Biggest moves: Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page) -50 pt (0 gained, 1 lost); Homepage answers CrawlCheck's census fetch (2xx after redirects) +2.8 pt (38 gained, 10 lost); Homepage refuses CrawlCheck's census fetch (400, 401, 403 or 429): blocked or rate-limited, not down -1.8 pt (5 gained, 23 lost); No website answers at that name (5xx, 530, timeout or no answer) -1.2 pt (3 gained, 15 lost); GPTBot refused where the browser got 2xx -1.2 pt (3 gained, 9 lost).
| Figure | This week | Change, same sites | Sites +/− |
|---|---|---|---|
| Homepage answers CrawlCheck's census fetch (2xx after redirects) of 991 | 54.2% | +2.8 pt | +38 / −10 |
| Homepage refuses CrawlCheck's census fetch (400, 401, 403 or 429): blocked or rate-limited, not down of 991 | 10.5% | -1.8 pt | +5 / −23 |
| No website answers at that name (5xx, 530, timeout or no answer) of 991 | 31.4% | -1.2 pt | +3 / −15 |
| Serves a robots.txt (200, not HTML) of 537 | 77.3% | +1.2 pt | +8 / −2 |
| robots.txt names a sitemap of 415 | 71.3% | +0.3 pt | +1 / −0 |
| robots.txt carries a Content-Signal line of 415 | 4.6% | 0 pt | +0 / −0 |
| Content-Signal says ai-train=no of 415 | 3.1% | 0 pt | +0 / −0 |
| Content-Signal says ai-input=no or search=no of 415 | 1% | 0 pt | +0 / −0 |
| robots.txt names the AI-training policy token (__usage_ai-training, IETF AIPREF draft) of 414 | 1% | — | — |
| robots.txt states a preference against AI training on / with that token of 414 | 1% | — | — |
| robots.txt closes / to at least one of the 12 AI agents of 415 | 24.3% | +0.3 pt | +1 / −0 |
| robots.txt closes / to GPTBot of 415 | 15.4% | 0 pt | +0 / −0 |
| robots.txt closes / to OAI-SearchBot of 415 | 7% | +0.3 pt | +1 / −0 |
| robots.txt closes / to ChatGPT-User of 415 | 8.9% | 0 pt | +0 / −0 |
| robots.txt closes / to ClaudeBot of 415 | 14.7% | 0 pt | +0 / −0 |
| robots.txt closes / to Claude-SearchBot of 415 | 8.2% | +0.3 pt | +1 / −0 |
| robots.txt closes / to PerplexityBot of 415 | 11.1% | +0.3 pt | +1 / −0 |
| robots.txt closes / to Google-Extended of 415 | 13% | 0 pt | +0 / −0 |
| robots.txt closes / to Applebot-Extended of 415 | 11.8% | 0 pt | +0 / −0 |
| robots.txt closes / to CCBot of 415 | 18.8% | 0 pt | +0 / −0 |
| robots.txt closes / to Bytespider of 415 | 18.3% | 0 pt | +0 / −0 |
| robots.txt closes / to Meta-ExternalAgent of 415 | 14.5% | +0.3 pt | +1 / −0 |
| robots.txt closes / to Amazonbot of 415 | 13.3% | 0 pt | +0 / −0 |
| Serves /llms.txt of 537 | 22.9% | +1.2 pt | +7 / −1 |
| Serves /agents.md of 537 | 3% | +0.6 pt | +3 / −0 |
| Publishes an agent-skills index of 537 | 2.6% | 0 pt | +0 / −0 |
| Publishes an MCP server card of 537 | 2.4% | 0 pt | +0 / −0 |
| Publishes an API catalog (RFC 9727) of 537 | 2.4% | +0.2 pt | +2 / −1 |
| Publishes an A2A agent card of 537 | 0.6% | 0 pt | +0 / −0 |
| Declares any agent capability (any of the four above) of 537 | 3.2% | -0.2 pt | +0 / −1 |
| At least one AI identity is refused (non-2xx) where the browser got 2xx of 537 | 20.9% | -0.6 pt | +7 / −10 |
| At least one AI identity gets under 70% of the browser's bytes (both 2xx) of 537 | 3.7% | 0 pt | +3 / −3 |
| Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one) of 537 | 10.8% | +0.6 pt | +3 / −0 |
| An MCP endpoint completes the handshake (initialize and tools/list; no tool called) of 537 | 1.7% | 0 pt | +1 / −1 |
| Of sites naming an MCP endpoint, one completes the handshake or asks for credentials of 58 | 79.3% | 0 pt | +0 / −0 |
| The A2A agent card names an endpoint url of 537 | 0.4% | 0 pt | +0 / −0 |
| Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page) of 2 | 0% | -50 pt | +0 / −1 |
| Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block) of 537 | 0.2% | 0 pt | +0 / −0 |
| Listed by Google Safe Browsing (all or some pages): browsers show a red warning of 357 | 0.8% | — | — |
| Listed as malware or phishing by the Cloudflare security resolver of 550 | 0% | — | — |
| On the browsers' HSTS preload list (HTTPS-only from the first visit) of 550 | 16.5% | — | — |
| Homepage loads scripts or styles from another domain of 550 | 63.3% | — | — |
| Homepage loads scripts or styles from 5 or more other domains of 550 | 5.6% | — | — |
| Pins at least one outside script or style with integrity= of 348 | 9.8% | — | — |
| Loads a library from a public CDN without integrity= of 550 | 8.5% | — | — |
| Still loads scripts from a CDN taken over in 2024 (polyfill.io and its sister CDNs) of 550 | 0% | — | — |
| Homepage shows hidden spam links or cryptominer code of 550 | 0% | — | — |
| GPTBot refused where the browser got 2xx of 537 | 18.1% | -1.2 pt | +3 / −9 |
| ClaudeBot refused where the browser got 2xx of 537 | 17.7% | -0.2 pt | +5 / −6 |
| OAI-SearchBot refused where the browser got 2xx of 537 | 13.8% | 0 pt | +5 / −5 |
| PerplexityBot refused where the browser got 2xx of 537 | 15.3% | +0.6 pt | +8 / −5 |
top 10,000, daily
Run 12 (finished 2026-11-08), 9776 sites, compared with run 5 (2026-10-31).
Biggest moves: PerplexityBot refused where the browser got 2xx +1.6 pt (113 gained, 27 lost); At least one AI identity gets under 70% of the browser's bytes (both 2xx) -0.8 pt (7 gained, 51 lost); Homepage answers CrawlCheck's census fetch (2xx after redirects) +0.6 pt (191 gained, 132 lost); No website answers at that name (5xx, 530, timeout or no answer) -0.6 pt (71 gained, 125 lost); Serves /llms.txt +0.5 pt (32 gained, 7 lost).
| Figure | This week | Change, same sites | Sites +/− |
|---|---|---|---|
| Homepage answers CrawlCheck's census fetch (2xx after redirects) of 9776 | 58.1% | +0.6 pt | +191 / −132 |
| Homepage refuses CrawlCheck's census fetch (400, 401, 403 or 429): blocked or rate-limited, not down of 9776 | 13.5% | -0.1 pt | +77 / −91 |
| No website answers at that name (5xx, 530, timeout or no answer) of 9776 | 26.3% | -0.6 pt | +71 / −125 |
| Serves a robots.txt (200, not HTML) of 5680 | 81.9% | +0.1 pt | +36 / −31 |
| robots.txt names a sitemap of 4651 | 70.2% | +0.3 pt | +13 / −0 |
| robots.txt carries a Content-Signal line of 4651 | 6.3% | +0.2 pt | +10 / −2 |
| Content-Signal says ai-train=no of 4651 | 4.9% | +0.1 pt | +6 / −3 |
| Content-Signal says ai-input=no or search=no of 4651 | 0.5% | 0 pt | +0 / −1 |
| robots.txt names the AI-training policy token (__usage_ai-training, IETF AIPREF draft) of 4647 | 0.3% | — | — |
| robots.txt states a preference against AI training on / with that token of 4647 | 0.3% | — | — |
| robots.txt closes / to at least one of the 12 AI agents of 4651 | 22% | +0.1 pt | +10 / −5 |
| robots.txt closes / to GPTBot of 4651 | 16.3% | +0.1 pt | +9 / −3 |
| robots.txt closes / to OAI-SearchBot of 4651 | 6.1% | +0.1 pt | +5 / −2 |
| robots.txt closes / to ChatGPT-User of 4651 | 8% | 0 pt | +2 / −3 |
| robots.txt closes / to ClaudeBot of 4651 | 15% | +0.1 pt | +8 / −5 |
| robots.txt closes / to Claude-SearchBot of 4651 | 5.7% | +0.1 pt | +5 / −2 |
| robots.txt closes / to PerplexityBot of 4651 | 8.9% | -0.2 pt | +3 / −14 |
| robots.txt closes / to Google-Extended of 4651 | 13.4% | +0.1 pt | +9 / −5 |
| robots.txt closes / to Applebot-Extended of 4651 | 13.1% | +0.2 pt | +12 / −4 |
| robots.txt closes / to CCBot of 4651 | 17.1% | 0 pt | +7 / −8 |
| robots.txt closes / to Bytespider of 4651 | 17% | +0.1 pt | +8 / −4 |
| robots.txt closes / to Meta-ExternalAgent of 4651 | 13.9% | +0.2 pt | +13 / −4 |
| robots.txt closes / to Amazonbot of 4651 | 13.4% | +0.2 pt | +11 / −3 |
| Serves /llms.txt of 5680 | 17% | +0.5 pt | +32 / −7 |
| Serves /agents.md of 5680 | 1.5% | +0.1 pt | +7 / −0 |
| Publishes an agent-skills index of 5680 | 1% | 0 pt | +2 / −0 |
| Publishes an MCP server card of 5680 | 1.1% | +0.1 pt | +5 / −0 |
| Publishes an API catalog (RFC 9727) of 5680 | 1.2% | +0.1 pt | +5 / −0 |
| Publishes an A2A agent card of 5680 | 0.3% | 0 pt | +2 / −0 |
| Declares any agent capability (any of the four above) of 5680 | 1.4% | +0.1 pt | +4 / −0 |
| At least one AI identity is refused (non-2xx) where the browser got 2xx of 5680 | 23% | -0.4 pt | +59 / −79 |
| At least one AI identity gets under 70% of the browser's bytes (both 2xx) of 5680 | 2.3% | -0.8 pt | +7 / −51 |
| Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one) of 5680 | 5.8% | +0.1 pt | +10 / −2 |
| An MCP endpoint completes the handshake (initialize and tools/list; no tool called) of 5680 | 0.9% | 0 pt | +3 / −1 |
| Of sites naming an MCP endpoint, one completes the handshake or asks for credentials of 327 | 78.9% | 0 pt | +1 / −1 |
| The A2A agent card names an endpoint url of 5680 | 0.1% | 0 pt | +1 / −0 |
| Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page) of 8 | 37.5% | 0 pt | +0 / −0 |
| Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block) of 5680 | 0.1% | 0 pt | +0 / −0 |
| Listed by Google Safe Browsing (all or some pages): browsers show a red warning of 4614 | 0.9% | — | — |
| Listed as malware or phishing by the Cloudflare security resolver of 5698 | 0.3% | — | — |
| On the browsers' HSTS preload list (HTTPS-only from the first visit) of 5698 | 8% | — | — |
| Homepage loads scripts or styles from another domain of 5698 | 67.5% | — | — |
| Homepage loads scripts or styles from 5 or more other domains of 5698 | 10.9% | — | — |
| Pins at least one outside script or style with integrity= of 3845 | 10.1% | — | — |
| Loads a library from a public CDN without integrity= of 5698 | 13.3% | — | — |
| Still loads scripts from a CDN taken over in 2024 (polyfill.io and its sister CDNs) of 5698 | 0.1% | — | — |
| Homepage shows hidden spam links or cryptominer code of 5698 | 0% | — | — |
| GPTBot refused where the browser got 2xx of 5680 | 19.2% | 0 pt | +37 / −38 |
| ClaudeBot refused where the browser got 2xx of 5680 | 20% | 0 pt | +52 / −54 |
| OAI-SearchBot refused where the browser got 2xx of 5680 | 12.4% | -0.4 pt | +25 / −46 |
| PerplexityBot refused where the browser got 2xx of 5680 | 14.5% | +1.6 pt | +113 / −27 |
Each frame compares the last run that finished in this ISO week (UTC) with the last run that finished before it. Change is like-for-like: points = (sites gained - sites lost) / sites compared, counting only sites measured in both runs and inside the figure's denominator both times, so a site that did not answer in one run cannot move a figure. points_all is the plain difference of the two percentages, and denominator_entered / denominator_left count the sites that moved in or out of a figure's base (often a site that did not answer one run). No site is named; per-site records: https://crawlcheck.io/api/census/records.
Signed JSON: /api/v1/census/weekly?week=2026-W45 · feed: Atom · method: /api/census. Editions: 2026-W46 · 2026-W45 · 2026-W44 · 2026-W43 · 2026-W42 · 2026-W41 · 2026-W40.
Check it yourself
Every number here comes from the signed JSON at /api/v1/census/weekly; each copy carries its own sha256 and an Ed25519 signature by key 1hHmE3fo1rg5…. Verify it with no call to us:
node crawlcheck-verify.mjs --online census-weekly.json
The verifier: /crawlcheck-verify.mjs, zero dependencies.