CrawlCheck
Data · census · weekly

Agentic web census, week 2026-W46

Sealed 2026-11-16. The same frozen sets of the most-visited sites, measured on a schedule from outside: what they tell and serve AI agents this week against last week. No site is named.

top 1,000, daily

Run 45 (finished 2026-11-15), 991 sites, compared with run 38 (2026-11-08).

Biggest moves: Homepage answers CrawlCheck's census fetch (2xx after redirects) -1.9 pt (18 gained, 37 lost); Homepage refuses CrawlCheck's census fetch (400, 401, 403 or 429): blocked or rate-limited, not down +1.4 pt (25 gained, 11 lost); PerplexityBot refused where the browser got 2xx +1 pt (6 gained, 1 lost); At least one AI identity is refused (non-2xx) where the browser got 2xx +0.8 pt (7 gained, 3 lost); At least one AI identity gets under 70% of the browser's bytes (both 2xx) -0.8 pt (0 gained, 4 lost).

FigureThis weekChange, same sitesSites +/−
Homepage answers CrawlCheck's census fetch (2xx after redirects)
of 991
52.3%-1.9 pt+18 / −37
Homepage refuses CrawlCheck's census fetch (400, 401, 403 or 429): blocked or rate-limited, not down
of 991
11.9%+1.4 pt+25 / −11
No website answers at that name (5xx, 530, timeout or no answer)
of 991
31.8%+0.4 pt+9 / −5
Serves a robots.txt (200, not HTML)
of 518
79.2%-0.6 pt+2 / −5
robots.txt names a sitemap
of 410
71.5%0 pt+1 / −1
robots.txt carries a Content-Signal line
of 410
4.4%+0.3 pt+1 / −0
Content-Signal says ai-train=no
of 410
3.2%+0.3 pt+1 / −0
Content-Signal says ai-input=no or search=no
of 410
1%0 pt+0 / −0
robots.txt names the AI-training policy token (__usage_ai-training, IETF AIPREF draft)
of 410
1.2%+0.3 pt+1 / −0
robots.txt states a preference against AI training on / with that token
of 410
1.2%+0.3 pt+1 / −0
robots.txt closes / to at least one of the 12 AI agents
of 410
25.4%0 pt+0 / −0
robots.txt closes / to GPTBot
of 410
16.3%-0.3 pt+0 / −1
robots.txt closes / to OAI-SearchBot
of 410
7.6%-0.3 pt+0 / −1
robots.txt closes / to ChatGPT-User
of 410
9.3%-0.5 pt+0 / −2
robots.txt closes / to ClaudeBot
of 410
16.1%+0.3 pt+1 / −0
robots.txt closes / to Claude-SearchBot
of 410
9.3%0 pt+0 / −0
robots.txt closes / to PerplexityBot
of 410
12.4%0 pt+0 / −0
robots.txt closes / to Google-Extended
of 410
13.9%-0.3 pt+0 / −1
robots.txt closes / to Applebot-Extended
of 410
12.7%+0.3 pt+1 / −0
robots.txt closes / to CCBot
of 410
19.8%0 pt+0 / −0
robots.txt closes / to Bytespider
of 410
19.8%+0.3 pt+1 / −0
robots.txt closes / to Meta-ExternalAgent
of 410
15.6%+0.3 pt+1 / −0
robots.txt closes / to Amazonbot
of 410
13.7%-0.3 pt+0 / −1
Serves /llms.txt
of 518
22%-0.2 pt+1 / −2
Serves /agents.md
of 518
3.1%0 pt+0 / −0
Publishes an agent-skills index
of 518
2.7%+0.2 pt+1 / −0
Publishes an MCP server card
of 518
2.5%+0.2 pt+1 / −0
Publishes an API catalog (RFC 9727)
of 518
2.3%0 pt+0 / −0
Publishes an A2A agent card
of 518
0.6%+0.2 pt+1 / −0
Declares any agent capability (any of the four above)
of 518
3.3%+0.2 pt+1 / −0
At least one AI identity is refused (non-2xx) where the browser got 2xx
of 518
19.1%+0.8 pt+7 / −3
At least one AI identity gets under 70% of the browser's bytes (both 2xx)
of 518
3.7%-0.8 pt+0 / −4
Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one)
of 518
10.8%+0.2 pt+2 / −1
An MCP endpoint completes the handshake (initialize and tools/list; no tool called)
of 518
1.4%-0.2 pt+0 / −1
Of sites naming an MCP endpoint, one completes the handshake or asks for credentials
of 56
76.8%0 pt+0 / −0
The A2A agent card names an endpoint url
of 518
0.4%+0.2 pt+1 / −0
Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page)
of 2
50%0 pt+0 / −0
Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block)
of 518
0.2%0 pt+0 / −0
Listed by Google Safe Browsing (all or some pages): browsers show a red warning
of 314
1%——
Listed as malware or phishing by the Cloudflare security resolver
of 532
0%——
On the browsers' HSTS preload list (HTTPS-only from the first visit)
of 532
17.1%——
Homepage loads scripts or styles from another domain
of 532
62%——
Homepage loads scripts or styles from 5 or more other domains
of 532
6.4%——
Pins at least one outside script or style with integrity=
of 330
9.4%——
Loads a library from a public CDN without integrity=
of 532
9%——
Still loads scripts from a CDN taken over in 2024 (polyfill.io and its sister CDNs)
of 532
0%——
Homepage shows hidden spam links or cryptominer code
of 532
0%——
GPTBot refused where the browser got 2xx
of 518
15.8%+0.4 pt+4 / −2
ClaudeBot refused where the browser got 2xx
of 518
15.6%+0.6 pt+6 / −3
OAI-SearchBot refused where the browser got 2xx
of 518
11.2%+0.2 pt+3 / −2
PerplexityBot refused where the browser got 2xx
of 518
13.3%+1 pt+6 / −1

top 10,000, daily

Run 19 (finished 2026-11-15), 9776 sites, compared with run 12 (2026-11-08).

Biggest moves: PerplexityBot refused where the browser got 2xx +1.2 pt (101 gained, 36 lost); Of sites naming an MCP endpoint, one completes the handshake or asks for credentials -0.6 pt (0 gained, 2 lost); Homepage refuses CrawlCheck's census fetch (400, 401, 403 or 429): blocked or rate-limited, not down +0.2 pt (103 gained, 88 lost); GPTBot refused where the browser got 2xx -0.2 pt (32 gained, 42 lost); OAI-SearchBot refused where the browser got 2xx +0.2 pt (34 gained, 25 lost).

FigureThis weekChange, same sitesSites +/−
Homepage answers CrawlCheck's census fetch (2xx after redirects)
of 9776
58%-0.1 pt+141 / −150
Homepage refuses CrawlCheck's census fetch (400, 401, 403 or 429): blocked or rate-limited, not down
of 9776
13.7%+0.2 pt+103 / −88
No website answers at that name (5xx, 530, timeout or no answer)
of 9776
26.2%-0.1 pt+60 / −67
Serves a robots.txt (200, not HTML)
of 5671
81.7%0 pt+31 / −30
robots.txt names a sitemap
of 4636
70.4%+0.1 pt+9 / −5
robots.txt carries a Content-Signal line
of 4636
6.5%+0.1 pt+7 / −2
Content-Signal says ai-train=no
of 4636
5%+0.1 pt+5 / −2
Content-Signal says ai-input=no or search=no
of 4636
0.5%0 pt+1 / −0
robots.txt names the AI-training policy token (__usage_ai-training, IETF AIPREF draft)
of 4632
0.4%+0.1 pt+4 / −0
robots.txt states a preference against AI training on / with that token
of 4632
0.4%+0.1 pt+4 / −0
robots.txt closes / to at least one of the 12 AI agents
of 4636
21.9%0 pt+5 / −3
robots.txt closes / to GPTBot
of 4636
16.1%-0.1 pt+3 / −6
robots.txt closes / to OAI-SearchBot
of 4636
6%0 pt+2 / −3
robots.txt closes / to ChatGPT-User
of 4636
7.8%0 pt+3 / −4
robots.txt closes / to ClaudeBot
of 4636
14.9%0 pt+4 / −4
robots.txt closes / to Claude-SearchBot
of 4636
5.8%0 pt+3 / −2
robots.txt closes / to PerplexityBot
of 4636
8.8%0 pt+1 / −3
robots.txt closes / to Google-Extended
of 4636
13.3%-0.2 pt+3 / −11
robots.txt closes / to Applebot-Extended
of 4636
13.1%0 pt+6 / −4
robots.txt closes / to CCBot
of 4636
17.1%0 pt+4 / −4
robots.txt closes / to Bytespider
of 4636
17%+0.1 pt+6 / −3
robots.txt closes / to Meta-ExternalAgent
of 4636
13.9%+0.1 pt+7 / −4
robots.txt closes / to Amazonbot
of 4636
13.4%0 pt+5 / −5
Serves /llms.txt
of 5671
17.1%+0.2 pt+14 / −4
Serves /agents.md
of 5671
1.5%0 pt+3 / −2
Publishes an agent-skills index
of 5671
1.1%0 pt+2 / −0
Publishes an MCP server card
of 5671
1.2%+0.1 pt+6 / −0
Publishes an API catalog (RFC 9727)
of 5671
1.3%+0.1 pt+6 / −0
Publishes an A2A agent card
of 5671
0.3%0 pt+2 / −0
Declares any agent capability (any of the four above)
of 5671
1.6%+0.1 pt+6 / −0
At least one AI identity is refused (non-2xx) where the browser got 2xx
of 5671
22.8%-0.1 pt+53 / −61
At least one AI identity gets under 70% of the browser's bytes (both 2xx)
of 5671
2.3%0 pt+13 / −12
Names an MCP endpoint (server card, llms.txt, agents.md, a homepage link, or /mcp answers like one)
of 5671
5.9%+0.1 pt+7 / −2
An MCP endpoint completes the handshake (initialize and tools/list; no tool called)
of 5671
1%0 pt+2 / −2
Of sites naming an MCP endpoint, one completes the handshake or asks for credentials
of 333
78.4%-0.6 pt+0 / −2
The A2A agent card names an endpoint url
of 5671
0.2%0 pt+1 / −0
Of A2A cards naming a url, the url answers (any non-5xx, not an HTML page)
of 9
44.4%0 pt+0 / −0
Homepage carries WebMCP (navigator.modelContext, a WebMCP bridge, or an mcp+json block)
of 5671
0.1%0 pt+0 / −0
Listed by Google Safe Browsing (all or some pages): browsers show a red warning
of 4393
0.9%——
Listed as malware or phishing by the Cloudflare security resolver
of 5687
0.3%——
On the browsers' HSTS preload list (HTTPS-only from the first visit)
of 5690
8%——
Homepage loads scripts or styles from another domain
of 5690
66.9%——
Homepage loads scripts or styles from 5 or more other domains
of 5690
10.8%——
Pins at least one outside script or style with integrity=
of 3808
9.8%——
Loads a library from a public CDN without integrity=
of 5690
13.2%——
Still loads scripts from a CDN taken over in 2024 (polyfill.io and its sister CDNs)
of 5690
0.1%——
Homepage shows hidden spam links or cryptominer code
of 5690
0%——
GPTBot refused where the browser got 2xx
of 5671
19.1%-0.2 pt+32 / −42
ClaudeBot refused where the browser got 2xx
of 5671
19.7%-0.1 pt+49 / −57
OAI-SearchBot refused where the browser got 2xx
of 5671
12.6%+0.2 pt+34 / −25
PerplexityBot refused where the browser got 2xx
of 5671
15.6%+1.2 pt+101 / −36

Each frame compares the last run that finished in this ISO week (UTC) with the last run that finished before it. Change is like-for-like: points = (sites gained - sites lost) / sites compared, counting only sites measured in both runs and inside the figure's denominator both times, so a site that did not answer in one run cannot move a figure. points_all is the plain difference of the two percentages, and denominator_entered / denominator_left count the sites that moved in or out of a figure's base (often a site that did not answer one run). No site is named; per-site records: https://crawlcheck.io/api/census/records.

Signed JSON: /api/v1/census/weekly?week=2026-W46 · feed: Atom · method: /api/census. Editions: 2026-W46 · 2026-W45 · 2026-W44 · 2026-W43 · 2026-W42 · 2026-W41 · 2026-W40.

Check it yourself

Every number here comes from the signed JSON at /api/v1/census/weekly; each copy carries its own sha256 and an Ed25519 signature by key 1hHmE3fo1rg5…. Verify it with no call to us:

node crawlcheck-verify.mjs --online census-weekly.json

The verifier: /crawlcheck-verify.mjs, zero dependencies.

Editions

One sealed edition per ISO week, never edited. Feed: Atom · all instruments: /api/v1/instruments.