Datasets
The web, measured every day
Every site we scan becomes one row: what the homepage is, what it runs, how it treats AI crawlers, and how it scores. Four lanes, filled daily on quota, measured by our own observer — not bought from anyone else.
179,247
measurements in the files so far (one per site per day)
12,539
measured today (2026-10-07, Mountain Time)
44
fields per row
86
technologies fingerprinted
The lanes
| Lane | Daily quota | Today | In the files | Source of the domains |
|---|---|---|---|---|
| SaaS and software, worldwide | 5,000 | 5,015 | 25,047 | Wikidata software companies, the Tranco top million, and links found on software sites |
| Government and education | 2,500 | 2,509 | 12,554 | The CISA .gov list, .edu and foreign government domains, and links between them |
| US local businesses, one state per day | 5,000 | 5,015 | 25,040 | OpenStreetMap businesses, one US state a day (new-york today) |
| Everything else the corpus reached (publishers, shops, directories, docs) | the rest | — | 116,606 | Links discovered by every scan: publishers, shops, directories, docs |
Most common technology, 2026-10-07
| Technology | Sites |
|---|---|
| jQuery | 7,830 |
| Google Fonts | 5,944 |
| Google Analytics | 5,430 |
| Google Tag Manager | 4,693 |
| Font Awesome | 4,590 |
| Bootstrap | 2,439 |
| Yoast SEO | 2,136 |
| Elementor | 1,576 |
| reCAPTCHA | 1,399 |
| Next.js | 1,238 |
| WooCommerce | 1,174 |
| Adobe Fonts | 1,092 |
What a row holds
| Field | Meaning |
|---|---|
| domain | Registrable domain scanned |
| at | When the homepage was read (UTC, ISO 8601) |
| lane | Lane the site was scanned in: saas, gov, local, filler |
| kind | Site kind: the page's own declaration (JSON-LD), else its patterns; for local rows with no page signal, the OpenStreetMap category |
| kind_detected | Kind read from the page alone |
| state | US state (local lane) |
| city | City from OpenStreetMap (local lane) |
| name | Business name from OpenStreetMap (local lane) |
| osm_category | OpenStreetMap category, e.g. trades:plumber (local lane) |
| src | Where the domain came from: dotgov:<type>, tranco:<tld>, wikidata, discovered, OSM category |
| title | Homepage <title> |
| description | Homepage meta description |
| language | en or non-en, from the page |
| grade | CrawlCheck grade, A to F |
| score | CrawlCheck overall score 0-100, before any cap |
| grade_cap_reason | When a finding caps the letter below what the score would give, which finding and why |
| crawl_clarity | Crawl clarity score 0-100 |
| user_clarity | User clarity score 0-100 |
| findings | Number of findings |
| finding_codes | Finding codes, ;-separated |
| platform | CMS band: wordpress, shopify, webflow, squarespace, wix, nextjs, drupal, unknown |
| cms | CMS and builders detected |
| tech | Technologies detected in the homepage: analytics, tag managers, chat, payments, booking, consent, frameworks |
| edge | Edge or CDN detected from headers |
| server | Server header |
| host | First hosting signal |
| rendering | server or client (under 1% of the payload is content) |
| content_ratio | Share of the homepage payload that is visible content, % |
| blocking_resources | Render-blocking resources in <head> |
| id_ratio | Share of JSON-LD nodes with an @id |
| ld_types | schema.org types declared in JSON-LD |
| llms_txt | Serves /llms.txt |
| surfaces | Agent surfaces served: agents.md, agent skills, MCP card, API catalog, media kit |
| robots_measured | robots.txt was read |
| ai_blocked | AI crawlers robots.txt closes / to |
| https | Homepage on HTTPS |
| hsts | HSTS header present |
| sitemap_urls | URLs declared in sitemaps |
| size | small (<50), medium (<2000), large |
| home_bytes | Homepage bytes |
| ttfb_ms | Time to first byte from our observer, ms |
| load_ms | Time to the full homepage from our observer, ms |
| unmeasured | true when the homepage could not be measured |
| reason | Why it could not be measured |
Free sample
25 measured rows from the latest files: CSV · NDJSON. The catalogue with daily counts is /api/datasets.
Delivery
One file per lane per day, CSV or NDJSON, filterable by kind, state, technology and platform: /api/datasets/file?day=YYYY-MM-DD&lane=saas&format=csv&tech=Stripe with your key in x-crawlcheck-key. Files are compacted every hour at :17.
How it is measured
- One observer, one request profile, homepage plus robots.txt, sitemaps and the agent files. Each row says when it was read.
- A site we could not measure is a row marked unmeasured with the reason — never a guess and never a failing grade.
- Kinds come from the page’s own JSON-LD first, then page patterns needing two independent signals. Local rows keep the OpenStreetMap category alongside.
- A site that opts out at /policy is never scanned again and leaves the files.
Pricing
State pack
$149 once
Every measured local business in one US state: technology, CMS, grade, AI-crawler policy, OpenStreetMap category and city. CSV or NDJSON, 30 days to download.
Request accessLane feed
$349 a month
One lane — SaaS, government and education, or local businesses (every state as it rolls) — daily files and all history.
Request accessAll lanes
$899 a month
Every lane including the wider corpus (publishers, shops, directories, docs), daily, all history, every filter.
Request accessEnterprise
From $12,000 a year
Redistribution rights, a custom lane (an industry, a country, a technology), and your own list of domains re-measured weekly.
Ask for a quoteMonthly plans cancel any time from /manage; the key stops at the end of the paid month. Prices are per organisation. Terms: /api-terms.
Enterprise, custom lanes and questions
A quote, a custom lane, or a fuller sample before you buy: tell us what you need and we reply by email.