Local businesses in your state, each with the website problem to open with
One file per US state: every local business we measured there, with its name, city and category, the phone and email its own website publishes, and the problems we found on that website. $149 a state, 30 days of downloads.
Phone and email were added on 14 November 2026. Each state is being read again to add them, one a day from 14 November: Colorado, Texas, California, Florida, New York, then the rest in the order they were first read. A state not yet read again has no contacts in its rows. A row never carries a contact the site does not publish itself.
Also sold
US local businesses · one state a day
- 782,981 distinct sites, 557,520 measured on their newest read (median 23.6 days old)
- 724,183 measurements, one per site per day
- 221,657 US local businesses with city and category
- 86 technologies fingerprinted per site
- 10,896 measured today, 2026-11-21 Mountain Time
Release ds-20261122-0120-e3b71a24 · counts are measured reads only; 297,456 reads that got no answer are left out. The same release id is in /api/datasets.
One state, or one problem.
One row per site per day, 79 fields a row, measured by us — never bought from a data broker. Counts are sites in the files today. Checkout is by Stripe; the key and the first download are on the next page.
| Dataset · what you get | Sites | Updated | Price | |
|---|---|---|---|---|
| One US state, local businessesEvery measured local business in one state: city, category, platform, grade, technologies, AI-crawler policy. 30 days of downloads. | 221,657all states so far | 2026-11-21 | $149once | Pick a state |
| Websites by problem52 lists: every site with a broken robots.txt, a sitemap search engines cannot read, no HSTS and more — businesses that need the fix you sell. | 60lists | 2026-11-22 | from $99once, by size | Pick a problem |
Everything else we sell: ready-made lists, technology lists, daily feeds, licences
| Dataset · what you get | Sites | Updated | Price | |
|---|---|---|---|---|
| Trades and home servicesPlumbers, electricians, roofers and trades, each with its city, platform, grade and the defect to open with. | 2,674 | 2026-11-22 | $299once | Buy the list |
| Trades and home services, one stateThe same list for the state you sell into: every trade or home-service business we measured there, with the same columns. 30 days of downloads. | 2,553in 7 states | 2026-11-22 | $99once | |
| Online storesEvery site that sells online, with its platform (Shopify, WooCommerce, BigCommerce), grade and AI-crawler policy. | 18,152 | 2026-11-22 | $299once | Buy the list |
| Publishers and news sitesNews and article sites with how each treats AI crawlers: licensing, ad tech and PR targets. | 13,193 | 2026-11-22 | $199once | Buy the list |
| Sites that block AI crawlersSites whose robots.txt shuts out a named AI agent (GPTBot, ClaudeBot, CCBot), with which ones. | 9,668 | 2026-11-22 | $149once | Buy the list |
| Sites with an llms.txtEvery site that publishes an llms.txt, by sector and platform — the early adopters. | 25,103 | 2026-11-22 | $149once | Buy the list |
| Domains open to email spoofingDomains that take email but have no enforcing DMARC policy, each with its mail setup, SPF and the platform it runs. | 69,079 | 2026-11-22 | $249once | Buy the list |
| Sites missing basic securitySites missing basic protections: which headers are absent, exposed server versions, outdated jQuery, insecure content. | 96,940 | 2026-11-22 | $199once | Buy the list |
| Domains expiring in the next 45 daysDomains whose registration expires within 45 days, with registrar, expiry date, DNS provider and hosting network. | 6,281 | 2026-11-22 | $149once | Buy the list |
| Domains registered in the last 6 monthsDomains registered in the last 6 months that already serve a live site, with registration date, registrar and stack. | 2,433 | 2026-11-22 | $149once | Buy the list |
| Sites running a marketing or support platformWho runs HubSpot, Klaviyo, Intercom, Zendesk and 12 more — a displacement list for competing vendors. | 6,293 | 2026-11-22 | $249once | Buy the list |
| Websites by technology86 lists: every site running HubSpot, Shopify, Stripe, Klaviyo, Intercom and 81 more, with platform, grade and AI-crawler policy. | 86lists | 2026-11-22 | from $99once, by size | Pick a technology |
| One lane, every daySaaS and software (119,583 sites), government and education (67,927), or US local as the states roll. Daily files and the whole history of the lane; the lists cut from your lane. | 409,167 | 2026-11-21 | $349a month | Buy a lane |
| All lanesEverything: the four lanes including the wider corpus (publishers, shops, directories, docs), every list in full, all history, every filter. | 724,183measurements, one per site per day | 2026-11-21 | $899a month or $8,990 a year | Buy all lanes |
| CreditsPrepaid, 1 credit = $1, spent on any list, technology list, state or lane. Never expire. $25 buys 30; $100 buys 150. | — | — | from $10once | Buy credits |
| AI consent licenceFor AI companies and researchers: what each site’s robots.txt tells 16 named AI agents, its Content-Signal line and llms.txt, with the read date. Signed releases, free sample. | 124,677 | 2026-11-21 | from $25,000a year | See the licence |
| The snapshot archiveEvery signed Resolve answer behind the public window, kept 90 days: the same files, listed and served on your key, so a change on any measured site can be placed to the day. | 99,784answers a day | 2026-11-21 | $199a month or $1,990 a year | Get the archive |
| EnterpriseRedistribution rights, a custom lane (an industry, a country, a technology), your own domains re-measured weekly. A contract and an invoice. | — | — | From $25,000a year | Ask for a quote |
A state pack and a list are one payment with 30 days of downloads; lanes are monthly and cancel any time from /manage. Prices are per organisation. 25 free rows · the catalogue as JSON · terms.
Every free dataset, in one place.
The current edition of everything CrawlCheck measures is free to reuse, commercially included: credit “CrawlCheck” with the URL of the file. What is sold above is depth: row-level files, lists cut to a buyer, and the archive.
For AI teams: every record is signed and each day is anchored in Bitcoin, so a value cannot be changed after the date it carries. That makes these files checkable ground truth for evaluating agents, or scoring them in training. The agent safety benchmark is built from them.
| Dataset · what it holds | Updated | Size | Get it |
|---|---|---|---|
| The Resolve snapshotEvery public signed answer about a measured site, one file set a day; each line verifies on its own with no call to us | daily, newest 7 public | 99,784answers in the newest (2026-11-21) | Index · File layout |
| Agentic-web censusThe most-visited domains on a fixed frame: crawler policy, machine files, declared capabilities, and whether AI crawlers get what a browser gets; every site named | daily and weekly | — | Daily records · Weekly · Atom |
| The instrumentsDecisions to outcomes, Metrics, Agent safety benchmark, Agent Traffic Index, Assurance, Status, Agent conduct, Proof: the Resolve log, Agentic web census, weekly, Agentic web map, Share of preflight, Vendor website benchmark, Registry: what sites declare to agents: each a signed JSON twin with one sealed edition per ISO week | signed hourly, sealed weekly | 13instruments | Index · The pages |
| Finding frequenciesEvery scan run here, counted by finding: how common, how serious; names no domain | continuous | — | JSON · Table |
| The Resolve logEvery signed answer’s leaf, merged into chained, witnessed batches with RFC 9162 proofs | every batch | 622,007leaves | Log · Signing keys |
| AI crawler catalogueEvery named AI, search and training agent: operator, purpose, and how its traffic can be verified | on change | 116agents | JSON · Page |
| Vendor website benchmarkSEO and AI-visibility vendors measured by the same rules as everyone else, with immutable daily snapshots | daily | — | JSON · CSV · Snapshots |
| GlossaryEvery defined term, with what it can support and what it cannot, as DefinedTermSet JSON-LD | weekly | 792terms | JSON-LD · Page |
| AI-use declarations, top 10,000What the top 10,000 sites declare about AI use, one signed line per site, for crawler operators to load before fetching | daily editions | — | Page · JSON |
| AI consent data (sample)What each site’s robots.txt tells named AI agents, its Content-Signal line and llms.txt, signed; the free sample edition | signed releases | — | Page · JSON |
| Finding counts by problemHow many measured sites carry each of the problems the scan can raise, by lane and tier | hourly | — | Page · JSON |
| Technology countsHow many measured sites run each of the technologies fingerprinted, by tier | hourly | — | Page · JSON |
| Agentic-web census, weeklyThe top 10,000 sites on a fixed frame: crawler policy, machine files, capabilities and delivery parity, sealed once a week | weekly | — | Page · JSON · Feed |
| Who AI recommends, by cityWhen answer engines are asked for the best home-service business in a city (Denver metro, plus the cities the census covers most): how many businesses they cite, how concentrated the citations are, and how much of what they cite is a directory | about every 3 days | — | Page · JSON |
| Agentic web mapThe month’s map of the agentic web: which sites declare what to agents and what was verified | monthly | — | Page · JSON |
| Similarity censusWhich measured sites serve near-identical pages, by homepage fingerprint | weekly | — | Page · JSON |
| State of AI visibilityWhat answer engines were asked, which sites they named, and how stable the answers were | weekly | — | Page · JSON · Feed |
| AI classifier atlasWhat the models that screen businesses punish or reward, measured on the same scenarios across models | every 15 minutes | — | Page · JSON |
| Agent Traffic IndexVerified crawler and agent arrivals across the observed network, by agent and by month | monthly | — | Page · JSON |
| Agent conductHow each named agent behaves on sites: robots obedience, identity, rate | hourly | — | Page · JSON |
| Share of preflightHow many agent arrivals carried a CrawlCheck decision, per site and per agent | monthly, sealed | — | Page · JSON |
| Decisions to outcomesDecisions issued, receipts checked, outcomes reported, and where CrawlCheck was wrong | hourly | — | Page · JSON |
| Live figuresScans, decisions, crawler visits and API use, counted | hourly | — | Page · JSON |
| Agent safety benchmarkEleven scenarios, three guards: what each agent setup refuses and what it lets through | per release | — | Page · JSON |
| Registry: what sites declare to agentsEvery registered site’s declarations, verified by a fetch | live | — | Page · JSON |
| TLS posture, weeklyA stratified sample of measured sites probed every week: post-quantum key exchange, old TLS versions, deprecated key exchange; no site names | weekly | — | Page · JSON |
| Embeddings of the cataloguesThe crawler catalogue, glossary, problem codes and census measures as 768-dimension vectors, one JSON file per set | weekly | — | Page · JSON |
| Phone reputationWhether a phone number appears in the FTC Do Not Call complaint files, by number | daily ingest | — | Page · JSON |
28 free datasets. Cite any of them: BibTeX · RIS · every number with its denominator on the data page.
Filter the rows. The count and the sample are live.
Pick a lane, a state, a category, a grade, a platform or a technology. The count is read from the same files you would buy; the ten rows are a spread of what matched.
Start from a preset, or set the filters
| Where | Category | Grade | Platform | Runs | AI crawlers |
|---|---|---|---|---|---|
| Loading… | |||||
Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.
Not on the shelf? Order it.
Any slice of the web we can measure: an industry, a technology, a country, a city, your own list of domains, or a field we do not publish yet. We measure it for you and deliver the file. Custom orders start at $999; we reply by email with a quote and a delivery date before anything is charged.
What a row holds, how it is made, how it compares.
Every row holds 79 fields, in four groups
Identity and place
domain, at, lane, kind, kind_detected, state, city, name, phone, email, contact_src, osm_category, src, title, description, languageQuality
grade, score, grade_cap_reason, crawl_clarity, user_clarity, findings, finding_codes, rendering, content_ratio, blocking_resources, id_ratio, ld_typesTechnology
platform, cms, tech, edge, server, host, https, hsts, sitemap_urls, size, home_bytes, ttfb_ms, load_msAI crawlers
llms_txt, surfaces, robots_measured, ai_blocked, unmeasured, reasonSecurity and email
sec_headers, sec_hsts_max_age, sec_csp, sec_version_leak, sec_jquery, sec_mixed_content, sec_cookies_insecure, sec_scripts_no_sri, dns_spf, dns_dmarc, dns_caa, dns_mx, dns_dnssecDomain and hosting
reg_registrar, reg_created, reg_expires, reg_age_days, reg_expires_in_days, reg_dns_provider, reg_dnssec, net_ip, net_asn, net_as_name, net_countryEvery field, with its meaning (79)
| Field | Meaning |
|---|---|
| domain | Registrable domain scanned |
| at | When the homepage was read (UTC, ISO 8601) |
| lane | Lane the site was scanned in: saas, gov, local, filler |
| kind | Site kind: the page's own declaration (JSON-LD), else its patterns; for local rows with no page signal, the OpenStreetMap category |
| kind_detected | Kind read from the page alone |
| state | US state (local lane) |
| city | City from OpenStreetMap (local lane) |
| name | Business name from OpenStreetMap (local lane) |
| phone | Phone number the business publishes on its own homepage (local lane, rows read from 2026-11-14) |
| Email address the business publishes on its own homepage (local lane, rows read from 2026-11-14); an address the site hides from scripts is left out | |
| contact_src | Where each contact was read: schema (the page's JSON-LD) or link (a tel: or mailto: link) |
| osm_category | OpenStreetMap category, e.g. trades:plumber (local lane) |
| src | Where the domain came from: dotgov:<type>, tranco:<tld>, wikidata, discovered, OSM category |
| title | Homepage <title> |
| description | Homepage meta description |
| language | en or non-en, from the page |
| grade | CrawlCheck grade, A to F |
| score | CrawlCheck overall score 0-100, before any cap |
| grade_cap_reason | When a finding caps the letter below what the score would give, which finding and why |
| crawl_clarity | Crawl clarity score 0-100 |
| user_clarity | User clarity score 0-100 |
| findings | Number of findings |
| finding_codes | Finding codes, ;-separated |
| platform | CMS band: wordpress, shopify, webflow, squarespace, wix, nextjs, drupal, unknown |
| cms | CMS and builders detected |
| tech | Technologies detected in the homepage: analytics, tag managers, chat, payments, booking, consent, frameworks |
| edge | Edge or CDN detected from headers |
| server | Server header |
| host | First hosting signal |
| rendering | server or client, by a rule on the HTML we received, not by rendering it: client when under 1% of the payload is visible text (rule content_ratio_lt_1pct/v1, see rendering_heuristic). A heavy server-rendered page can read client |
| content_ratio | Share of the homepage payload that is visible content, % |
| blocking_resources | Render-blocking resources in <head> |
| id_ratio | Share of JSON-LD nodes with an @id |
| ld_types | schema.org types declared in JSON-LD |
| llms_txt | Serves /llms.txt |
| surfaces | Agent surfaces served: agents.md, agent skills, MCP card, API catalog, media kit |
| robots_measured | robots.txt was read |
| ai_blocked | AI crawlers robots.txt closes / to |
| ai_closed | Every AI agent (training and answer) robots.txt closes / to, of ~60 tested; reads from 16 Oct 2026 |
| content_signal | The Content-Signal line, verbatim, when one was read |
| https | Homepage on HTTPS |
| hsts | HSTS header present |
| sitemap_urls | URLs declared in sitemaps |
| size | small (<50), medium (<2000), large |
| home_bytes | Homepage bytes |
| ttfb_ms | Time to first byte from our observer, ms |
| load_ms | Deprecated alias of fetch_ms, same value; kept so existing imports do not break |
| sec_headers | Standard security headers sent, of 6: HSTS, a CSP that restricts scripts, X-Frame-Options or frame-ancestors, nosniff, Referrer-Policy, Permissions-Policy |
| sec_hsts_max_age | HSTS max-age in seconds (blank when no HSTS) |
| sec_csp | Content Security Policy: none, report-only, no-script-rule, unsafe-inline, strict |
| sec_version_leak | Server or framework version exposed in headers, e.g. Apache/2.4.41 or PHP/7.4 |
| sec_jquery | jQuery version loaded by the homepage |
| sec_mixed_content | http:// resources loaded on the https homepage |
| sec_cookies_insecure | Cookies set without the Secure flag |
| sec_scripts_no_sri | Scripts from other domains without an integrity hash |
| dns_spf | SPF policy: fail, softfail, neutral, pass, none (no all), missing |
| dns_dmarc | DMARC policy: reject, quarantine, none, missing (inherited from the parent domain when the subdomain has none) |
| dns_caa | CAA record limits which certificate authorities may issue |
| dns_mx | Mail servers (a null MX counts as 0) |
| dns_dnssec | DNS answers validated with DNSSEC |
| reg_registrar | Registrar of the registrable domain (RDAP) |
| reg_created | Domain registered on (YYYY-MM-DD) |
| reg_expires | Registration expires on (YYYY-MM-DD) |
| reg_age_days | Days since the domain was registered |
| reg_expires_in_days | Days until the registration expires |
| reg_dns_provider | Who runs the domain's name servers, e.g. Cloudflare, GoDaddy, Amazon Route 53 |
| reg_dnssec | DNSSEC delegation signed at the registry |
| net_ip | Homepage IPv4 address |
| net_asn | Network (autonomous system) the IP belongs to |
| net_as_name | Network owner, e.g. Cloudflare, Amazon, Automattic |
| net_country | Country the network registered the IP block in |
| unmeasured | true when the homepage could not be measured |
| reason | Why it could not be measured |
| rendering_heuristic | Id of the rule that set rendering (content_ratio_lt_1pct/v1); a new rule gets a new id |
| browser_render_verified | true only when a real browser rendered the page and confirmed rendering; false for every corpus row |
| fetch_ms | Time from our request to the last byte of the homepage HTML document (read up to 4 MB), from one datacentre. No images, scripts or styles are fetched and nothing is rendered: not a page-load time |
| tech_evidence | For each technology, where in the HTML it was matched: script-src, inline-script, script-attribute, link-href, iframe-src, meta-tag, img-src, attribute, text, markup. Rows from 29 Oct 2026 |
| tech_first_seen | For each technology, the first day (UTC) any read of this site in this lane saw it, since the lane's files began |
| tech_last_seen | For each technology, the latest day a read saw it |
Removals: a site whose owner opts out leaves every file from the next read. If you hold earlier rows, delete those listed at /api/datasets/removals (dated, hashed) at least every 30 days; counts already published are history, named records are removed. Every export name carries the release id.
See it before you buy: 25 measured rows as CSV · NDJSON · the catalogue with daily counts at /api/datasets.
Measured by us, every day. Delivered as a file, in a minute
1 · Source
Our own lists of domains. OpenStreetMap businesses one state a day, Wikidata software companies, the CISA .gov list, the Tranco top million, and links every scan discovers. Nothing bought from a data broker.
2 · Read
One observer, one request profile. The homepage, robots.txt, the sitemaps and the agent files, read the way a crawler reads them. Each row says when.
3 · Judge
Published rules, no guesses. Kinds come from the page’s own JSON-LD first, then patterns needing two independent signals. A site we could not measure is a row marked unmeasured, with the reason, never a failing grade.
4 · Deliver
One file per lane per day, compacted every hour at :17. /api/datasets/file?day=YYYY-MM-DD&lane=saas&format=csv&tech=Stripe with your key in x-crawlcheck-key, or a button in the browser. A site that opts out at /policy leaves the files.
The four lanes: quota, today, and where the domains come from
| Lane | Daily quota | Today | In the files | Source of the domains |
|---|---|---|---|---|
| SaaS and software, worldwide | 5,000 | 1,725 | 119,583 | Wikidata software companies, the Tranco top million, and links found on software sites |
| Government and education | 2,500 | 1,948 | 67,927 | The CISA .gov list, .edu and foreign government domains, and links between them |
| US local businesses, one state per day | 10,000 | 7,223 | 221,657 | OpenStreetMap businesses, one US state a day (ohio today) |
| Everything else the corpus reached (publishers, shops, directories, docs) | the rest | — | 315,016 | Links discovered by every scan: publishers, shops, directories, docs |
Most common technology in the newest files (2026-11-21)
| Technology | Sites |
|---|---|
| jQuery | 6,528 |
| Google Fonts | 4,876 |
| Google Analytics | 4,425 |
| Font Awesome | 4,249 |
| Google Tag Manager | 3,452 |
| Bootstrap | 1,971 |
| Yoast SEO | 1,879 |
| Elementor | 1,833 |
| WooCommerce | 1,258 |
| Adobe Fonts | 1,211 |
| reCAPTCHA | 1,182 |
| YouTube embed | 940 |
Compared with where lead and technology lists come from today
CrawlCheck publishes this page. Products change; this was checked on 15 October 2026. Only prices shown on a vendor’s own pricing page are printed.
| Source | What a row is built from | US local list, by state and category | Site grade | AI-crawler policy | Price |
|---|---|---|---|---|---|
| CrawlCheck | A fresh read of each site: platform, technology, grade with its findings, what it serves AI crawlers | Yes, from OpenStreetMap, one state a day | Yes | Yes | $149 a state, once · $349 a month a lane |
| BuiltWith | Technology detected on sites, with install history | Not as an OpenStreetMap-style local list | No | No | From $295 a month (builtwith.com/plans) |
| Wappalyzer | Technology detected on sites, lookups and lists by technology | No | No | No | Free extension; paid plans for lists and the API |
| Apollo | A company and contact database for sales outreach | Company records, not a read of the website | No | No | Per-seat plans with credits |