CrawlCheck
Website data, as files

Local businesses in your state, each with the website problem to open with

One file per US state: every local business we measured there, with its name, city and category, the phone and email its own website publishes, and the problems we found on that website. $149 a state, 30 days of downloads.

Phone and email were added on 14 November 2026. Each state is being read again to add them, one a day from 14 November: Colorado, Texas, California, Florida, New York, then the rest in the order they were first read. A state not yet read again has no contacts in its rows. A row never carries a contact the site does not publish itself.

Also sold

US local businesses · one state a day

Lit states are in the files; today’s read is Ohio. Click a lit state to filter the rows below.
  • 782,981 distinct sites, 557,520 measured on their newest read (median 23.6 days old)
  • 724,183 measurements, one per site per day
  • 221,657 US local businesses with city and category
  • 86 technologies fingerprinted per site
  • 10,896 measured today, 2026-11-21 Mountain Time

Release ds-20261122-0120-e3b71a24 · counts are measured reads only; 297,456 reads that got no answer are left out. The same release id is in /api/datasets.

Start here

One state, or one problem.

One row per site per day, 79 fields a row, measured by us — never bought from a data broker. Counts are sites in the files today. Checkout is by Stripe; the key and the first download are on the next page.

Dataset · what you getSitesUpdatedPrice
One US state, local businessesEvery measured local business in one state: city, category, platform, grade, technologies, AI-crawler policy. 30 days of downloads.221,657all states so far2026-11-21$149oncePick a state
Websites by problem52 lists: every site with a broken robots.txt, a sitemap search engines cannot read, no HSTS and more — businesses that need the fix you sell.60lists2026-11-22from $99once, by sizePick a problem
Everything else we sell: ready-made lists, technology lists, daily feeds, licences
Dataset · what you getSitesUpdatedPrice
Trades and home servicesPlumbers, electricians, roofers and trades, each with its city, platform, grade and the defect to open with.2,6742026-11-22$299onceBuy the list
Trades and home services, one stateThe same list for the state you sell into: every trade or home-service business we measured there, with the same columns. 30 days of downloads.2,553in 7 states2026-11-22$99once
Online storesEvery site that sells online, with its platform (Shopify, WooCommerce, BigCommerce), grade and AI-crawler policy.18,1522026-11-22$299onceBuy the list
Publishers and news sitesNews and article sites with how each treats AI crawlers: licensing, ad tech and PR targets.13,1932026-11-22$199onceBuy the list
Sites that block AI crawlersSites whose robots.txt shuts out a named AI agent (GPTBot, ClaudeBot, CCBot), with which ones.9,6682026-11-22$149onceBuy the list
Sites with an llms.txtEvery site that publishes an llms.txt, by sector and platform — the early adopters.25,1032026-11-22$149onceBuy the list
Domains open to email spoofingDomains that take email but have no enforcing DMARC policy, each with its mail setup, SPF and the platform it runs.69,0792026-11-22$249onceBuy the list
Sites missing basic securitySites missing basic protections: which headers are absent, exposed server versions, outdated jQuery, insecure content.96,9402026-11-22$199onceBuy the list
Domains expiring in the next 45 daysDomains whose registration expires within 45 days, with registrar, expiry date, DNS provider and hosting network.6,2812026-11-22$149onceBuy the list
Domains registered in the last 6 monthsDomains registered in the last 6 months that already serve a live site, with registration date, registrar and stack.2,4332026-11-22$149onceBuy the list
Sites running a marketing or support platformWho runs HubSpot, Klaviyo, Intercom, Zendesk and 12 more — a displacement list for competing vendors.6,2932026-11-22$249onceBuy the list
Websites by technology86 lists: every site running HubSpot, Shopify, Stripe, Klaviyo, Intercom and 81 more, with platform, grade and AI-crawler policy.86lists2026-11-22from $99once, by sizePick a technology
One lane, every daySaaS and software (119,583 sites), government and education (67,927), or US local as the states roll. Daily files and the whole history of the lane; the lists cut from your lane.409,1672026-11-21$349a monthBuy a lane
All lanesEverything: the four lanes including the wider corpus (publishers, shops, directories, docs), every list in full, all history, every filter.724,183measurements, one per site per day2026-11-21$899a month
or $8,990 a year
Buy all lanes
CreditsPrepaid, 1 credit = $1, spent on any list, technology list, state or lane. Never expire. $25 buys 30; $100 buys 150.——from $10onceBuy credits
The snapshot archiveEvery signed Resolve answer behind the public window, kept 90 days: the same files, listed and served on your key, so a change on any measured site can be placed to the day.99,784answers a day2026-11-21$199a month
or $1,990 a year
Get the archive
EnterpriseRedistribution rights, a custom lane (an industry, a country, a technology), your own domains re-measured weekly. A contract and an invoice.——From $25,000a yearAsk for a quote

A state pack and a list are one payment with 30 days of downloads; lanes are monthly and cancel any time from /manage. Prices are per organisation. 25 free rows · the catalogue as JSON · terms.

Free · CC BY 4.0 · no key

Every free dataset, in one place.

The current edition of everything CrawlCheck measures is free to reuse, commercially included: credit “CrawlCheck” with the URL of the file. What is sold above is depth: row-level files, lists cut to a buyer, and the archive.

For AI teams: every record is signed and each day is anchored in Bitcoin, so a value cannot be changed after the date it carries. That makes these files checkable ground truth for evaluating agents, or scoring them in training. The agent safety benchmark is built from them.

Dataset · what it holdsUpdatedSizeGet it
The Resolve snapshotEvery public signed answer about a measured site, one file set a day; each line verifies on its own with no call to usdaily, newest 7 public99,784answers in the newest (2026-11-21)Index · File layout
Agentic-web censusThe most-visited domains on a fixed frame: crawler policy, machine files, declared capabilities, and whether AI crawlers get what a browser gets; every site nameddaily and weekly—Daily records · Weekly · Atom
The instrumentsDecisions to outcomes, Metrics, Agent safety benchmark, Agent Traffic Index, Assurance, Status, Agent conduct, Proof: the Resolve log, Agentic web census, weekly, Agentic web map, Share of preflight, Vendor website benchmark, Registry: what sites declare to agents: each a signed JSON twin with one sealed edition per ISO weeksigned hourly, sealed weekly13instrumentsIndex · The pages
Finding frequenciesEvery scan run here, counted by finding: how common, how serious; names no domaincontinuous—JSON · Table
The Resolve logEvery signed answer’s leaf, merged into chained, witnessed batches with RFC 9162 proofsevery batch622,007leavesLog · Signing keys
AI crawler catalogueEvery named AI, search and training agent: operator, purpose, and how its traffic can be verifiedon change116agentsJSON · Page
Vendor website benchmarkSEO and AI-visibility vendors measured by the same rules as everyone else, with immutable daily snapshotsdaily—JSON · CSV · Snapshots
GlossaryEvery defined term, with what it can support and what it cannot, as DefinedTermSet JSON-LDweekly792termsJSON-LD · Page
AI-use declarations, top 10,000What the top 10,000 sites declare about AI use, one signed line per site, for crawler operators to load before fetchingdaily editions—Page · JSON
Finding counts by problemHow many measured sites carry each of the problems the scan can raise, by lane and tierhourly—Page · JSON
Technology countsHow many measured sites run each of the technologies fingerprinted, by tierhourly—Page · JSON
Agentic-web census, weeklyThe top 10,000 sites on a fixed frame: crawler policy, machine files, capabilities and delivery parity, sealed once a weekweekly—Page · JSON · Feed
Who AI recommends, by cityWhen answer engines are asked for the best home-service business in a city (Denver metro, plus the cities the census covers most): how many businesses they cite, how concentrated the citations are, and how much of what they cite is a directoryabout every 3 days—Page · JSON
Agentic web mapThe month’s map of the agentic web: which sites declare what to agents and what was verifiedmonthly—Page · JSON
Similarity censusWhich measured sites serve near-identical pages, by homepage fingerprintweekly—Page · JSON
State of AI visibilityWhat answer engines were asked, which sites they named, and how stable the answers wereweekly—Page · JSON · Feed
AI classifier atlasWhat the models that screen businesses punish or reward, measured on the same scenarios across modelsevery 15 minutes—Page · JSON
Agent Traffic IndexVerified crawler and agent arrivals across the observed network, by agent and by monthmonthly—Page · JSON
Agent conductHow each named agent behaves on sites: robots obedience, identity, ratehourly—Page · JSON
Share of preflightHow many agent arrivals carried a CrawlCheck decision, per site and per agentmonthly, sealed—Page · JSON
Decisions to outcomesDecisions issued, receipts checked, outcomes reported, and where CrawlCheck was wronghourly—Page · JSON
Live figuresScans, decisions, crawler visits and API use, countedhourly—Page · JSON
Agent safety benchmarkEleven scenarios, three guards: what each agent setup refuses and what it lets throughper release—Page · JSON
Registry: what sites declare to agentsEvery registered site’s declarations, verified by a fetchlive—Page · JSON
TLS posture, weeklyA stratified sample of measured sites probed every week: post-quantum key exchange, old TLS versions, deprecated key exchange; no site namesweekly—Page · JSON
Embeddings of the cataloguesThe crawler catalogue, glossary, problem codes and census measures as 768-dimension vectors, one JSON file per setweekly—Page · JSON
Phone reputationWhether a phone number appears in the FTC Do Not Call complaint files, by numberdaily ingest—Page · JSON

28 free datasets. Cite any of them: BibTeX · RIS · every number with its denominator on the data page.

Try it

Filter the rows. The count and the sample are live.

Pick a lane, a state, a category, a grade, a platform or a technology. The count is read from the same files you would buy; the ten rows are a spread of what matched.

Start from a preset, or set the filters

…measured sites match, over the last – days
WhereCategoryGradePlatformRunsAI crawlers
Loading…

Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Order a dataset

Not on the shelf? Order it.

Any slice of the web we can measure: an industry, a technology, a country, a city, your own list of domains, or a field we do not publish yet. We measure it for you and deliver the file. Custom orders start at $999; we reply by email with a quote and a delivery date before anything is charged.

Tell us what you needwho, where, which fields, how many sites
Get a quoteprice and delivery date, by email
We measure itevery site read fresh by our crawler, the same 79 fields
You get the fileCSV or NDJSON, plus re-measure options

Nothing is charged until you accept the quote.

Details

What a row holds, how it is made, how it compares.

Every row holds 79 fields, in four groups

Identity and place

domain, at, lane, kind, kind_detected, state, city, name, phone, email, contact_src, osm_category, src, title, description, language

Quality

grade, score, grade_cap_reason, crawl_clarity, user_clarity, findings, finding_codes, rendering, content_ratio, blocking_resources, id_ratio, ld_types

Technology

platform, cms, tech, edge, server, host, https, hsts, sitemap_urls, size, home_bytes, ttfb_ms, load_ms

AI crawlers

llms_txt, surfaces, robots_measured, ai_blocked, unmeasured, reason

Security and email

sec_headers, sec_hsts_max_age, sec_csp, sec_version_leak, sec_jquery, sec_mixed_content, sec_cookies_insecure, sec_scripts_no_sri, dns_spf, dns_dmarc, dns_caa, dns_mx, dns_dnssec

Domain and hosting

reg_registrar, reg_created, reg_expires, reg_age_days, reg_expires_in_days, reg_dns_provider, reg_dnssec, net_ip, net_asn, net_as_name, net_country
Every field, with its meaning (79)
FieldMeaning
domainRegistrable domain scanned
atWhen the homepage was read (UTC, ISO 8601)
laneLane the site was scanned in: saas, gov, local, filler
kindSite kind: the page's own declaration (JSON-LD), else its patterns; for local rows with no page signal, the OpenStreetMap category
kind_detectedKind read from the page alone
stateUS state (local lane)
cityCity from OpenStreetMap (local lane)
nameBusiness name from OpenStreetMap (local lane)
phonePhone number the business publishes on its own homepage (local lane, rows read from 2026-11-14)
emailEmail address the business publishes on its own homepage (local lane, rows read from 2026-11-14); an address the site hides from scripts is left out
contact_srcWhere each contact was read: schema (the page's JSON-LD) or link (a tel: or mailto: link)
osm_categoryOpenStreetMap category, e.g. trades:plumber (local lane)
srcWhere the domain came from: dotgov:<type>, tranco:<tld>, wikidata, discovered, OSM category
titleHomepage <title>
descriptionHomepage meta description
languageen or non-en, from the page
gradeCrawlCheck grade, A to F
scoreCrawlCheck overall score 0-100, before any cap
grade_cap_reasonWhen a finding caps the letter below what the score would give, which finding and why
crawl_clarityCrawl clarity score 0-100
user_clarityUser clarity score 0-100
findingsNumber of findings
finding_codesFinding codes, ;-separated
platformCMS band: wordpress, shopify, webflow, squarespace, wix, nextjs, drupal, unknown
cmsCMS and builders detected
techTechnologies detected in the homepage: analytics, tag managers, chat, payments, booking, consent, frameworks
edgeEdge or CDN detected from headers
serverServer header
hostFirst hosting signal
renderingserver or client, by a rule on the HTML we received, not by rendering it: client when under 1% of the payload is visible text (rule content_ratio_lt_1pct/v1, see rendering_heuristic). A heavy server-rendered page can read client
content_ratioShare of the homepage payload that is visible content, %
blocking_resourcesRender-blocking resources in <head>
id_ratioShare of JSON-LD nodes with an @id
ld_typesschema.org types declared in JSON-LD
llms_txtServes /llms.txt
surfacesAgent surfaces served: agents.md, agent skills, MCP card, API catalog, media kit
robots_measuredrobots.txt was read
ai_blockedAI crawlers robots.txt closes / to
ai_closedEvery AI agent (training and answer) robots.txt closes / to, of ~60 tested; reads from 16 Oct 2026
content_signalThe Content-Signal line, verbatim, when one was read
httpsHomepage on HTTPS
hstsHSTS header present
sitemap_urlsURLs declared in sitemaps
sizesmall (<50), medium (<2000), large
home_bytesHomepage bytes
ttfb_msTime to first byte from our observer, ms
load_msDeprecated alias of fetch_ms, same value; kept so existing imports do not break
sec_headersStandard security headers sent, of 6: HSTS, a CSP that restricts scripts, X-Frame-Options or frame-ancestors, nosniff, Referrer-Policy, Permissions-Policy
sec_hsts_max_ageHSTS max-age in seconds (blank when no HSTS)
sec_cspContent Security Policy: none, report-only, no-script-rule, unsafe-inline, strict
sec_version_leakServer or framework version exposed in headers, e.g. Apache/2.4.41 or PHP/7.4
sec_jqueryjQuery version loaded by the homepage
sec_mixed_contenthttp:// resources loaded on the https homepage
sec_cookies_insecureCookies set without the Secure flag
sec_scripts_no_sriScripts from other domains without an integrity hash
dns_spfSPF policy: fail, softfail, neutral, pass, none (no all), missing
dns_dmarcDMARC policy: reject, quarantine, none, missing (inherited from the parent domain when the subdomain has none)
dns_caaCAA record limits which certificate authorities may issue
dns_mxMail servers (a null MX counts as 0)
dns_dnssecDNS answers validated with DNSSEC
reg_registrarRegistrar of the registrable domain (RDAP)
reg_createdDomain registered on (YYYY-MM-DD)
reg_expiresRegistration expires on (YYYY-MM-DD)
reg_age_daysDays since the domain was registered
reg_expires_in_daysDays until the registration expires
reg_dns_providerWho runs the domain's name servers, e.g. Cloudflare, GoDaddy, Amazon Route 53
reg_dnssecDNSSEC delegation signed at the registry
net_ipHomepage IPv4 address
net_asnNetwork (autonomous system) the IP belongs to
net_as_nameNetwork owner, e.g. Cloudflare, Amazon, Automattic
net_countryCountry the network registered the IP block in
unmeasuredtrue when the homepage could not be measured
reasonWhy it could not be measured
rendering_heuristicId of the rule that set rendering (content_ratio_lt_1pct/v1); a new rule gets a new id
browser_render_verifiedtrue only when a real browser rendered the page and confirmed rendering; false for every corpus row
fetch_msTime from our request to the last byte of the homepage HTML document (read up to 4 MB), from one datacentre. No images, scripts or styles are fetched and nothing is rendered: not a page-load time
tech_evidenceFor each technology, where in the HTML it was matched: script-src, inline-script, script-attribute, link-href, iframe-src, meta-tag, img-src, attribute, text, markup. Rows from 29 Oct 2026
tech_first_seenFor each technology, the first day (UTC) any read of this site in this lane saw it, since the lane's files began
tech_last_seenFor each technology, the latest day a read saw it

Removals: a site whose owner opts out leaves every file from the next read. If you hold earlier rows, delete those listed at /api/datasets/removals (dated, hashed) at least every 30 days; counts already published are history, named records are removed. Every export name carries the release id.

See it before you buy: 25 measured rows as CSV · NDJSON · the catalogue with daily counts at /api/datasets.

Measured by us, every day. Delivered as a file, in a minute

1 · Source

Our own lists of domains. OpenStreetMap businesses one state a day, Wikidata software companies, the CISA .gov list, the Tranco top million, and links every scan discovers. Nothing bought from a data broker.

2 · Read

One observer, one request profile. The homepage, robots.txt, the sitemaps and the agent files, read the way a crawler reads them. Each row says when.

3 · Judge

Published rules, no guesses. Kinds come from the page’s own JSON-LD first, then patterns needing two independent signals. A site we could not measure is a row marked unmeasured, with the reason, never a failing grade.

4 · Deliver

One file per lane per day, compacted every hour at :17. /api/datasets/file?day=YYYY-MM-DD&lane=saas&format=csv&tech=Stripe with your key in x-crawlcheck-key, or a button in the browser. A site that opts out at /policy leaves the files.

The four lanes: quota, today, and where the domains come from
LaneDaily quotaTodayIn the filesSource of the domains
SaaS and software, worldwide5,0001,725119,583Wikidata software companies, the Tranco top million, and links found on software sites
Government and education2,5001,94867,927The CISA .gov list, .edu and foreign government domains, and links between them
US local businesses, one state per day10,0007,223221,657OpenStreetMap businesses, one US state a day (ohio today)
Everything else the corpus reached (publishers, shops, directories, docs)the rest—315,016Links discovered by every scan: publishers, shops, directories, docs
Most common technology in the newest files (2026-11-21)
TechnologySites
jQuery6,528
Google Fonts4,876
Google Analytics4,425
Font Awesome4,249
Google Tag Manager3,452
Bootstrap1,971
Yoast SEO1,879
Elementor1,833
WooCommerce1,258
Adobe Fonts1,211
reCAPTCHA1,182
YouTube embed940
Compared with where lead and technology lists come from today

CrawlCheck publishes this page. Products change; this was checked on 15 October 2026. Only prices shown on a vendor’s own pricing page are printed.

SourceWhat a row is built fromUS local list, by state and categorySite gradeAI-crawler policyPrice
CrawlCheckA fresh read of each site: platform, technology, grade with its findings, what it serves AI crawlersYes, from OpenStreetMap, one state a dayYesYes$149 a state, once · $349 a month a lane
BuiltWithTechnology detected on sites, with install historyNot as an OpenStreetMap-style local listNoNoFrom $295 a month (builtwith.com/plans)
WappalyzerTechnology detected on sites, lookups and lists by technologyNoNoNoFree extension; paid plans for lists and the API
ApolloA company and contact database for sales outreachCompany records, not a read of the websiteNoNoPer-seat plans with credits

Questions about this page

QWhat is in a row?
One website on one day, in 79 fields: what it is (kind, and for US local businesses the state, city, OpenStreetMap category and name), what it runs (platform, CMS, technologies, edge, server), how well it is built (A to F grade, score, every finding by code) and what it serves AI crawlers (llms.txt, the robots.txt reading, which named AI agents are blocked). Every row carries the time it was read.
QWhere do the websites come from?
From CrawlCheck’s own lists, never a data broker: OpenStreetMap businesses one US state a day, Wikidata software companies and the Tranco top million for SaaS, the CISA .gov list and .edu domains for government and education, and the links every scan discovers. Each site is then read by CrawlCheck’s own observer.
QHow fresh is it?
Files are compacted every hour at 17 minutes past. The local lane covers one US state a day, so a state pack holds every day that state was scanned; the SaaS and government lanes fill daily on quota. A monthly plan gets each day’s file as it lands.
QHow do I get the file after paying?
Checkout is by Stripe. The page after payment shows your key once and a download button for your file; the same file is at /api/datasets/file or /api/datasets/state with the key in the x-crawlcheck-key header, as CSV or NDJSON, filterable by state, kind, technology and platform.
QWhy are names withheld in the preview?
The preview counts and shows a spread of matching rows without the business name or website, because a business found with a defect is never named publicly before its owner has been contacted. The names and domains are in the file you buy.
QWhat is free?
The 25-row sample, the catalogue with daily counts at /api/datasets, the live preview, and the open datasets about the agentic web (the Resolve snapshot, the census, the AI crawler catalogue and the rest) under CC BY 4.0. What is sold is the row-level files and the ready-made lists.
QCan I cancel, and can I resell the data?
Monthly plans cancel any time from /manage; the key stops at the end of the paid month and files already downloaded stay yours to use. Reselling or republishing the rows needs the Enterprise plan. The standard plans cover your own organisation’s use, including campaigns you run for your clients: a client may receive your reports and findings, but not the rows, a filtered extract or a reusable list. A client who wants the rows buys its own licence. Terms: /api-terms.