Website data, as files

# Local businesses in your state, each with the website problem to open with

One file per US state: every local business we measured there, with its name, city and category, the phone and email its own website publishes, and the problems we found on that website. $149 a state, 30 days of downloads.

Pick a state · $149[Download 25 free rows (CSV)](/api/datasets/sample)

Phone and email were added on 14 November 2026. Each state is being read again to add them, one a day from 14 November: Colorado, Texas, California, Florida, New York, then the rest in the order they were first read. A state not yet read again has no contacts in its rows. A row never carries a contact the site does not publish itself.

Also sold

[By problemSites with a specific defect, for agencies and tool vendors who fix it. 52 lists, from $99.](/datasets/problems)[By technologySites running a given platform or tool, for sales teams and integrators. From $99.](/datasets/technologies)

Everything else we sell · Not on the shelf? Order a dataset

US local businesses · one state a day

AK ME WI VT NH WA ID MT ND MN IL MI NY MA RI OR NV WY SD IA IN OH PA NJ CT CA UT CO NE MO KY WV VA MD DE AZ NM KS AR TN NC SC DC OK LA MS AL GA HI TX FL
Lit states are in the files; today’s read is Ohio. Click a lit state to filter the rows below.

- **782,981** distinct sites, 557,520 measured on their newest read (median 23.6 days old)
- **724,183** measurements, one per site per day
- **221,657** US local businesses with city and category
- **86** technologies fingerprinted per site
- **10,896** measured today, 2026-11-21 Mountain Time

Release ds-20261122-0120-e3b71a24 · counts are measured reads only; 297,456 reads that got no answer are left out. The same release id is in [/api/datasets](/api/datasets).

 Start here

## One state, or one problem.

One row per site per day, 79 fields a row, measured by us — never bought from a data broker. Counts are sites in the files today. Checkout is by Stripe; the key and the first download are on the next page.

| Dataset · what you get | Sites | Updated | Price | |
| --- | --- | --- | --- | --- |
| **One US state, local businesses**Every measured local business in one state: city, category, platform, grade, technologies, AI-crawler policy. 30 days of downloads. | 221,657all states so far | 2026-11-21 | **$149**once | Pick a state |
| **Websites by problem**52 lists: every site with a broken robots.txt, a sitemap search engines cannot read, no HSTS and more — businesses that need the fix you sell. | 60lists | 2026-11-22 | **from $99**once, by size | [Pick a problem](/datasets/problems) |

#### Everything else we sell: ready-made lists, technology lists, daily feeds, licences

| Dataset · what you get | Sites | Updated | Price | |
| --- | --- | --- | --- | --- |
| **Trades and home services**Plumbers, electricians, roofers and trades, each with its city, platform, grade and the defect to open with. | 2,674 | 2026-11-22 | **$299**once | [Buy the list](/credits?item=list_home-services#buy) |
| **Trades and home services, one state**The same list for the state you sell into: every trade or home-service business we measured there, with the same columns. 30 days of downloads. | 2,553in 7 states | 2026-11-22 | **$99**once | State California (572)Texas (467)Florida (426)Pennsylvania (329)Illinois (262)Ohio (252)New York (245) Buy one state |
| **Online stores**Every site that sells online, with its platform (Shopify, WooCommerce, BigCommerce), grade and AI-crawler policy. | 18,152 | 2026-11-22 | **$299**once | [Buy the list](/credits?item=list_stores#buy) |
| **Publishers and news sites**News and article sites with how each treats AI crawlers: licensing, ad tech and PR targets. | 13,193 | 2026-11-22 | **$199**once | [Buy the list](/credits?item=list_publishers#buy) |
| **Sites that block AI crawlers**Sites whose robots.txt shuts out a named AI agent (GPTBot, ClaudeBot, CCBot), with which ones. | 9,668 | 2026-11-22 | **$149**once | [Buy the list](/credits?item=list_ai-blocking#buy) |
| **Sites with an llms.txt**Every site that publishes an llms.txt, by sector and platform — the early adopters. | 25,103 | 2026-11-22 | **$149**once | [Buy the list](/credits?item=list_llms-txt#buy) |
| **Domains open to email spoofing**Domains that take email but have no enforcing DMARC policy, each with its mail setup, SPF and the platform it runs. | 69,079 | 2026-11-22 | **$249**once | [Buy the list](/credits?item=list_email-spoofable#buy) |
| **Sites missing basic security**Sites missing basic protections: which headers are absent, exposed server versions, outdated jQuery, insecure content. | 96,940 | 2026-11-22 | **$199**once | [Buy the list](/credits?item=list_security-gaps#buy) |
| **Domains expiring in the next 45 days**Domains whose registration expires within 45 days, with registrar, expiry date, DNS provider and hosting network. | 6,281 | 2026-11-22 | **$149**once | [Buy the list](/credits?item=list_expiring-soon#buy) |
| **Domains registered in the last 6 months**Domains registered in the last 6 months that already serve a live site, with registration date, registrar and stack. | 2,433 | 2026-11-22 | **$149**once | [Buy the list](/credits?item=list_new-domains#buy) |
| **Sites running a marketing or support platform**Who runs HubSpot, Klaviyo, Intercom, Zendesk and 12 more — a displacement list for competing vendors. | 6,293 | 2026-11-22 | **$249**once | [Buy the list](/credits?item=list_martech#buy) |
| **Websites by technology**86 lists: every site running HubSpot, Shopify, Stripe, Klaviyo, Intercom and 81 more, with platform, grade and AI-crawler policy. | 86lists | 2026-11-22 | **from $99**once, by size | [Pick a technology](/datasets/technologies) |
| **One lane, every day**SaaS and software (119,583 sites), government and education (67,927), or US local as the states roll. Daily files and the whole history of the lane; the lists cut from your lane. | 409,167 | 2026-11-21 | **$349**a month | [Buy a lane](https://buy.stripe.com/00w6oHgkPdIt6jc0M52VG0j) |
| **All lanes**Everything: the four lanes including the wider corpus (publishers, shops, directories, docs), every list in full, all history, every filter. | 724,183measurements, one per site per day | 2026-11-21 | **$899**a month
or [$8,990 a year](https://buy.stripe.com/28E8wP0lR7k58rkdyR2VG0o) | [Buy all lanes](https://buy.stripe.com/28E3cvb0vgUF0YSfGZ2VG0k) |
| **Credits**Prepaid, 1 credit = $1, spent on any list, technology list, state or lane. Never expire. $25 buys 30; $100 buys 150. | — | — | **from $10**once | [Buy credits](/credits) |
| **AI consent licence**For AI companies and researchers: what each site’s robots.txt tells 16 named AI agents, its Content-Signal line and llms.txt, with the read date. Signed releases, free sample. | 124,677 | 2026-11-21 | **from $25,000**a year | [See the licence](/datasets/ai-consent) |
| **The snapshot archive**Every signed Resolve answer behind the public window, kept 90 days: the same files, listed and served on your key, so a change on any measured site can be placed to the day. | 99,784answers a day | 2026-11-21 | **$199**a month
or $1,990 a year | [Get the archive](https://buy.stripe.com/fZu14ngkP7k50YS8ex2VG0g) |
| **Enterprise**Redistribution rights, a custom lane (an industry, a country, a technology), your own domains re-measured weekly. A contract and an invoice. | — | — | **From $25,000**a year | Ask for a quote |

A state pack and a list are one payment with 30 days of downloads; lanes are monthly and cancel any time from [/manage](/manage). Prices are per organisation. [25 free rows](/api/datasets/sample) · [the catalogue as JSON](/api/datasets) · [terms](/api-terms).

Free · CC BY 4.0 · no key

## Every free dataset, in one place.

The current edition of everything CrawlCheck measures is free to reuse, commercially included: credit “CrawlCheck” with the URL of the file. What is sold above is depth: row-level files, lists cut to a buyer, and the archive.

**For AI teams:** every record is signed and each day is anchored in Bitcoin, so a value cannot be changed after the date it carries. That makes these files checkable ground truth for evaluating agents, or scoring them in training. The [agent safety benchmark](/benchmark) is built from them.

| Dataset · what it holds | Updated | Size | Get it |
| --- | --- | --- | --- |
| **The Resolve snapshot**Every public signed answer about a measured site, one file set a day; each line verifies on its own with no call to us | daily, newest 7 public | 99,784answers in the newest (2026-11-21) | [Index](/api/v1/snapshot) · [File layout](/docs/api#snapshot) |
| **Agentic-web census**The most-visited domains on a fixed frame: crawler policy, machine files, declared capabilities, and whether AI crawlers get what a browser gets; every site named | daily and weekly | — | [Daily records](/api/census) · [Weekly](/api/v1/census/weekly) · [Atom](/api/v1/census/weekly/feed.xml) |
| **The instruments**Decisions to outcomes, Metrics, Agent safety benchmark, Agent Traffic Index, Assurance, Status, Agent conduct, Proof: the Resolve log, Agentic web census, weekly, Agentic web map, Share of preflight, Vendor website benchmark, Registry: what sites declare to agents: each a signed JSON twin with one sealed edition per ISO week | signed hourly, sealed weekly | 13instruments | [Index](/api/v1/instruments) · [The pages](/data) |
| **Finding frequencies**Every scan run here, counted by finding: how common, how serious; names no domain | continuous | — | [JSON](/api/public/counts) · [Table](/data#findings) |
| **The Resolve log**Every signed answer’s leaf, merged into chained, witnessed batches with RFC 9162 proofs | every batch | 622,007leaves | [Log](/api/v1/log) · [Signing keys](/keys) |
| **AI crawler catalogue**Every named AI, search and training agent: operator, purpose, and how its traffic can be verified | on change | 116agents | [JSON](/ai-crawlers.json) · [Page](/ai-crawlers) |
| **Vendor website benchmark**SEO and AI-visibility vendors measured by the same rules as everyone else, with immutable daily snapshots | daily | — | [JSON](/directory.json) · [CSV](/directory.csv) · [Snapshots](/api/v2/directory/snapshots) |
| **Glossary**Every defined term, with what it can support and what it cannot, as DefinedTermSet JSON-LD | weekly | 792terms | [JSON-LD](/glossary.json) · [Page](/glossary) |
| **AI-use declarations, top 10,000**What the top 10,000 sites declare about AI use, one signed line per site, for crawler operators to load before fetching | daily editions | — | [Page](/spec/rights-feed) · [JSON](/api/v1/rights-feed) |
| **AI consent data (sample)**What each site’s robots.txt tells named AI agents, its Content-Signal line and llms.txt, signed; the free sample edition | signed releases | — | [Page](/datasets/ai-consent) · [JSON](/datasets/ai-consent/sample.manifest.json) |
| **Finding counts by problem**How many measured sites carry each of the problems the scan can raise, by lane and tier | hourly | — | [Page](/datasets/problems) · [JSON](/api/datasets/problems) |
| **Technology counts**How many measured sites run each of the technologies fingerprinted, by tier | hourly | — | [Page](/datasets/technologies) · [JSON](/api/datasets/technologies) |
| **Agentic-web census, weekly**The top 10,000 sites on a fixed frame: crawler policy, machine files, capabilities and delivery parity, sealed once a week | weekly | — | [Page](/data/census/weekly) · [JSON](/api/v1/census/weekly) · [Feed](/api/v1/census/weekly/feed.xml) |
| **Who AI recommends, by city**When answer engines are asked for the best home-service business in a city (Denver metro, plus the cities the census covers most): how many businesses they cite, how concentrated the citations are, and how much of what they cite is a directory | about every 3 days | — | [Page](/api/v1/ai-recommends) · [JSON](/api/v1/ai-recommends) |
| **Agentic web map**The month’s map of the agentic web: which sites declare what to agents and what was verified | monthly | — | [Page](/data/agentic-web-map) · [JSON](/api/v1/agentic-web-map) |
| **Similarity census**Which measured sites serve near-identical pages, by homepage fingerprint | weekly | — | [Page](/data/census/weekly) · [JSON](/api/v1/census/similarity) |
| **State of AI visibility**What answer engines were asked, which sites they named, and how stable the answers were | weekly | — | [Page](/data/state-of-ai-visibility) · [JSON](/data/state-of-ai-visibility/feed.xml) · [Feed](/data/state-of-ai-visibility/feed.xml) |
| **AI classifier atlas**What the models that screen businesses punish or reward, measured on the same scenarios across models | every 15 minutes | — | [Page](/data/north-star) · [JSON](/api/v1/ai-classifiers) |
| **Agent Traffic Index**Verified crawler and agent arrivals across the observed network, by agent and by month | monthly | — | [Page](/traffic-index) · [JSON](/api/v1/traffic-index) |
| **Agent conduct**How each named agent behaves on sites: robots obedience, identity, rate | hourly | — | [Page](/agents/conduct) · [JSON](/api/v1/conduct) |
| **Share of preflight**How many agent arrivals carried a CrawlCheck decision, per site and per agent | monthly, sealed | — | [Page](/data/share-of-preflight) · [JSON](/api/v1/share-of-preflight) |
| **Decisions to outcomes**Decisions issued, receipts checked, outcomes reported, and where CrawlCheck was wrong | hourly | — | [Page](/data/north-star) · [JSON](/api/v1/north-star) |
| **Live figures**Scans, decisions, crawler visits and API use, counted | hourly | — | [Page](/metrics) · [JSON](/api/v1/metrics) |
| **Agent safety benchmark**Eleven scenarios, three guards: what each agent setup refuses and what it lets through | per release | — | [Page](/benchmark) · [JSON](/api/v1/benchmark) |
| **Registry: what sites declare to agents**Every registered site’s declarations, verified by a fetch | live | — | [Page](/registry) · [JSON](/registry/capabilities) |
| **TLS posture, weekly**A stratified sample of measured sites probed every week: post-quantum key exchange, old TLS versions, deprecated key exchange; no site names | weekly | — | [Page](/data/tls-posture) · [JSON](/api/v1/tls-posture) |
| **Embeddings of the catalogues**The crawler catalogue, glossary, problem codes and census measures as 768-dimension vectors, one JSON file per set | weekly | — | [Page](/data/embeddings) · [JSON](/data/embeddings/index.json) |
| **Phone reputation**Whether a phone number appears in the FTC Do Not Call complaint files, by number | daily ingest | — | [Page](/docs/api#phone-reputation) · [JSON](/api/v1/phone-reputation?number=2025550123) |

28 free datasets. Cite any of them: [BibTeX](/data.bib) · [RIS](/data.ris) · every number with its denominator on [the data page](/data#catalogue).

Try it

## Filter the rows. The count and the sample are live.

Pick a lane, a state, a category, a grade, a platform or a technology. The count is read from the same files you would buy; the ten rows are a spread of what matched.

Start from a preset, or set the filters

Plumbers on WordPress, D or worse Restaurants with no CMS detected Software companies on HubSpot Universities and agencies on Drupal Dentists and clinics with an F

LaneUS local businessesSaaS and softwareGovernment and education StateAny stateColoradoTexasCaliforniaFloridaNew YorkPennsylvaniaIllinoisOhioGeorgiaNorth CarolinaMichiganNew JerseyVirginiaWashingtonArizonaTennesseeMassachusettsIndianaMarylandMissouriWisconsinMinnesotaSouth CarolinaAlabamaLouisianaKentuckyOregonOklahomaConnecticutUtahIowaNevadaArkansasKansasMississippiNew MexicoNebraskaIdahoWest VirginiaHawaiiNew HampshireMaineMontanaRhode IslandDelawareSouth DakotaNorth DakotaAlaskaDistrict Of ColumbiaVermontWyoming CategoryAny category GradeAny gradeABCD or worseF PlatformAny platformWordPressShopifyWixSquarespaceWebflowNext.jsDrupalNo CMS detected TechnologyAny technologyGoogle AnalyticsGoogle Tag ManagerGoogle AdsMeta PixelLinkedIn InsightTikTok PixelMicrosoft ClarityBing UETHotjarSegmentMixpanelAmplitudeHeapPlausibleMatomoHubSpotSalesforce PardotMarketoKlaviyoMailchimpIntercomDriftZendeskLiveChatTawk.toCrispPodiumBirdeyeCallRailCalendlyServiceTitanHousecall ProJobberStripePayPalSquareShopifyWooCommerceBigCommerceYotpoJudge.meTrustpilotOneTrustCookiebotTermlyreCAPTCHAhCaptchaCloudflare TurnstileCloudflare Web AnalyticsSentryNew RelicDatadog RUMOptimizelyVWOAlgoliajQueryReactNext.jsNuxtVue

**…**measured sites match, over the last – days

| Where | Category | Grade | Platform | Runs | AI crawlers |
| --- | --- | --- | --- | --- | --- |
| Loading… | | | | | |

Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Buy a state — $149[Download the free 25-row sample](/api/datasets/sample)What a row holds

Order a dataset

## Not on the shelf? Order it.

Any slice of the web we can measure: an industry, a technology, a country, a city, your own list of domains, or a field we do not publish yet. We measure it for you and deliver the file. Custom orders start at $999; we reply by email with a quote and a delivery date before anything is charged.

**Tell us what you need**who, where, which fields, how many sites

**Get a quote**price and delivery date, by email

**We measure it**every site read fresh by our crawler, the same 79 fields

**You get the file**CSV or NDJSON, plus re-measure options

Email

Company

What sites

How many sites (roughly)
Not sureUnder 1,0001,000 – 10,00010,000 – 100,000Over 100,000

Needed by
No rushWithin a weekWithin 2 weeksWithin a monthRecurring (weekly or monthly)

Anything else
Request a quote Nothing is charged until you accept the quote.

Details

## What a row holds, how it is made, how it compares.

#### Every row holds 79 fields, in four groups

Identity and place
`domain, at, lane, kind, kind_detected, state, city, name, phone, email, contact_src, osm_category, src, title, description, language`

Quality
`grade, score, grade_cap_reason, crawl_clarity, user_clarity, findings, finding_codes, rendering, content_ratio, blocking_resources, id_ratio, ld_types`

Technology
`platform, cms, tech, edge, server, host, https, hsts, sitemap_urls, size, home_bytes, ttfb_ms, load_ms`

AI crawlers
`llms_txt, surfaces, robots_measured, ai_blocked, unmeasured, reason`

Security and email
`sec_headers, sec_hsts_max_age, sec_csp, sec_version_leak, sec_jquery, sec_mixed_content, sec_cookies_insecure, sec_scripts_no_sri, dns_spf, dns_dmarc, dns_caa, dns_mx, dns_dnssec`

Domain and hosting
`reg_registrar, reg_created, reg_expires, reg_age_days, reg_expires_in_days, reg_dns_provider, reg_dnssec, net_ip, net_asn, net_as_name, net_country`

#### Every field, with its meaning (79)

| Field | Meaning |
| --- | --- |
| domain | Registrable domain scanned |
| at | When the homepage was read (UTC, ISO 8601) |
| lane | Lane the site was scanned in: saas, gov, local, filler |
| kind | Site kind: the page's own declaration (JSON-LD), else its patterns; for local rows with no page signal, the OpenStreetMap category |
| kind_detected | Kind read from the page alone |
| state | US state (local lane) |
| city | City from OpenStreetMap (local lane) |
| name | Business name from OpenStreetMap (local lane) |
| phone | Phone number the business publishes on its own homepage (local lane, rows read from 2026-11-14) |
| email | Email address the business publishes on its own homepage (local lane, rows read from 2026-11-14); an address the site hides from scripts is left out |
| contact_src | Where each contact was read: schema (the page's JSON-LD) or link (a tel: or mailto: link) |
| osm_category | OpenStreetMap category, e.g. trades:plumber (local lane) |
| src | Where the domain came from: dotgov:`<type>`, tranco:`<tld>`, wikidata, discovered, OSM category |
| title | Homepage `<title>` |
| description | Homepage meta description |
| language | en or non-en, from the page |
| grade | CrawlCheck grade, A to F |
| score | CrawlCheck overall score 0-100, before any cap |
| grade_cap_reason | When a finding caps the letter below what the score would give, which finding and why |
| crawl_clarity | Crawl clarity score 0-100 |
| user_clarity | User clarity score 0-100 |
| findings | Number of findings |
| finding_codes | Finding codes, ;-separated |
| platform | CMS band: wordpress, shopify, webflow, squarespace, wix, nextjs, drupal, unknown |
| cms | CMS and builders detected |
| tech | Technologies detected in the homepage: analytics, tag managers, chat, payments, booking, consent, frameworks |
| edge | Edge or CDN detected from headers |
| server | Server header |
| host | First hosting signal |
| rendering | server or client, by a rule on the HTML we received, not by rendering it: client when under 1% of the payload is visible text (rule content_ratio_lt_1pct/v1, see rendering_heuristic). A heavy server-rendered page can read client |
| content_ratio | Share of the homepage payload that is visible content, % |
| blocking_resources | Render-blocking resources in `<head>` |
| id_ratio | Share of JSON-LD nodes with an @id |
| ld_types | schema.org types declared in JSON-LD |
| llms_txt | Serves /llms.txt |
| surfaces | Agent surfaces served: agents.md, agent skills, MCP card, API catalog, media kit |
| robots_measured | robots.txt was read |
| ai_blocked | AI crawlers robots.txt closes / to |
| ai_closed | Every AI agent (training and answer) robots.txt closes / to, of ~60 tested; reads from 16 Oct 2026 |
| content_signal | The Content-Signal line, verbatim, when one was read |
| https | Homepage on HTTPS |
| hsts | HSTS header present |
| sitemap_urls | URLs declared in sitemaps |
| size | small (<50), medium (<2000), large |
| home_bytes | Homepage bytes |
| ttfb_ms | Time to first byte from our observer, ms |
| load_ms | Deprecated alias of fetch_ms, same value; kept so existing imports do not break |
| sec_headers | Standard security headers sent, of 6: HSTS, a CSP that restricts scripts, X-Frame-Options or frame-ancestors, nosniff, Referrer-Policy, Permissions-Policy |
| sec_hsts_max_age | HSTS max-age in seconds (blank when no HSTS) |
| sec_csp | Content Security Policy: none, report-only, no-script-rule, unsafe-inline, strict |
| sec_version_leak | Server or framework version exposed in headers, e.g. Apache/2.4.41 or PHP/7.4 |
| sec_jquery | jQuery version loaded by the homepage |
| sec_mixed_content | http:// resources loaded on the https homepage |
| sec_cookies_insecure | Cookies set without the Secure flag |
| sec_scripts_no_sri | Scripts from other domains without an integrity hash |
| dns_spf | SPF policy: fail, softfail, neutral, pass, none (no all), missing |
| dns_dmarc | DMARC policy: reject, quarantine, none, missing (inherited from the parent domain when the subdomain has none) |
| dns_caa | CAA record limits which certificate authorities may issue |
| dns_mx | Mail servers (a null MX counts as 0) |
| dns_dnssec | DNS answers validated with DNSSEC |
| reg_registrar | Registrar of the registrable domain (RDAP) |
| reg_created | Domain registered on (YYYY-MM-DD) |
| reg_expires | Registration expires on (YYYY-MM-DD) |
| reg_age_days | Days since the domain was registered |
| reg_expires_in_days | Days until the registration expires |
| reg_dns_provider | Who runs the domain's name servers, e.g. Cloudflare, GoDaddy, Amazon Route 53 |
| reg_dnssec | DNSSEC delegation signed at the registry |
| net_ip | Homepage IPv4 address |
| net_asn | Network (autonomous system) the IP belongs to |
| net_as_name | Network owner, e.g. Cloudflare, Amazon, Automattic |
| net_country | Country the network registered the IP block in |
| unmeasured | true when the homepage could not be measured |
| reason | Why it could not be measured |
| rendering_heuristic | Id of the rule that set rendering (content_ratio_lt_1pct/v1); a new rule gets a new id |
| browser_render_verified | true only when a real browser rendered the page and confirmed rendering; false for every corpus row |
| fetch_ms | Time from our request to the last byte of the homepage HTML document (read up to 4 MB), from one datacentre. No images, scripts or styles are fetched and nothing is rendered: not a page-load time |
| tech_evidence | For each technology, where in the HTML it was matched: script-src, inline-script, script-attribute, link-href, iframe-src, meta-tag, img-src, attribute, text, markup. Rows from 29 Oct 2026 |
| tech_first_seen | For each technology, the first day (UTC) any read of this site in this lane saw it, since the lane's files began |
| tech_last_seen | For each technology, the latest day a read saw it |

Removals: a site whose owner opts out leaves every file from the next read. If you hold earlier rows, delete those listed at [/api/datasets/removals](/api/datasets/removals) (dated, hashed) at least every 30 days; counts already published are history, named records are removed. Every export name carries the release id.

See it before you buy: [25 measured rows as CSV](/api/datasets/sample) · [NDJSON](/api/datasets/sample?format=ndjson) · the catalogue with daily counts at [/api/datasets](/api/datasets).

#### Measured by us, every day. Delivered as a file, in a minute

1 · Source

**Our own lists of domains.** OpenStreetMap businesses one state a day, Wikidata software companies, the CISA .gov list, the Tranco top million, and links every scan discovers. Nothing bought from a data broker.

2 · Read

**One observer, one request profile.** The homepage, robots.txt, the sitemaps and the agent files, read the way a crawler reads them. Each row says when.

3 · Judge

**Published rules, no guesses.** Kinds come from the page’s own JSON-LD first, then patterns needing two independent signals. A site we could not measure is a row marked unmeasured, with the reason, never a failing grade.

4 · Deliver

**One file per lane per day,** compacted every hour at :17. /api/datasets/file?day=YYYY-MM-DD&lane=saas&format=csv&tech=Stripe with your key in x-crawlcheck-key , or a button in the browser. A site that opts out at [/policy](/policy) leaves the files.

#### The four lanes: quota, today, and where the domains come from

| Lane | Daily quota | Today | In the files | Source of the domains |
| --- | --- | --- | --- | --- |
| **SaaS and software, worldwide** | 5,000 | 1,725 | 119,583 | Wikidata software companies, the Tranco top million, and links found on software sites |
| **Government and education** | 2,500 | 1,948 | 67,927 | The CISA .gov list, .edu and foreign government domains, and links between them |
| **US local businesses, one state per day** | 10,000 | 7,223 | 221,657 | OpenStreetMap businesses, one US state a day (ohio today) |
| **Everything else the corpus reached (publishers, shops, directories, docs)** | the rest | — | 315,016 | Links discovered by every scan: publishers, shops, directories, docs |

#### Most common technology in the newest files (2026-11-21)

| Technology | Sites |
| --- | --- |
| jQuery | 6,528 |
| Google Fonts | 4,876 |
| Google Analytics | 4,425 |
| Font Awesome | 4,249 |
| Google Tag Manager | 3,452 |
| Bootstrap | 1,971 |
| Yoast SEO | 1,879 |
| Elementor | 1,833 |
| WooCommerce | 1,258 |
| Adobe Fonts | 1,211 |
| reCAPTCHA | 1,182 |
| YouTube embed | 940 |

#### Compared with where lead and technology lists come from today

CrawlCheck publishes this page. Products change; this was checked on 15 October 2026. Only prices shown on a vendor’s own pricing page are printed.

| Source | What a row is built from | US local list, by state and category | Site grade | AI-crawler policy | Price |
| --- | --- | --- | --- | --- | --- |
| **CrawlCheck** | A fresh read of each site: platform, technology, grade with its findings, what it serves AI crawlers | Yes, from OpenStreetMap, one state a day | Yes | Yes | $149 a state, once · $349 a month a lane |
| **BuiltWith** | Technology detected on sites, with install history | Not as an OpenStreetMap-style local list | No | No | From $295 a month (builtwith.com/plans) |
| **Wappalyzer** | Technology detected on sites, lookups and lists by technology | No | No | No | Free extension; paid plans for lists and the API |
| **Apollo** | A company and contact database for sales outreach | Company records, not a read of the website | No | No | Per-seat plans with credits |

## Questions about this page

#### Q What is in a row?

One website on one day, in 79 fields: what it is (kind, and for US local businesses the state, city, OpenStreetMap category and name), what it runs (platform, CMS, technologies, edge, server), how well it is built (A to F grade, score, every finding by code) and what it serves AI crawlers (llms.txt, the robots.txt reading, which named AI agents are blocked). Every row carries the time it was read.

#### Q Where do the websites come from?

From CrawlCheck’s own lists, never a data broker: OpenStreetMap businesses one US state a day, Wikidata software companies and the Tranco top million for SaaS, the CISA .gov list and .edu domains for government and education, and the links every scan discovers. Each site is then read by CrawlCheck’s own observer.

#### Q How fresh is it?

Files are compacted every hour at 17 minutes past. The local lane covers one US state a day, so a state pack holds every day that state was scanned; the SaaS and government lanes fill daily on quota. A monthly plan gets each day’s file as it lands.

#### Q How do I get the file after paying?

Checkout is by Stripe. The page after payment shows your key once and a download button for your file; the same file is at /api/datasets/file or /api/datasets/state with the key in the x-crawlcheck-key header, as CSV or NDJSON, filterable by state, kind, technology and platform.

#### Q Why are names withheld in the preview?

The preview counts and shows a spread of matching rows without the business name or website, because a business found with a defect is never named publicly before its owner has been contacted. The names and domains are in the file you buy.

#### Q What is free?

The 25-row sample, the catalogue with daily counts at /api/datasets, the live preview, and the open datasets about the agentic web (the Resolve snapshot, the census, the AI crawler catalogue and the rest) under CC BY 4.0. What is sold is the row-level files and the ready-made lists.

#### Q Can I cancel, and can I resell the data?

Monthly plans cancel any time from /manage; the key stops at the end of the paid month and files already downloaded stay yours to use. Reselling or republishing the rows needs the Enterprise plan. The standard plans cover your own organisation’s use, including campaigns you run for your clients: a client may receive your reports and findings, but not the rows, a filtered extract or a reusable list. A client who wants the rows buys its own licence. Terms: /api-terms.

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [/](/) · [Listing check](/listing-check) · [Staple](/spec/staple) · [Conduct: Bingbot](/agents/conduct/bingbot) | **Registry:** [restyle.pl](/registry/restyle.pl) · [Places](/registry/places) · [peerpush.com](/registry/capabilities/peerpush.com) | **Datasets:** [Technologies: Hotjar](/datasets/technologies/hotjar) · [Technologies: Font awesome](/datasets/technologies/font-awesome) | **Blog:** [Agent conduct, October 2026: 34,676 verified requests](/blog/agent-conduct-october-2026)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
