CrawlCheck

Datasets · by problem

HTTPS is served without a Strict-Transport-Security header

Strict-Transport-Security tells a browser that has visited once to use HTTPS for every later request without trying HTTP first. Without it, a typed or linked http:// address makes one plaintext request before the redirect. It does not change what crawlers receive or how the site ranks; it is a small, reversible hardening step, and it is listed at the lowest severity for that reason. 49,868 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

  • 49,868 sites, last 7 days
  • 16,485 US local businesses
  • 4% also block an AI crawler
  • $299 once, 30 days of downloads

Who has it

Platform

No CMS detected 56%WordPress 37%Next.js 4%Drupal 2%Webflow 0%Shopify 0%

Grade

D 35%C 33%A 14%B 11%F 7%

Where

Wider corpus 43%US local businesses 33%SaaS and software 14%Government and education 11%

WhereCategoryGradePlatformAI crawlers
SaaS and softwareSaasBWordPressallowed
SaaS and softwareSaasAWordPressallowed
SaaS and softwareSaasCWordPressallowed
SaaS and softwareOtherDNo CMS detectedallowed
SaaS and softwareDocsDNo CMS detectedallowed
SaaS and softwareSaasANo CMS detectedallowed
SaaS and softwareOtherDNo CMS detectedallowed
SaaS and softwareSaasANo CMS detectedallowed

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.