CrawlCheck

Datasets · by problem

The HSTS header asks for browser preloading but does not qualify for it

The header carries the `preload` token, which is a request to be compiled into the HTTPS-only list shipped inside browsers. That list only accepts a header with max-age of at least 31536000 (one year) AND includeSubDomains AND preload. As sent, the token is ignored: the site is declaring an intent it does not meet, and if it ever was on the list, this header is what gets it removed. The rest of the HSTS header still works. 2,058 sites had this problem when we measured them in the last 7 days. The list gives each one with its domain, name where known, platform, every technology we detected, its A–F grade with all its findings, location where known, and what it serves AI crawlers.

  • 2,058 sites, last 7 days
  • 541 US local businesses
  • 7% also block an AI crawler
  • $199 once, 30 days of downloads

Who has it

Platform

No CMS detected 69%WordPress 19%Next.js 7%Drupal 3%Webflow 1%Squarespace 1%

Grade

C 32%D 28%A 22%B 10%F 8%

Where

Wider corpus 54%US local businesses 26%SaaS and software 12%Government and education 8%

WhereCategoryGradePlatformAI crawlers
SaaS and softwareOtherAWordPressallowed
SaaS and softwareSaasCNo CMS detectedallowed
SaaS and softwarePublisherAWordPressallowed
SaaS and softwareSaasDNo CMS detectedallowed
SaaS and softwareOtherCNo CMS detectedallowed
SaaS and softwareOtherFNo CMS detectedallowed
SaaS and softwareOtherANo CMS detectedblocked
SaaS and softwareOtherDNext.jsallowed

A sample from the live files. Names and websites are withheld here; they are in the file you buy. A business found with a defect is never named publicly before its owner has been contacted.

Every field: what a row holds · 25 free rows · a problem, country or industry we do not list: order a custom dataset.