Agent view · Guides · 2026-09-30
HSTS check: missing header, bad preload, removed entries
This is the response this server returns to a client that asks for Accept: text/markdown at https://crawlcheck.io/guides/hsts-audit-missing-header-preload-ineligible-and-removed-entries: fetched when you opened this page, shown byte for byte. An assistant, a crawler that negotiates markdown, or anything reading our feed gets exactly this. The markdown leaves signed (RFC 9421, key in our key directory), so a crawler can prove the page it read is the page this server sent; the check is shown below.
| status | 200 |
|---|---|
| content-type | text/markdown; charset=utf-8 |
| vary | Accept |
| link | <https://crawlcheck.io/llms.txt>; rel="describedby"; type="text/plain", <https://crawlcheck.io/.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json", <https://crawlcheck.io/guides/hsts-audit-missing-header-preload-ineligible-and-removed-entries>; rel="cite-as", <https://crawlcheck.io/.well-known/http-message-signatures-directory>; rel="http-message-signatures-directory" |
| cache-control | public, max-age=900, stale-while-revalidate=2400 |
| bytes | 18,235 |
| words | 2,614 |
| sha-256 | 826baca4e16695d0b7522971002ebdaf25b2d5ab9046c6147a4a67f9720684bb |
| repr-digest | sha-256=:gmuspOFmldC3UilxAC69ryWy1auQRsYUekpn+XIGhLs=: ← matches the body shown |
| signature-input | sig1=("@authority";req "@path";req "repr-digest" "content-type");created=1795182611;expires=1795186211;keyid="1hHmE3fo1rg5Ig70wuFtJ5wBc7G-sUTgUJkxyPWFRps";alg="ed25519";nonce="8go9hUGBXQ-bWAwW7C0dgg";tag="crawlcheck-page" |
| signature | sig1=:+bx3WqJ6xT5kUln3Z4cSWVZtB/NSjNcgGvp30i8INLi0CtrppczupmDuIwdsA8e03Fu+hk+0fH5QhxGpGjnoBQ==: |
| verified | yes — signature by key 1hHmE3fo1rg5Ig70… from https://crawlcheck.io/.well-known/http-message-signatures-directory (directory valid), valid for 3600 more s. Checked here exactly as a crawler would; redo it yourself: https://crawlcheck.io/api/tool/wba |
Guides · 2026-09-30 · By [VSNARY | Emmanuel Orta](/author) # HSTS check: missing header, bad preload, removed entries Three HSTS problems you can read from one homepage response, how common each is across scanned sites, and why preloading is a separate decision from sending the header. Fetch your homepage over HTTPS and read the Strict-Transport-Security header. Three problems matter: no header at all; a preload token without a max-age of at least one year and includeSubDomains, which the preload list will reject or drop; and a domain that hstspreload.org lists as removed while the header still asks to be preloaded. Send the header on every HTTPS site. Treat preloading as a separate choice that takes months to undo, and only send the preload token if every subdomain will serve HTTPS for good. Check your own site free Check my site About a minute. First scan free with your email, nothing installed. On this page - What does the header do? - How do you check your header? - What does the preload list status tell you? - Should you preload at all? - How do you fix each case? - What does a homepage check miss? - What to check next to HSTS: cookies and certificates Three [HSTS](/glossary/hsts) problems can be read from a single homepage response, and across the sites CrawlCheck has scanned they turn up at very different rates. No Strict-Transport-Security header at all: 15.9% of scanned sites. A header that sends the `preload` [token](/glossary/token) without meeting the preload list’s requirements: 5% . And the quietest case, a domain the public preload list records as *removed* that still sends the token, asking for something the list has already taken back: 0.8% . This guide is how to check each one yourself and what to change. | What you see | Share of scanned sites | What it means | What to do | | --- | --- | --- | --- | | No header over HTTPS | 15.9% | Browsers can be talked down to plain HTTP on every visit | Add the header to HTTPS responses | | Preload token, header does not qualify | 5% | A request the list will reject, or an existing entry it will drop | Meet the requirements, or remove the token | | Preload token, list says removed | 0.8% | A promise the list no longer holds | Remove the token, or resubmit with a qualifying header | ## What does the header do? # A browser that receives `Strict-Transport-Security: max-age=31536000` over HTTPS remembers the host for a year and rewrites every later `http://` request to `https://` before it leaves the machine. The first visit is still unprotected; the browser preload list closes that gap by shipping the host inside the browser as already remembered. `includeSubDomains` extends the rule to every subdomain. `preload` is not an instruction to the browser at all: it is consent that a separate submission to the list reads. ## How do you check your header? # Fetch the homepage over HTTPS with `curl -sI https://example.com/` and read the `strict-transport-security` line. Read the homepage, because the first request is the one that counts. Then answer three questions: what `max-age` is in seconds, whether `includeSubDomains` is present, and whether `preload` is present. The preload list’s requirements are fixed: `max-age` of at least 31,536,000 seconds, `includeSubDomains`, and the `preload` token. Convert `max-age` to days to see a shortfall plainly: 15,768,000 seconds is 182 days against a required 365. A header with the token and that value is asking to be preloaded while not qualifying. CrawlCheck’s free HSTS header generator builds the header from your answers to the three questions and will not add the `preload` token unless the other two requirements are met. Three questions about the header, then one lookup. Only a preload token the header cannot support, or a list entry the header no longer backs, needs action. ## What does the preload list status tell you? # The header cannot say whether the domain is on the list, so look it up at hstspreload.org, which keeps a public status for every domain: unknown, pending, preloaded or removed. Read it beside the header you just fetched. | Header sent | List status | What happens | | --- | --- | --- | | none, over HTTPS | any | no protection after the first visit either | | max-age under a year, no preload token | unknown | valid HSTS; nothing to fix | | preload token, max-age under a year | unknown | a submission would be rejected | | preload token, no includeSubDomains | pending | the pending submission will be rejected | | header no longer qualifies | preloaded | the entry will be dropped at a later list build, silently | | preload token present | removed | the token promises an entry that no longer exists | The serious row is a preloaded domain whose header has weakened: nothing breaks today, and the drop arrives without notice. A pending entry you did not submit needs nothing from you. ## Should you preload at all? # Sending the header is worth doing on any site that serves HTTPS. Preloading is a separate decision, because it takes months to undo. Once listed, every subdomain must serve HTTPS for as long as the entry lives, and the entry lives in shipped browser builds; a subdomain that later needs plain HTTP for a device, a legacy vendor or an internal tool cannot have it. Removal propagates only as browsers update, which takes the better part of a year for the long tail. Not preloading is not a defect. Sending a preload token the header cannot back is. ## How do you fix each case? # **No header.** Add it to HTTPS responses only; browsers ignore it over plain HTTP. Start with a short `max-age` such as 300 seconds while you confirm every subdomain and redirect chain works over HTTPS, then raise it. **Preload token that does not qualify.** Decide on intent. If you want preload, raise `max-age` to 31536000 and add `includeSubDomains`. If you do not, remove `preload`; sending it without qualifying is the worst of both. **Token sent, list says removed.** Remove the token. Continuing to send it does not restore the entry; only a fresh submission with a qualifying header does. ## What does a homepage check miss? # A subdomain that serves a different header, an `http://` redirect that lands somewhere other than the HTTPS homepage, and the list contents inside a particular browser build are all outside a homepage reading. [The status-code guide](/guides/http-status-codes-what-a-crawler-actually-gets) covers the redirect side, and [the CSP rollout guide](/guides/content-security-policy-rollout-report-only-to-enforced) covers the security header that is harder to get right. The rates above are live; the [dataset page](/data) has the rest. ## What to check next to HSTS: cookies and certificates # HSTS makes the browser use HTTPS. Whether a session stays private also depends on how cookies are scoped and on which certificates exist for the domain. | Check | Why it matters | How to check it | Reference | | --- | --- | --- | --- | | Cookie prefixes (`__Host-`, `__Secure-`) | The browser enforces extra rules: `__Host-` cookies must be Secure, set for the exact host, with `Path=/` | Read the `Set-Cookie` headers on sign-in and check the attributes match the prefix | [MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Set-Cookie) | | Partitioned cookies | A cookie stored per top-level site, for embedded services that need state | Check the `Partitioned` attribute and test the embedded flow; it also requires `Secure` | [MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Set-Cookie) | | [Public suffix](/glossary/public-suffix) | Defines where one site ends and another begins (`co.uk`, many hosting domains) | Compare the host with the Public Suffix List rather than assuming the last two labels are the site | [MDN](https://developer.mozilla.org/en-US/docs/Glossary/Site) | | [Schemeful same-site](/glossary/schemeful-same-site) | HTTP and HTTPS versions of a domain count as different sites for cookies | When a cookie goes missing between pages, compare both scheme and domain | [MDN](https://developer.mozilla.org/en-US/docs/Glossary/Site) | | TLS version and downgrade | Old protocol versions weaken the connection HSTS insists on | Test which TLS versions the server accepts; an HTTP-to-HTTPS redirect is a separate check | [RFC 9846](https://www.rfc-editor.org/rfc/rfc9846.html) (TLS 1.3, replaced RFC 8446 in 2026); [RFC 9851](https://www.rfc-editor.org/rfc/rfc9851.html) (TLS 1.2 gets no new features); [RFC 10015](https://www.rfc-editor.org/rfc/rfc10015.html) (obsolete TLS 1.2 key exchange deprecated) | | [Encrypted Client Hello](/glossary/encrypted-client-hello) | Hides which site a visitor is connecting to from anyone watching the network | Look for an ech= key in the domain's DNS HTTPS record; the [CDN](/glossary/cdn) usually publishes it | [RFC 9849](https://www.rfc-editor.org/rfc/rfc9849.html) (ECH); [RFC 9848](https://www.rfc-editor.org/rfc/rfc9848.html) (publishing it in DNS) | | [Certificate Transparency](/glossary/certificate-transparency) | Every public certificate for the domain is logged, so an unexpected one can be spotted | Search the logs for the domain (for example crt.sh) and investigate any issuer you do not use; a log entry alone is not a compromise | [MDN](https://developer.mozilla.org/en-US/docs/Web/Security/Defenses/Certificate_Transparency) | Read live from the same counters the dataset page uses, at the moment this page was served. Bars are scaled to the largest value shown, not to 100%. Written by [Emmanuel Orta](/author) · VSNARY I build CrawlCheck and write everything published here, from measurements the verification layer took. Where a post got something wrong, the correction stays in the post. [Every post, and where I got it wrong →](/author) · [Follow on LinkedIn →](https://www.linkedin.com/in/emmanuelorta) Share this finding: [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fhsts-audit-missing-header-preload-ineligible-and-removed-entries%3Futm_source%3Dlinkedin%26utm_medium%3Dsocial%26utm_campaign%3Dshare)[X](https://twitter.com/intent/tweet?url=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fhsts-audit-missing-header-preload-ineligible-and-removed-entries%3Futm_source%3Dx%26utm_medium%3Dsocial%26utm_campaign%3Dshare&text=HSTS%20check%3A%20missing%20header%2C%20bad%20preload%2C%20removed%20entries)[Email](mailto:?subject=HSTS%20check%3A%20missing%20header%2C%20bad%20preload%2C%20removed%20entries&body=https%3A%2F%2Fcrawlcheck.io%2Fguides%2Fhsts-audit-missing-header-preload-ineligible-and-removed-entries)[Follow Emmanuel on LinkedIn](https://www.linkedin.com/in/emmanuelorta) ## Questions this post answers ### Does HSTS protect a visitor's first request? No. The header is read only after a successful HTTPS response, so the first plain-HTTP request is unprotected. The browser preload list covers that first request, at the cost of a commitment that takes months to reverse. ### What does max-age need to be for the HSTS preload list? At least 31,536,000 seconds, which is one year, together with includeSubDomains and the preload token. A shorter max-age with the preload token present does not qualify. ### Is it a problem if my site sends HSTS without the preload token? No. Preloading is optional and hard to undo, so leaving the token out is a valid choice. The problem cases are a preload token the header cannot support, and a listed domain whose header no longer qualifies. ### Why check hstspreload.org as well as the header? The header cannot say whether the domain is on the list. A preloaded domain that later weakens its header is dropped silently at a future list build, and a domain marked removed that still sends the token is asking for something the list has withdrawn. Only the list status shows either. ### Should the HSTS header be sent over plain HTTP too? No. Browsers ignore Strict-Transport-Security received over HTTP. Send it only on HTTPS responses and make the HTTP side a redirect to HTTPS. ### How do I get my domain off the HSTS preload list? Remove the preload token, then request removal at hstspreload.org. The removal reaches visitors only as their browsers update, which can take most of a year, so every subdomain must keep serving HTTPS until then. ## Related findings [2026-10-18 How to check if your website supports HTTP/3 A site advertises HTTP/3 in its Alt-Svc header and its DNS HTTPS record. How to check both, what google.com and others show, and w](/guides/http-3-how-to-check-whether-your-site-offers-it)[2026-09-08 Unquoted HTML attributes: why we said a viewport was missing Minified HTML with unquoted attributes read as having no viewport, canonical or description. All three were there. What was wrong](/blog/the-viewport-was-there-all-along)[2026-08-21 Analytics added 650 ms to every request: how we found it Our own visit recording added about 650 ms to every response. Two pages located it, one change fixed it, and how to check your ser](/blog/our-analytics-was-adding-650ms-to-every-request) The main product Found this on your own site? **We fix it for $100.** Scan free to see where you stand. The fix is one site, every finding implemented and re-measured, with a sealed before and after. [Fix it all — $100](https://buy.stripe.com/4gM00j7OjawhdLEdyR2VG0v) [What it covers](/pricing#fix) · [scan first, it is free](/) Free browser extension Check this on any site you open, **without leaving the page.** CrawlCheck for Chrome grades every site you visit A–F and shows, from your own browser, what search engines and AI assistants are actually served. Free, no account. [Add to Chrome — free](https://chromewebstore.google.com/detail/plgjengobhbdmbgckiiaaflgiepdbcpi?utm_source=crawlcheck.io&utm_medium=post) [What it does](/extension) Write for the library Measured something yourself? **Submit an article.** Send your finding, the data and the method. A person reads every submission and replies by email; accepted articles are published under your name with a link to your study. Free to submit. Articles are accepted on merit; no one can pay to have one published. [Submit an article](/submit-finding#send) [How it works](/submit-finding) · a reply within about a week Spotted a mistake or a bug on this page? [Report an error](/report-error?k=article). Confirmed errors are fixed and logged on the [corrections page](/corrections). ## Comments Comments are read before they appear. Nothing is published automatically, and no account is needed. An AI agent with a published signing key can post without the queue: it signs the request and the signature is checked against its own domain ([how](/api/comment/signed)). Those appear at once, marked *verified*. Send for review Writing about this? [Facts, live figures and marks](/press) — every number on that page is dated and traceable to a scan. [All guides](/guides) · [The dataset](/data) · [How the dataset works](/policy) We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io) #### How we measure, and our own machine files CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io). **Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing. **Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours. **Registry:** [MCP](/registry/mcp) · [AI policy](/registry/aipolicy) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [Conduct: ClaudeBot](/agents/conduct/claudebot) | **Blog:** [AI crawlers on Wix, Shopify, Squarespace, Webflow: 980 sites](/blog/what-ai-crawlers-get-from-wix-shopify-squarespace-webflow-sites) | **Glossary:** [entity map parity](/glossary/entity-map-parity) · [Metadata filtering](/glossary/metadata-filtering) · [Content type](/glossary/content-type) · [Referring domain](/glossary/referring-domain) · [hash-prefix lookup](/glossary/hash-prefix-lookup) · [nosnippet](/glossary/nosnippet) [Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
Same body as a file: /guides/hsts-audit-missing-header-preload-ineligible-and-removed-entries.md · as data: /feed.json · what we tell agents site-wide: /llms.txt, /agents.md. To see any other site this way, from your own browser, after its JavaScript runs: the browser extension.