Rulebook

# The rules behind every answer an agent receives.

Every finding in a Resolve answer names its rule code and the revision that decided it, and every decision receipt names the engine and a digest of its rule tables. This page is what each of those means, so an agent, or the person behind it, can read the reason as well as the verdict.

111

rules a finding can cite

82

graded on every site scan

10

revised, old revisions kept

9,304

counted scans behind each share

For agents

## Three things in an answer this page explains

Each one is also a file an agent can fetch.

1 · A finding

### Code and revision

Each item in an answer’s `findings` carries `code` and `rule_revision`. `/rules/<CODE>` opens the rule here; the revision says which wording decided it, and a replay uses that one, never today’s.

The rules[/api/rules](/api/rules)

2 · A decision

### Engine and rule digest

A decision receipt names `engine.version` and `engine.rules_sha256`. The tables that digest covers, the action templates, the ranking and how long a receipt holds, are published so anyone can recompute it.

[/rules/preflight.json](/rules/preflight.json)[The receipt format](/spec/resolve)

3 · A grade

### Level and effect

Every graded finding lowers its section’s score; an open finding at medium or above also caps the letter. A self-audit rule is a defect in our own record, never in the scanned site.

Self-audit rules[Schema](/schemas/rulebook.json)

[/api/rules](/api/rules) (JSON) · [schema](/schemas/rulebook.json) · [decision rule tables](/rules/preflight.json) · [compatibility policy](/docs/api/compatibility) · [trust](/trust)

Each finding has a level from info to critical. Every graded finding lowers the score of its section; an open finding at medium or above also caps the letter grade, so fix those first. A rule that changes what it reports gets a new revision; a record keeps the revision that decided it, and a replay uses that revision, never today's.

Measured

## How often each decision rule was right

1 outcome scored in 90 days — too few to estimate a rate; the rows are the raw reports, and one report counts under every rule that fired in its receipt.

Agents that act on a decision receipt can report what happened. Each report is scored against the rules that fired in that receipt: for a warn, confirmation or block, *hit* is the share of acted-on decisions where acting went wrong; for allow, *held* is the share where it went fine. A block that stopped the agent leaves no outcome, so cautionary rates cover the agents that acted anyway. No blocklist publishes this.

| Rule | Effect | Outcomes | Not ok | Rate |
| --- | --- | --- | --- | --- |
| decision | require_confirmation | 1 | 1 | 100% hit |
| resolve.actions.transact | require_confirmation | 1 | 1 | 100% hit |

[/api/v1/decisions/accuracy](/api/v1/decisions/accuracy) (JSON, 90 days) · every new receipt carries the same figures next to each rule that fired (`steps[].measured`) · report outcomes at [/api/v1/outcome](/docs/api#api-v-outcome) with the receipt’s `decision_sha256`.

Every rule

## 111 rules in 17 families

Open a rule for what it means, how to fix it and how many scans carried it.

Site, hosts and page 18 Headers 4 Mostly code 3 Machine files 12 Other 9 Agentic resource discovery (ARD) 4 WebMCP form tools 2 Trust path 2 NAP and claims 3 Robots 17 Delivery parity 9 JSON-LD and entities 3 Answer-engine citations 5 Capabilities 6 Self-audit: defects in our own record 12 MCP servers 1 High-risk field watch 1

## Site, hosts and page

Code Title Level Scans Revision

#### CSP_REPORTING_BROKEN CSP violation reporting is misconfigured info 0.3% rev 1

Why it matters The Content-Security-Policy header declares where to send violation reports, and the declaration is wrong. `report-to` takes the NAME of a group defined in a separate header — given a URL it is silently ignored, so nothing is reported at all. And `report-uri` aimed at your own origin turns every violation into another request to the page you are already trying to keep fast. Neither breaks the page; both mean the reporting you think you have is either absent or self-inflicted. Fix The Content-Security-Policy report endpoint does not accept reports. Fix the report-uri / report-to target or remove the directive.
effort: config Share of scans 0.3% — 24 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=CSP_REPORTING_BROKEN) · [glossary: content-security-policy](/glossary/content-security-policy)

#### BROKEN_INTERNAL_LINK Links on this site point at pages that do not answer medium 2.7% rev 1

Why it matters A link on this site was followed and the target answered 404 or 410. A crawler spends budget on every one of these and learns nothing; a reader following one gets an error page. The usual causes are a route renamed without updating the links to it, and a nav or footer that renders links to authenticated-only routes for everyone — those answer 404 when signed out, which is what a crawler always is. Only 404 and 410 count here. A link answering 401, 403 or 429 means something declined to answer CrawlCheck — bot protection, a login wall, a rate limit — which is a fact about the request, not about the link, so those are counted and never reported as broken. Links are sampled rather than exhaustively crawled, so this is a floor and not a full link audit. Fix Fix or remove the internal links that return 404 so crawlers and visitors are not sent to dead pages.
effort: content Share of scans 2.7% — 255 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=BROKEN_INTERNAL_LINK) · [glossary: broken-link](/glossary/broken-link) · [glossary: internal-link](/glossary/internal-link) · [glossary: dead-anchor](/glossary/dead-anchor)

#### BROKEN_EMBED An embedded frame on the homepage cannot load low 0.2% rev 1

Why it matters A frame, map or player on the page was fetched the way a browser loads it, and its source answered an error or forbade being shown inside this site (X-Frame-Options or Content-Security-Policy frame-ancestors). Visitors see an empty box or a browser error where it should be, and a crawler that renders the page sees the same gap. The usual causes are a deleted video or form, and a map or page address pasted as a frame instead of the provider's embed address. Fix Replace each listed source with the provider's embed address (Google Maps: Share, then Embed a map; YouTube: youtube.com/embed/`<id>`) or remove the frame. A source that forbids framing cannot be fixed from your page: link to it instead.
effort: content Share of scans 0.2% — 22 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=BROKEN_EMBED)

#### AD_DENSITY_HIGH Ads take too much of the page on a phone low 0% rev 1

Why it matters The homepage was rendered at phone width and its ads measured: more than 30% of the page height, more than 30% of the first screen, or a sticky ad over 30% of the screen. Those are the limits of the Coalition for Better Ads mobile standard, which Chrome's ad filter enforces, and a page over them reads as made for ads. Visitors scroll past ads to reach the content, and search engines that judge page experience see the same layout. Fix Cut the ad slots on the mobile layout until ads stay under 30% of the page height and of the first screen, and keep any sticky ad under 30% of the screen.
effort: build Share of scans 0% — 3 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=AD_DENSITY_HIGH)

#### MOBILE_INTERSTITIAL A pop-up covers the page on a phone when it loads medium 0.3% rev 1

Why it matters The homepage was rendered at phone width and, on load, a fixed overlay covered at least half of the first screen. Google treats a pop-up that covers the main content as soon as a visitor arrives from search as an intrusive interstitial, and visitors who wanted the answer leave. Cookie consent and age-verification notices are exempt and are never counted here. Fix Show the pop-up later (after a scroll or on exit), make it a banner that leaves most of the screen free, or remove it on mobile. Consent and age notices may stay.
effort: build Share of scans 0.3% — 31 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=MOBILE_INTERSTITIAL)

#### SECRETS_EXPOSED Private files or live keys are readable by anyone high 0.3% rev 1

Why it matters A file the site should never serve (a .git or .svn folder, a .env file, a config backup) answered with its own contents, or a live API key or private key is written into the homepage or the site's own scripts. Anyone who requests the same address gets the same bytes: database passwords, keys and source history are the first things an attacker tries. Each path was requested once and counts only when it answers with the file itself, never with a page. Fix Deny the listed paths at the server (or delete the files), then treat every password and key they contained as public: change the passwords and revoke and reissue each key shown. Remove keys from page code and scripts; a key a browser needs must be a publishable one.
effort: config Share of scans 0.3% — 30 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=SECRETS_EXPOSED)

#### IMAGES_BLOCKED_GOOGLEBOT_IMAGE robots.txt keeps Google's image crawler off the site's images low 0.8% rev 1

Why it matters Images on the homepage are under a path the robots.txt group for Googlebot-Image disallows (that crawler obeys its own group, else Googlebot's, else the * group). Those images cannot appear in Google Images or as image results, and a page whose pictures are blocked loses them in visual search. The usual cause is a Disallow on an uploads or assets folder written to save crawl budget. Fix Remove the Disallow that covers the image folder, or add an Allow for it in the group Googlebot-Image follows.
effort: config Share of scans 0.8% — 76 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=IMAGES_BLOCKED_GOOGLEBOT_IMAGE)

#### IMAGE_FORMAT_MISMATCH Images are not the format their file names say info 1.8% rev 1

Why it matters An image's first bytes say one format while its file name says another (a .jpg that is a PNG, or an image address that returns an HTML page). It was fetched without asking for WebP or AVIF, so a CDN that converts on request is not counted. Browsers usually cope; image indexers and some clients trust the name, and an image address that returns a page is a broken image. Fix Save each listed image under the extension of its real format (or re-export it in the format its name says), and fix image addresses that return a page.
effort: content Share of scans 1.8% — 168 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=IMAGE_FORMAT_MISMATCH)

#### DATE_MOVED_NO_CHANGE The page's date changed but its words did not info 1.4% rev 1

Why it matters Between two of CrawlCheck's reads, the page's declared modification date (dateModified or article:modified_time) moved forward while the main content, read with navigation, ads and dates removed, stayed the same. A date that moves without new words claims freshness the page does not have; engines that compare the two learn to ignore the date. Fix Update the modification date only when the content changes; turn off plugins or templates that stamp the current date on every build.
effort: config Share of scans 1.4% — 132 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=DATE_MOVED_NO_CHANGE)

#### CHANGED_DATE_STAYED The page's words changed but its date did not info 0.8% rev 1

Why it matters Between two of CrawlCheck's reads, the page's main content changed (read with navigation, ads and dates removed) while the modification date it states (dateModified or article:modified_time), and the sitemap's date for the page when one was read, stayed the same. An engine that decides what to re-read from those dates is told nothing changed, so it keeps the old copy. Fix When the content changes, update dateModified (and article:modified_time) and the sitemap lastmod for the page in the same publish; most CMSs do this when the page is saved rather than edited in place by a widget or a cache.
effort: config Share of scans 0.8% — 78 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=CHANGED_DATE_STAYED)

#### PAGES_SNIPPET_RESTRICTED Pages meant to be found hide their own search snippet low 0.1% rev 1

Why it matters Pages that allow indexing also tell search engines to show little or nothing of them: nosnippet, a max-snippet under 50 characters, max-image-preview:none, or an unavailable_after date that has passed. The page can rank but appears as a bare title, with no description and no image, and answer engines that respect nosnippet will not quote it. It is usually a theme or SEO-plugin setting applied site-wide by mistake, or a header added at the server. Fix Remove nosnippet and max-snippet / max-image-preview limits from pages that should show in search (check the SEO plugin's advanced robots settings and any X-Robots-Tag header), or noindex the page if it should not appear at all.
effort: config Share of scans 0.1% — 7 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=PAGES_SNIPPET_RESTRICTED)

#### REDIRECT_CHAIN The homepage is reached through a chain of redirects info 22.6% rev 1

Why it matters Reaching the homepage takes more than one redirect, for example http to https and then the domain to www. Each hop is another request and round trip before a crawler or visitor gets the page, temporary hops (302, 307) tell search engines to keep the old address, and Google stops following after ten. Fix Send every old address straight to the final one in a single permanent (301 or 308) redirect: in the host, CDN or plugin rules, redirect http://domain and http://www straight to the final https address instead of chaining scheme, host and path fixes.
effort: config Share of scans 22.6% — 2102 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=REDIRECT_CHAIN)

#### HOST_DUPLICATE_200 www and the apex both serve the site instead of one redirecting medium 5.5% rev 1

Why it matters Both www and the bare domain answer 200 with the same pages, so every URL on this site exists at two addresses and a crawler fetches the whole site twice. A canonical tag does not prevent this: it is read after the response arrives, so the second fetch has already been spent. Nothing looks broken from a browser, which is why this survives for years. The fix is a 301 from one host to the other so there is one address per page. Fix Pick one host (apex or www) and redirect the other to it with a 301 so there is one copy of every page.
effort: config Share of scans 5.5% — 515 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=HOST_DUPLICATE_200) · [glossary: canonical](/glossary/canonical) · [glossary: canonical-url](/glossary/canonical-url) · [glossary: host-duplicate](/glossary/host-duplicate)

#### ORIGIN_REFUSED_SCANNER The homepage and robots.txt both refused this client, so the site was never measured critical 0.2% rev 1

Fix Allow CrawlCheck's address through the security layer, or scan from the browser extension, which uses your own address.
effort: config Share of scans 0.2% — 14 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ORIGIN_REFUSED_SCANNER) · [glossary: refusal](/glossary/refusal) · [glossary: datacenter-ip](/glossary/datacenter-ip) · [glossary: vantage-point](/glossary/vantage-point)

#### DOMAIN_IS_ALIAS This domain is an alias: every path redirects to another site low 1.1% rev 1

Why it matters Every request to this hostname is redirected to a different site, so nothing measured here describes a site at this address. Scan the destination instead; the redirect itself is the only fact about this domain. Fix Nothing to fix on this domain: it forwards to another site. Scan the destination host instead. Share of scans 1.1% — 102 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=DOMAIN_IS_ALIAS) · [glossary: alias-domain](/glossary/alias-domain)

#### HOMEPAGE_REDIRECTS_OFF_HOST The homepage sends crawlers to a different hostname medium 3.4% rev 1

Why it matters A request for this domain's homepage answers with a redirect to a hostname outside its own www/apex pair - a regional storefront, a checkout host, a vendor platform. This scan stopped at the redirect; the destination was not evaluated as part of this report, so nothing on the other host counts for this domain here. A deliberate forward (a brand domain, a migration) is a choice this scan cannot distinguish from a misrouted one; what it can say is that a crawler reading this hostname alone finds no page. Fix Point the domain at the site it is meant to serve, or make the redirect a single permanent hop to the canonical host.
effort: config Share of scans 3.4% — 317 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=HOMEPAGE_REDIRECTS_OFF_HOST) · [glossary: off-host-redirect](/glossary/off-host-redirect)

#### ORIGIN_BLOCKED_BEHIND_CACHE A security layer refused this file — source unconfirmed medium 0.2% rev 1

Why it matters The cached copy is fine; behind it the origin refused this request. IMPORTANT LIMIT: a scanner runs from one IP, and application firewalls block by IP reputation — so this may mean the site refuses crawlers, or it may only mean the site refuses CrawlCheck specifically. The two are indistinguishable from outside. Confirm in the server access logs, grouped by source IP, before acting on it. Fix The cached copy works but the origin refuses. Fix the origin block before the cache expires, or the next miss serves the block to everyone.
effort: config Share of scans 0.2% — 20 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ORIGIN_BLOCKED_BEHIND_CACHE) · [glossary: edge-cache](/glossary/edge-cache) · [glossary: cache-buster](/glossary/cache-buster)

#### STAGING_LEAK A staging hostname is exposed in the page source low 0% rev 1

Why it matters A staging hostname appears in the page source. If that hostname is reachable and indexable it competes with the production site for the same content. Fix Remove staging or development hostnames, paths and comments from the production HTML, and block the staging site from crawlers.
effort: config Share of scans 0% — 3 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=STAGING_LEAK) · [glossary: staging-leak](/glossary/staging-leak)

## Headers

Code Title Level Scans Revision

#### HSTS_MISSING HTTPS is served without a Strict-Transport-Security header info 15.9% rev 1

Why it matters Strict-Transport-Security tells a browser that has visited once to use HTTPS for every later request without trying HTTP first. Without it, a typed or linked http:// address makes one plaintext request before the redirect. It does not change what crawlers receive or how the site ranks; it is a small, reversible hardening step, and it is listed at the lowest severity for that reason. Fix Send Strict-Transport-Security: max-age=15552000; includeSubDomains on HTTPS responses (Cloudflare: SSL/TLS, Edge Certificates, HSTS). Leave preload off unless you intend a change that takes months to undo.
generate the fix: `/api/fix/hsts?domain={domain}`
effort: config Share of scans 15.9% — 1479 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=HSTS_MISSING) · [glossary: hsts](/glossary/hsts)

#### HSTS_PRELOAD_INELIGIBLE The HSTS header asks for browser preloading but does not qualify for it low 5% rev 1

Why it matters The header carries the `preload` token, which is a request to be compiled into the HTTPS-only list shipped inside browsers. That list only accepts a header with max-age of at least 31536000 (one year) AND includeSubDomains AND preload. As sent, the token is ignored: the site is declaring an intent it does not meet, and if it ever was on the list, this header is what gets it removed. The rest of the HSTS header still works. Fix Either raise max-age to 31536000 and keep includeSubDomains so the preload token qualifies, or remove the preload token. As sent it does nothing.
generate the fix: `/api/fix/hsts?domain={domain}`
effort: config Share of scans 5% — 465 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=HSTS_PRELOAD_INELIGIBLE) · [glossary: hsts-preload-list](/glossary/hsts-preload-list)

#### HSTS_PRELOAD_REMOVED This domain was removed from the browser HSTS preload list low 0.8% rev 1

Why it matters The browser preload list (hstspreload.org) records this domain as removed. Domains are removed when the header stops meeting the requirements, or on request. Browsers that shipped with the old list still force HTTPS on every subdomain; newer ones do not, so behaviour now differs by browser version. Either the removal was intended and the `preload` token should go, or it was not and the header needs to qualify again before resubmitting. Fix If preloading was never intended, remove the preload token. If it was, restore a qualifying header (max-age at least 31536000, includeSubDomains, preload) and resubmit at hstspreload.org.
effort: config Share of scans 0.8% — 74 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=HSTS_PRELOAD_REMOVED)

#### STALE_CACHE_SERVED A cached copy of this page older than an hour was served low 13.5% rev 3 · 2026-09-30

Why it matters The page handed to us carried a cache age older than an hour. The detail says whether the copy still matches what the origin produces now: if it matches, nothing is broken today; if it differs, an edit is already waiting behind the cache. It matters the moment you change something: an edit, a new price, a corrected phone number stays invisible to every visitor and every crawler for about that long, and nothing in your dashboard says so. Fix Lower the cache lifetime on HTML (an hour or less), or purge the edge and origin caches after every publish so crawlers see the current copy.
generate the fix: `/api/fix/cache?domain={domain}`
effort: config Share of scans 13.5% — 1256 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=STALE_CACHE_SERVED) · [glossary: stale-response](/glossary/stale-response) · [glossary: age-header](/glossary/age-header) · [the fix end to end on a real site](/workflow/stale-cache-served)

## Mostly code

Code Title Level Scans Revision

#### CONTENT_NEEDS_JAVASCRIPT The page’s content only exists after JavaScript runs medium 2.1% rev 1

Why it matters The HTML the server sends is an application shell: a framework mount point and a script bundle, with almost no readable text. A browser runs the bundle and the page appears. A crawler that does not execute JavaScript — which is most AI crawlers and every training-data fetch — receives the shell and quotes nothing. This is measured on the bytes the server delivered, not on a rendered view; a framework by itself is not the defect, an empty delivery is. Fix Render the page on the server or at build time (Next.js/Nuxt/SvelteKit SSR or SSG, or a prerender step) so the delivered HTML already contains the text. Check by fetching the page with curl: the words a crawler needs must be in that response.
effort: build Share of scans 2.1% — 196 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=CONTENT_NEEDS_JAVASCRIPT) · [glossary: render-dependence](/glossary/render-dependence) · [glossary: render-dependence-gap](/glossary/render-dependence-gap) · [glossary: render-gap](/glossary/render-gap)

#### PAGE_IS_MOSTLY_CODE Almost nothing a machine receives from this page is readable text medium 28.4% rev 1

Why it matters A crawler pays for every byte it fetches and can only quote the text. Under 5% readable content on the homepage means an answer engine downloads the whole page and comes away with almost nothing it can quote. The content-ratio row above passes at 20%, so without this a page at 2% and a page at 19% were scored identically. Fix Move inline CSS and JavaScript into external files, drop unused page-builder styling, and make sure the words a visitor reads are in the HTML itself.
generate the fix: `/api/fix/code?id={report_id}`
effort: build Share of scans 28.4% — 2642 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=PAGE_IS_MOSTLY_CODE) · [glossary: payload-text-share](/glossary/payload-text-share) · [the fix end to end on a real site](/workflow/page-is-mostly-code)

#### PLACEHOLDER_CONTENT Unfinished placeholder text is live on this page high 0.4% rev 1

Why it matters Template text that was never replaced is being served to visitors and to crawlers. An answer engine cannot tell draft copy from real copy — it reads and may quote whatever is there. Placeholder in a title or meta description is worse than in body copy, because that is the text a search result shows and an assistant repeats. Fix Replace the placeholder text (lorem ipsum, 'test content', template copy) with the real page, or unpublish the page until it is written.
effort: content Share of scans 0.4% — 34 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=PLACEHOLDER_CONTENT) · [glossary: placeholder-content](/glossary/placeholder-content)

## Machine files

Code Title Level Scans Revision

#### SITEMAP_NO_LASTMOD The sitemap does not say when anything changed low 7.4% rev 3 · 2026-11-14

Why it matters The sitemap lists URLs but carries no usable lastmod dates: either none at all, or one date copied onto every URL, which says everything changed at once and so says nothing. Either way it tells a crawler what exists and not what changed. A crawler with a budget uses lastmod to decide what to re-fetch; without it, the choice is made for you — usually by re-crawling little and late. This is the change signal every crawler already understands, and unlike a push protocol it needs no key, no account and no participation from the search engine. Fix Give each sitemap entry its own `<lastmod>`: the date that page last changed, not the date the sitemap was built. Then a crawler can prioritise the pages that changed.
generate the fix: `/api/fix/sitemap?domain={domain}`
effort: content Share of scans 7.4% — 684 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=SITEMAP_NO_LASTMOD) · [glossary: sitemap](/glossary/sitemap) · [glossary: sitemap-lastmod](/glossary/sitemap-lastmod)

#### MACHINE_CHAIN_HEAVY The machine files cost more than the page info 22.7% rev 4 · 2026-11-14

Why it matters A crawler reads robots.txt, the sitemap, llms.txt and the agent files before the first page, and pays for every 404 that answers with a full page. Keep machine files as small as their job allows and serve absent paths with a short 404 body. Fix Answer absent machine paths (/agents.md, /.well-known/...) with a short plain 404 body instead of a full HTML page: that is the part that is pure waste. Then keep robots.txt, llms.txt and the sitemap index as small as their job allows; a large sitemap index is normal on a large site.
generate the fix: `/api/fix/chain?id={report_id}` Share of scans 22.7% — 2108 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=MACHINE_CHAIN_HEAVY) · [glossary: machine-layer](/glossary/machine-layer)

#### MACHINE_FILE_CACHE_SPLIT This file answers differently depending on whether the cache is hit high 0% rev 1

Fix Make the apex and www hosts serve the same machine files, or redirect one host to the other so there is only one copy to keep current.
effort: config Share of scans 0% — 2 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=MACHINE_FILE_CACHE_SPLIT) · [glossary: edge-cache-pinning](/glossary/edge-cache-pinning) · [glossary: cache-split](/glossary/cache-split)

#### MACHINE_FILE_CACHE_STALE The cache is still serving a file the origin no longer has low 0.4% rev 1

Fix Give robots.txt, sitemap.xml and llms.txt a short cache lifetime (five minutes is plenty) so an edit reaches crawlers the same day.
effort: config Share of scans 0.4% — 41 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=MACHINE_FILE_CACHE_STALE)

#### SITEMAP_IS_HTML The sitemap returns HTML, not XML critical 0.3% rev 1

Why it matters A crawler that cannot parse the sitemap cannot enumerate the site. It falls back to following links, so deep and newly published pages go undiscovered. Fix The sitemap URL is returning an HTML page. Serve XML at that path, or update robots.txt to point at the sitemap that actually exists.
effort: config Share of scans 0.3% — 28 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=SITEMAP_IS_HTML) · [glossary: soft-404-machine-file](/glossary/soft-404-machine-file)

#### SITEMAP_BLOCKED The sitemap is refused high 1.9% rev 1

Why it matters SAME LIMIT AS ABOVE: if a named application firewall appears in the response, this may be blocking CrawlCheck's IP rather than crawlers. Confirm in server logs grouped by source IP. Otherwise: the server refuses this sitemap rather than returning it or a clean 404. Crawlers cannot enumerate the site and fall back to link-following, so deep and newly published pages go undiscovered. Fix Allow the sitemap URL through the security layer so crawlers can read it.
effort: config Share of scans 1.9% — 178 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=SITEMAP_BLOCKED)

#### SITEMAP_ORIGIN_ERROR The origin returned a server error on the sitemap path low 0.2% rev 1

Fix The sitemap URL returned a server error. Regenerate it (most CMSs have a setting) and check the URL loads in a private window.
effort: config Share of scans 0.2% — 20 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=SITEMAP_ORIGIN_ERROR) · [glossary: sitemap-origin-error](/glossary/sitemap-origin-error)

#### SITEMAP_UNREADABLE Whether a sitemap exists could not be determined — every path was refused low 0.5% rev 1

Why it matters A refusal is not an absence. The paths where a sitemap would live answered with a block, so this scan cannot say whether one exists — and it does not guess. Confirm from a second network, and check server logs grouped by source IP: if the block is aimed at CrawlCheck's address, the finding is about our access; if named crawlers see the same refusal, the sitemap is unreachable to them too. Fix Find out why the sitemap paths were refused (security layer, bot rule, rate limit) and confirm from a second network or your server logs; validate the XML only once the file can be fetched.
effort: build Share of scans 0.5% — 49 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=SITEMAP_UNREADABLE) · [glossary: sitemap-index](/glossary/sitemap-index)

#### SITEMAP_DECLARES_NOTHING A sitemap answers 200 and lists nothing low 1.8% rev 1

Why it matters The file parses as a sitemap and contains no URLs. A crawler that follows the declaration fetches it on every visit and learns nothing, and Search Console reports 0 discovered URLs for it. The usual cause is a generator with an empty source — an entity-map, image or news sitemap whose feed returned nothing — or a rewrite rule left behind. Either populate it or stop declaring it. Fix Populate the empty sitemap or remove it from robots.txt and from the sitemap index. A file that lists nothing should not be declared.
effort: build Share of scans 1.8% — 170 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=SITEMAP_DECLARES_NOTHING)

#### NO_SITEMAP_FOUND No XML sitemap found high 9.8% rev 1

Why it matters Without a sitemap, discovery depends entirely on internal linking. Orphaned and recently published pages may never be found. Fix Publish a sitemap at /sitemap.xml (most CMSs generate one) and name it in robots.txt.
effort: build Share of scans 9.8% — 911 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=NO_SITEMAP_FOUND)

#### LLMS_LINK_BROKEN llms.txt links to pages that do not load low 1.4% rev 1

Why it matters llms.txt is the map an AI agent reads to find your key pages. Some of the links in it answer 404, 410 or a server error, so an agent following the map lands on a dead page or gives up on it. Fix Fix or remove each dead link in llms.txt so every entry points at a page that answers 200.
effort: content Share of scans 1.4% — 132 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=LLMS_LINK_BROKEN) · [glossary: llms-txt](/glossary/llms-txt)

#### NO_LLMS_TXT No llms.txt info 34.1% rev 1

Why it matters Not a defect. llms.txt is an emerging convention for telling AI systems what a site is and which pages matter. Fix Publish /llms.txt: a short Markdown file naming the business, what it does, and the pages worth reading, served as text/plain or text/markdown.
generate the fix: `/api/tool/llms?domain={domain}&format=txt`
effort: content Share of scans 34.1% — 3173 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=NO_LLMS_TXT) · [the fix end to end on a real site](/workflow/no-llms-txt)

## Other

Code Title Level Scans Revision

#### SCRIPT_ONLY_LINKS Some pages can only be reached by a click handler, not a link low 0.1% rev 1

Why it matters Search and AI crawlers follow `<a href>` links. They do not click. A page that is reached only through a data-href attribute or an onclick that changes the address is, to a crawler, not linked at all: it is found only if the sitemap lists it, and it receives none of the link signals that tell a search engine or an AI system how important it is. This is the commonest way a site builds part of the “invisible web” by accident: product tiles, listing cards and menus wired in JavaScript. Only internal targets with no plain link anywhere on the pages read are counted, so a scripted card that duplicates a real link is not reported. Fix Make each scripted element a real link: wrap it in `<a href="/the-page">` (keep the click handler if you need it), so crawlers can follow it. Until then, list those pages in the sitemap.
effort: content Share of scans 0.1% — 13 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=SCRIPT_ONLY_LINKS)

#### FORM_ONLY_CONTENT Some listings can only be reached by filling in a form info 0.1% rev 1

Why it matters Crawlers do not fill in forms. A results page reached only by picking options in a filter form (a location, a category, a price band) and pressing submit is part of the “deep web”: the listings exist for people but no search engine or AI crawler can find them unless a plain link or the sitemap leads there. A lone site-search box is not counted, because search results are meant to stay out of indexes. Reported only when no plain link to the results page exists on the pages read. Fix Give the main results a crawlable route: plain links to category, location or filter pages (for example /services/denver), and list those pages in the sitemap. Keep the form for people.
effort: content Share of scans 0.1% — 10 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=FORM_ONLY_CONTENT)

#### FRONTIER_BLOAT Links send crawlers to the same pages under many different addresses low 1.2% rev 1

Why it matters A crawler keeps a queue of every address it has found and still has to fetch (its crawl frontier). Each distinct URL is a separate entry and a separate fetch, so one page linked under several query strings, a session id in every link, “add to cart” and “reply to comment” links, or a calendar that links one month further forever all fill that queue with addresses that teach it nothing. Search engines and AI crawlers spend a limited budget per site; what goes on duplicates is not spent on the pages that matter, and the duplicates compete with each other. Counted from real links on the pages read, not guessed from the sitemap. Fix Link every page by one address: drop session ids and tracking parameters from internal links, add rel="nofollow" (or a robots.txt Disallow) to add-to-cart, reply-to and calendar-paging links, and give sort/filter variants a canonical tag pointing at the plain page.
effort: content Share of scans 1.2% — 116 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=FRONTIER_BLOAT)

#### SCANNER_REFUSED_BROWSER_READ The site refuses plain automated requests; it was measured through a real browser low 0.8% rev 1

Why it matters The site’s security layer refused CrawlCheck’s plain HTTP request but served the page to a real browser. Most AI crawlers (GPTBot, ClaudeBot, PerplexityBot and others) make plain HTTP requests and do not run JavaScript, so a shield that tells the two apart this way may be refusing them too. This scan cannot verify that: it measured its own request from its own network, not a verified crawler. Confirm with your server logs or the crawler operators' own diagnostics before treating the site as blocked. The rest of this report was measured through the browser, so its findings describe the site, not the wall. Fix Decide which automated readers you want. If AI and search crawlers should reach the site, allow verified bots in the bot-protection settings (Cloudflare, Akamai and others have a verified-bot or signed-agent allow list) instead of challenging every client that does not run JavaScript.
effort: config Share of scans 0.8% — 77 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=SCANNER_REFUSED_BROWSER_READ)

#### SHOPPING_BLOCKED_STOREBOT robots.txt keeps Google's Shopping crawler off the store high 0% rev 1

Why it matters The site sells online, and robots.txt disallows Storebot-Google on its homepage or product pages (that crawler obeys its own group, else the * group). Google says rules for Storebot-Google affect all surfaces of Google Shopping, so those products cannot be listed in the Shopping tab or other Shopping results from the crawl. Googlebot being allowed does not cover it. The usual cause is a Disallow: / under User-agent: * written to keep unknown bots out, or a block-all-bots preset. Fix Add a User-agent: Storebot-Google group that allows the product pages (Allow: /), or remove the Disallow that covers them from the group it follows.
effort: config Share of scans 0% — 0 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=SHOPPING_BLOCKED_STOREBOT)

#### MACHINE_FILE_ACCEPT_DEPENDENT A machine file is served to browsers but not to crawlers medium 0% rev 1

Why it matters A browser asks for text/html and gets the file; a crawler asks with Accept: */* or a text type and gets an error page, so the file it was written for is invisible to it. The cause is usually a CDN, framework or redirect rule that sends non-HTML requests to the 404 handler. Serve the file whatever the Accept header says, then check with: curl -H 'Accept: */*' on the file's URL. Fix Serve the file the same way whatever Accept header the request carries: the same status and body for Accept: */* as for a browser. Share of scans 0% — 4 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=MACHINE_FILE_ACCEPT_DEPENDENT)

#### SITEMAP_LASTMOD_INVALID A sitemap lastmod is not a valid date low 0.1% rev 1

Why it matters A lastmod value in the sitemap is not in W3C date format (2026-11-06 or 2026-11-06T09:00:00Z). Crawlers ignore a date they cannot parse, so those pages lose the one change signal a sitemap carries. Fix Write every lastmod as a W3C date: YYYY-MM-DD, or a full timestamp with a time zone.
effort: build Share of scans 0.1% — 6 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=SITEMAP_LASTMOD_INVALID)

#### SITEMAP_URL_OFF_HOST The sitemap lists URLs on another host low 0.4% rev 1

Why it matters The sitemap names URLs on a different host from the sitemap itself. The sitemap protocol allows only URLs on its own host, so crawlers drop them; if the other host is a staging or old domain, it also advertises that copy. Fix List only URLs on this host. Give another host its own sitemap, declared in that host's robots.txt.
effort: build Share of scans 0.4% — 37 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=SITEMAP_URL_OFF_HOST)

#### SITEMAP_CHILD_BROKEN A child sitemap named in the sitemap index does not load medium 0.2% rev 1

Why it matters The sitemap index lists a child sitemap that answers 404, an error or an HTML page. Every page listed only in that child is missing from the sitemap a crawler reads, so new and deep pages there are found late or not at all. Fix Publish the missing child sitemap at the URL the index names, or remove that `<sitemap>` entry from the index. Then confirm every child loads as XML.
effort: build Share of scans 0.2% — 14 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=SITEMAP_CHILD_BROKEN)

## Agentic resource discovery (ARD)

Code Title Level Scans Revision

#### ARD_SOURCE_BROKEN The site advertises an ARD entry source that does not answer medium 0% rev 1

Why it matters Agentic Resource Discovery (ARD) lets an agent registry find a site's MCP servers, agent cards and skills. The site points at an entry source - an Agentmap line in robots.txt, a `<link rel="ard">` tag or a Link header - and that address does not return a JSON document. A consumer that honours the pointer (the specification says it must) gets nothing; a registry that follows it indexes nothing. Fix Make the advertised address answer 200 with the JSON manifest, or remove the Agentmap line, `<link rel="ard">` or Link header that points at it. Publish the manifest at /.well-known/ard.json, which every consumer reads.
effort: config Share of scans 0% — 1 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ARD_SOURCE_BROKEN)

#### ARD_MANIFEST_INVALID The ARD manifest breaks the specification medium 0.8% rev 1

Why it matters The site's ARD manifest (/.well-known/ard.json or the source it advertises) fails the rules every entry must meet: a urn:air:`<publisher>`:`<name>` identifier, a displayName, a media type, and exactly one of url or data. A registry rejects the entry, so the resource it describes is not discoverable; where the trust manifest names an identity that does not belong to the URN's publisher, the entry is rejected as namespace squatting. Fix Give every entry an identifier of the form urn:air:`<your-domain>`:`<name>`, a displayName, a type (an IANA media type such as application/mcp-server-card+json) and exactly one of url or data. If an entry carries a trustManifest, its identity must belong to the same domain as the URN. Validate with the official tester: npx ajv-cli validate -s ard-entry.schema.json -d ard.json.
effort: build Share of scans 0.8% — 74 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ARD_MANIFEST_INVALID)

#### ARD_MANIFEST_WARNINGS The ARD manifest is valid but its entries will not be found by search info 0.4% rev 1

Why it matters The ARD manifest validates, but entries are missing representativeQueries (or carry fewer than 2 or more than 5), use a renamed media type, or carry non-standard parameters. The semantic index a registry builds comes from representativeQueries: an entry without them is a valid catalogue row that search never returns. Fix Add 2 to 5 representativeQueries to every entry: the natural-language questions a person would ask that this resource answers. Rename application/mcp-server+json to application/mcp-server-card+json. Use the standard media types without extra parameters.
effort: content Share of scans 0.4% — 41 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ARD_MANIFEST_WARNINGS)

#### ARD_LEGACY_PATH The ARD manifest is served only under its old name info 0.3% rev 1

Why it matters The manifest is served only at /.well-known/ai-catalog.json or with rel="ai-catalog", the names ARD's predecessor used. Consulting those names is optional for consumers; a conformant consumer that reads only /.well-known/ard.json and rel="ard" does not find this site. Fix Serve the same document at /.well-known/ard.json and change the link relation to rel="ard". The old name can stay for older readers; the new one is the only one consumers must consult.
effort: config Share of scans 0.3% — 29 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ARD_LEGACY_PATH)

## WebMCP form tools

Code Title Level Scans Revision

#### WEBMCP_SCHEMA_INVALID A WebMCP form tool is missing what Chrome needs to register it low 0% rev 1

Why it matters WebMCP lets a page offer its forms to an AI agent as tools: a form with toolname and tooldescription becomes a tool, and each field a parameter. This page has a form with only one of the two attributes, or a required field with no name, so Chrome reports the tool as invalid and an agent cannot call it as written. Lighthouse's Agentic Browsing category scores this as a failure. Fix Give every annotated form both toolname (a short verb-noun name such as book_appointment) and tooldescription (one sentence saying what submitting it does), and give every required field a name attribute.
effort: content Share of scans 0% — 0 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=WEBMCP_SCHEMA_INVALID)

#### WEBMCP_SCHEMA_WARNINGS WebMCP form tools have fields an agent cannot identify info 0% rev 1

Why it matters The page's WebMCP form tools register, but some fields have no name or no description (toolparamdescription, a `<label>` or aria-description). An agent sees a parameter it cannot identify, or none at all, and fills it by guessing. Fix Give every field a name and a description: toolparamdescription="..." on the field, or a `<label>` tied to it with for=, or aria-description.
effort: content Share of scans 0% — 0 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=WEBMCP_SCHEMA_WARNINGS)

## Trust path

Code Title Level Scans Revision

#### TRUST_PATH_LOOKALIKE The site sends agents' data to a domain that imitates its own medium 0% rev 1

Why it matters Following this site the way an agent does (its forms, the MCP server and agent card it advertises, and the login service those name), a place that receives data is on a domain one or two characters off the site's own, written with look-alike characters, or made of the site's name plus a word such as -login or -secure. That is how a phishing or hijacked integration looks; it can also be a domain the owner registered on purpose. An agent cannot tell the difference and trusts it because the site pointed there. Fix If you own the domain, move the endpoint onto your own domain (or a subdomain of it) so nothing an agent follows looks like an imitation; if you do not, remove the link from the form, card or manifest that names it now and find out how it got there.
effort: config Share of scans 0% — 1 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=TRUST_PATH_LOOKALIKE)

#### TRUST_PATH_CLEARTEXT The site sends agents' data over plain http low 0% rev 1

Why it matters A form, tool endpoint or login service that this site points agents to is addressed with http:// rather than https://. Anything an agent submits there (customer details, a login, a token) travels unencrypted and can be read or altered by anyone on the network path. Fix Change the address to https:// in the form action, server card, agent card or protected-resource metadata, and serve that endpoint over TLS.
effort: config Share of scans 0% — 0 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=TRUST_PATH_CLEARTEXT)

## NAP and claims

Code Title Level Scans Revision

#### PHONE_SHARED_BY_OTHER_BUSINESSES Directories list this phone number for other businesses low 0.8% rev 1

Why it matters The site's phone number was searched in directories, and listings under other business names came back (names that are not a spelling of the site's own). One number behind several businesses is a known spam pattern in local search: it can merge the businesses, split their reviews, or make an engine distrust all of them. Old names and sister brands are the usual cause; a number reused after a business closed is the other. Fix Give each brand its own number, or update the old listings to the current name; if a listed business is not yours, ask the directory to correct the number.
effort: content Share of scans 0.8% — 73 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=PHONE_SHARED_BY_OTHER_BUSINESSES)

#### NAP_NAME_DRIFT A directory prints a different business name than the site declares low 0.6% rev 1

Why it matters Two names on one phone number read as two entities, or as one entity with an unsettled name; an engine reconciling them may print either. Fix Correct the business name on the directory listing (claim the listing if you have to) so it matches the name the site declares. Share of scans 0.6% — 58 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=NAP_NAME_DRIFT) · [glossary: nap](/glossary/nap) · [glossary: nap-consistency](/glossary/nap-consistency) · [glossary: name-drift](/glossary/name-drift)

#### NAP_UNDECLARED_LISTING Off-site listings carry this phone number and the site does not declare them info 3.6% rev 1

Why it matters A listing the site does not declare is one an engine corroborates without you: whatever it prints becomes your name, hours and category in the answer. Fix Add the listings you own to sameAs in the business's structured data, and claim or correct the ones you do not. Share of scans 3.6% — 331 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=NAP_UNDECLARED_LISTING)

## Robots

Code Title Level Scans Revision

#### ROBOTS_RULES_SHADOWED robots.txt rules do not apply to named agents medium 14% rev 2 · 2026-09-30

Why it matters A crawler reads only the User-agent group that matches it best and ignores every other group, including `*`. Naming an agent and giving it only `Allow: /` therefore deletes all of your `*` Disallow rules for that agent. The file still parses and the agent still reaches your homepage, so this is invisible on inspection — but the paths you meant to keep out of search are open, and search engines will crawl and may index them. Fix Every User-agent group that names a crawler must repeat the Disallow rules you want applied. A named group with no rules means 'allow everything' for that crawler.
generate the fix: `/api/fix/robots?domain={domain}`
effort: config Share of scans 14% — 1303 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ROBOTS_RULES_SHADOWED) · [glossary: robots-txt](/glossary/robots-txt) · [glossary: robots-group-merging](/glossary/robots-group-merging)

#### INSTRUCTION_CONFLICT The instruction files contradict each other low 0.6% rev 1

Why it matters A crawler obeys only its own User-agent group. A Content-Signal or Disallow written under * says nothing to an agent that has its own group, and a group that both invites and forbids leaves the crawler to pick. Put the signal in every group it is meant for, and let the rules under it agree. Fix Decide the rule once and state the same thing everywhere it is stated: robots.txt, llms.txt, the meta robots tag and the response headers. Share of scans 0.6% — 59 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=INSTRUCTION_CONFLICT) · [glossary: x-robots-tag](/glossary/x-robots-tag)

#### ROBOTS_IS_HTML robots.txt is HTML, not text critical 0.1% rev 1

Why it matters robots.txt must be plain text. Served as HTML it cannot be parsed, so no crawl directives and no sitemap reference are read. Fix robots.txt is returning an HTML page. Serve a real text file at /robots.txt, and check that no catch-all route is answering instead.
effort: config Share of scans 0.1% — 9 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ROBOTS_IS_HTML) · [glossary: robots-parse-error](/glossary/robots-parse-error)

#### ROBOTS_IS_CATCHALL robots.txt answers with the catch-all page - there is no robots.txt low 0.6% rev 1

Why it matters The site answers robots.txt with the same page it serves for a path that does not exist. Crawlers cannot parse HTML as robots rules, so they proceed as if no file existed: everything allowed. Publish a real text/plain robots.txt. Fix A catch-all route returns the homepage for any path, including robots.txt. Add a real robots.txt and return 404 for paths that do not exist.
effort: config Share of scans 0.6% — 53 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ROBOTS_IS_CATCHALL)

#### ROBOTS_LEGACY_AGENT_ONLY robots.txt rules name a retired crawler, not the one that visits medium 0.2% rev 1

Why it matters The file writes rules for a crawler name its operator no longer lists (anthropic-ai or Claude-Web for Anthropic, FacebookBot for Meta). The crawler that actually visits uses its current name and obeys only a group naming it, so it follows the `*` group instead and the rule you wrote never reaches it. Reported only when that changes what the crawler may fetch. Fix Add the current name to the same group: `User-agent: ClaudeBot` under the anthropic-ai / Claude-Web line, `User-agent: meta-externalagent` under FacebookBot. Keep the old line or remove it; it no longer matters.
effort: config Share of scans 0.2% — 21 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ROBOTS_LEGACY_AGENT_ONLY)

#### ROBOTS_SITEMAP_RELATIVE robots.txt declares a sitemap with a relative path low 0.1% rev 1

Why it matters A Sitemap line in robots.txt gives a path (/sitemap.xml) instead of a full URL. The standard requires a full URL, and crawlers drop a relative one, so the sitemap is not declared to them through robots.txt. Fix Write the Sitemap line as a full URL: Sitemap: https://yourdomain.com/sitemap.xml.
effort: config Share of scans 0.1% — 13 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ROBOTS_SITEMAP_RELATIVE)

#### ROBOTS_NOINDEX_LINE robots.txt uses a Noindex line no crawler obeys low 0.2% rev 1

Why it matters robots.txt carries a Noindex: (or Nofollow:) line. Those were never robots.txt directives; Google stopped reading its unofficial Noindex on 1 September 2019 and no other crawler supports it, so the pages it names stay in the index. Fix Remove the Noindex line and put `<meta name="robots" content="noindex">` on those pages (or send X-Robots-Tag: noindex). Do not also Disallow them, or crawlers never see the tag.
effort: config Share of scans 0.2% — 14 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ROBOTS_NOINDEX_LINE)

#### ROBOTS_GROUP_NAME_TYPO robots.txt has a group for a misspelt crawler name medium 1.9% rev 1

Why it matters A group is written for a name no crawler sends, one spelling away from a real one (Perplexity-Bot for PerplexityBot, GPT-Bot for GPTBot). A crawler obeys only a group naming its own token, so the real crawler follows the `*` group instead and the rules you wrote for it never reach it. Reported only when that changes what the crawler may fetch. Fix Correct the User-agent line to the crawler's exact token as its operator documents it (PerplexityBot, GPTBot, ClaudeBot). Matching ignores case but not hyphens or spaces.
effort: config Share of scans 1.9% — 180 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ROBOTS_GROUP_NAME_TYPO)

#### ROBOTS_BLOCKED robots.txt is unreadable high 0.4% rev 1

Why it matters A 5xx on robots.txt tells a standards-following crawler to treat the whole site as disallowed (RFC 9309). A 401 or 403 is a 4xx, which the standard treats as no file and so allows everything, though some crawlers read it as a refusal. Either way the rules you wrote are not the rules a crawler read. This is the single most expensive file on the site to get wrong. Fix robots.txt could not be fetched. Allow it through the security layer; it is the one file every crawler reads first.
effort: config Share of scans 0.4% — 37 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ROBOTS_BLOCKED) · [glossary: robots-unavailable](/glossary/robots-unavailable)

#### ROBOTS_DISALLOW_ALL robots.txt blocks every crawler it does not name critical 1.5% rev 2 · 2026-09-30

Why it matters robots.txt instructs every crawler to stay off the entire site. Fix robots.txt disallows everything. Remove the blanket Disallow: / rule (or restrict it to the paths you actually mean).
effort: config Share of scans 1.5% — 137 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ROBOTS_DISALLOW_ALL)

#### DECLARED_SITEMAP_REFUSED The sitemap robots.txt declares refused this client low 0.8% rev 1

Fix The sitemap named in robots.txt refuses crawlers. Allow it through the security layer.
effort: config Share of scans 0.8% — 77 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=DECLARED_SITEMAP_REFUSED)

#### DECLARED_SITEMAP_BROKEN robots.txt points at a sitemap that fails high 0.8% rev 1

Why it matters robots.txt advertises a sitemap URL that does not resolve. Crawlers that trust the declaration and fail get nothing. Fix robots.txt names a sitemap that does not resolve. Fix the URL in robots.txt or publish the sitemap at the path it names.
effort: build Share of scans 0.8% — 75 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=DECLARED_SITEMAP_BROKEN)

#### ROBOTS_NOT_200 robots.txt does not return 200 medium 4.7% rev 1

Why it matters The response code on robots.txt determines how crawlers treat the whole site. Anything other than 200 or a clean 404 is risky. Fix Serve robots.txt with status 200 as text/plain. A 404 or 410 means no file: under RFC 9309 a crawler may then fetch everything, so any rule you meant to publish is not in force. A 429 is read by Google as a server error, which pauses crawling: exempt robots.txt from rate limits.
effort: config Share of scans 4.7% — 440 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ROBOTS_NOT_200)

#### ROBOTS_NO_SITEMAP robots.txt declares no sitemap medium 9.2% rev 1

Why it matters Adding one Sitemap: line lets crawlers enumerate the site instead of guessing at it. Fix Add a Sitemap: line to robots.txt pointing at the full sitemap URL, so crawlers find it without guessing.
generate the fix: `/api/fix/robots?domain={domain}`
effort: config Share of scans 9.2% — 857 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ROBOTS_NO_SITEMAP)

#### AI_OPTOUT_SET An AI opt-out signal is set medium 1.8% rev 1

Why it matters A Content-Signal directive is telling AI systems not to use this site. CDNs inject this by default on some plans, so it is frequently set without the owner deciding to set it. Fix This is a choice, not a defect. If you want answer engines to quote the site, remove the AI opt-out directives; if you do not, leave it.
effort: config Share of scans 1.8% — 165 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=AI_OPTOUT_SET) · [glossary: ai-opt-out](/glossary/ai-opt-out) · [glossary: machine-readable-opt-out](/glossary/machine-readable-opt-out)

#### ROBOTS_CTYPE robots.txt has an unexpected content type low 0.2% rev 1

Why it matters robots.txt should be served as text/plain. Some parsers are strict about this. Fix Serve robots.txt as text/plain. Some validators refuse any other content type.
effort: config Share of scans 0.2% — 15 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ROBOTS_CTYPE)

#### RENDER_RESOURCE_BLOCKED The homepage depends on files this site's own robots.txt disallows medium 3.1% rev 2 · 2026-09-30

Why it matters A crawler that obeys robots.txt cannot load those files, so it renders a different page from the one visitors see. Fix Allow crawlers to fetch the CSS and JavaScript the page depends on. Remove the Disallow rules covering those paths, or serve the content without them.
effort: config Share of scans 3.1% — 290 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=RENDER_RESOURCE_BLOCKED) · [glossary: blocked-render-resource](/glossary/blocked-render-resource)

## Delivery parity

Code Title Level Scans Revision

#### CHALLENGE_SERVED_200 A verification page is being served instead of the file critical 0.2% rev 1

Why it matters The server answered with HTTP 200 — so uptime monitors, Search Console and your host all report this as healthy — but the body is a JavaScript verification page, not the file. Crawlers do not run JavaScript on these requests. They receive the interstitial and move on. Fix A verification page must not answer with status 200. Return 403 or 503 for challenges so crawlers do not index the challenge as the page.
effort: config Share of scans 0.2% — 23 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=CHALLENGE_SERVED_200) · [glossary: challenge-page-at-200](/glossary/challenge-page-at-200)

#### ANSWER_ENGINE_REFUSED An answer engine's crawler was refused while a browser was served high 5.8% rev 2 · 2026-11-07

Why it matters The same URL, the same minute: an ordinary client was served the page and a named answer-engine crawler was not. That is the edge rather than a robots.txt decision. One caveat this site's own guides state and this finding must too: the scan sent the crawler's name from an address that operator does not publish, and an edge that verifies identity by IP is right to refuse it. So this proves the edge treats the identity differently - not that it refuses the real crawler. Your own logs, checked against the operator's published ranges, settle which. If the edge is refusing the real crawler, nothing else in this report reaches that engine, however good it is. Fix Allow the named answer-engine crawlers through the security layer (Cloudflare bot rules, WAF, rate limits), or verify their published IP ranges instead of blocking by name.
effort: config Share of scans 5.8% — 535 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ANSWER_ENGINE_REFUSED) · [glossary: delivery-comparison](/glossary/delivery-comparison)

#### UNIFORM_REFUSAL This origin refused CrawlCheck, so the site was never measured critical 16.1% rev 1

Fix The site refused every request from CrawlCheck. Allow CrawlCheck's address, or run the browser extension from your own network, then re-scan.
effort: config Share of scans 16.1% — 1501 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=UNIFORM_REFUSAL) · [glossary: uniform-refusal](/glossary/uniform-refusal)

#### CRAWLER_SERVED_LESS A named crawler was served materially less text than an ordinary client high 1.1% rev 3 · 2026-11-07

Why it matters Same URL, same minute, both answered 200 — but the named crawler received materially fewer words than an unnamed client did. A refusal is loud and easy to find; being served a thinner page is silent, and it is the version an answer engine quotes from. Fix Serve every crawler the same HTML a browser gets. Check bot-management rules, 'lite' templates and user-agent conditions in the CMS or CDN.
effort: config Share of scans 1.1% — 100 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=CRAWLER_SERVED_LESS)

#### CRAWLER_REDIRECTED_AWAY A named crawler was sent somewhere the ordinary client was not high 0.2% rev 3 · 2026-11-07

Why it matters The ordinary client stayed on the URL and the named crawler was sent elsewhere. Whatever is at the destination is what that engine will read and quote instead of this page. Fix Remove the user-agent based redirect so named crawlers land on the page a visitor lands on.
effort: config Share of scans 0.2% — 18 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=CRAWLER_REDIRECTED_AWAY)

#### CRAWLER_SERVED_EXTRA Crawlers are given scripts, links or instructions that visitors never see high 1.4% rev 2 · 2026-11-07

Why it matters When an AI or search crawler asks for this page it receives something an ordinary visitor does not: a script from another host, links to other sites, or text written to an AI. Nobody reviewing the page in a browser sees it, and the AI treats it as part of the site. That is how a site, its CMS plugin or an attacker who got into it plants instructions or links for AI answers; it can also be a bot-only tag the owner added on purpose. Either way the AI is reading a page the owner's visitors are not. Fix Find what adds content by user agent (a plugin, an edge rule, a prerender service, or injected code) and serve crawlers the page visitors get; if the extra is yours on purpose, put it in the visitor page too.
effort: config Share of scans 1.4% — 132 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=CRAWLER_SERVED_EXTRA)

#### CHALLENGE_PINNED_AT_EDGE The verification page is cached at the CDN edge critical 0.2% rev 1

Why it matters The origin marked this response private and uncacheable. A cache rule with a TTL override stored it anyway, so the verification page is now being handed to every visitor and every crawler from the edge until it expires or is purged. Fix Purge the edge cache: a challenge page was cached in place of the real file. Then exclude machine files from HTML caching rules.
effort: config Share of scans 0.2% — 15 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=CHALLENGE_PINNED_AT_EDGE)

#### HOMEPAGE_REFUSED The homepage refused this client, so every homepage section is unmeasured low 1.2% rev 1

Fix The homepage was refused. Check bot-protection and WAF rules; a challenge page served with a 200 status reads as the site to a crawler.
effort: config Share of scans 1.2% — 107 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=HOMEPAGE_REFUSED) · [glossary: homepage-refused](/glossary/homepage-refused)

#### HOMEPAGE_IS_INTERSTITIAL The homepage answered with an interstitial, not a page critical 0.5% rev 1

Fix Remove or lower the interstitial (age gate, consent wall, JS challenge) for crawlers, or make sure the real content is in the first response.
effort: config Share of scans 0.5% — 45 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=HOMEPAGE_IS_INTERSTITIAL) · [glossary: 202-interstitial](/glossary/202-interstitial)

## JSON-LD and entities

Code Title Level Scans Revision

#### ENTITY_COLLISION Two different entities share the same identifying data medium 2.6% rev 1

Why it matters Two entities on the page resolve to the same identifying data (e.g. name, phone, or place ID), so they cannot both be correct at once. Fix Merge the duplicate business nodes into one, keep a single @id, and remove any plugin that emits a second empty LocalBusiness record.
generate the fix: `/api/fix/entitymap?domain={domain}`
effort: build Share of scans 2.6% — 246 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ENTITY_COLLISION) · [glossary: entity-collision](/glossary/entity-collision)

#### ENTITY_NO_COORDINATES An entity is missing geographic coordinates info 6.2% rev 1

Why it matters An entity is missing latitude/longitude coordinates, so it cannot be verified against a real-world location. Fix Add latitude and longitude to the LocalBusiness node's geo property so an engine can place the business on a map without guessing.
generate the fix: `/api/fix/entitymap?domain={domain}`
effort: content Share of scans 6.2% — 573 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=ENTITY_NO_COORDINATES) · [glossary: geocoordinates](/glossary/geocoordinates) · [glossary: geocoding](/glossary/geocoding) · [glossary: postaladdress](/glossary/postaladdress)

#### GEO_COUNTRY_MISMATCH Published coordinates are not in the country the address names low 0.5% rev 1

Why it matters The latitude/longitude on the page falls outside the country its own address names. A range check cannot catch this: a missing minus sign is still a legal coordinate. Coordinates are read only by machines, so the error produces no visible symptom. Fix Correct the latitude and longitude in the page's structured data so the point falls at the address. The usual causes are a missing minus sign (western longitudes and southern latitudes are negative) or the two values swapped; the evidence names which. Share of scans 0.5% — 49 of 9304 counted scans Links [JSON](https://crawlcheck.io/api/rules?code=GEO_COUNTRY_MISMATCH)

## Answer-engine citations

Code Title Level Scans Revision

#### CITED_WITH_WRONG_FACTS An answer engine cites the site with the wrong phone or address medium — rev 1

Why it matters The engine read the business facts from somewhere other than the site, usually a directory, and is repeating them. Fix Find the source that prints the wrong phone or address (an old directory listing, a stale page) and correct it, and make the site's own structured data state the right one. Share of scans measured in answer runs, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=CITED_WITH_WRONG_FACTS)

#### REACHABLE_NOT_CITED An answer engine can reach the site and does not cite it info — rev 1

Why it matters Access is not the problem: the content and authority are. Fix Nothing blocks this engine. Put the answer to the question in the page's visible text and keep the business facts in structured data, then ask again after the engine has re-read the site. Share of scans measured in answer runs, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=REACHABLE_NOT_CITED)

#### BOT_BLOCKED_NOT_CITED An answer engine does not cite the site and robots.txt blocks its crawler medium — rev 1

Why it matters An engine cannot quote a page its crawler is told not to read. Fix If you want this engine to cite the site, allow its crawler in robots.txt and at the firewall. Share of scans measured in answer runs, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=BOT_BLOCKED_NOT_CITED)

#### BOT_RULES_SHADOWED_NOT_CITED An answer engine does not cite the site and its crawler group is shadowed low — rev 1

Why it matters The crawler's own group in robots.txt does not carry the rules the site meant it to follow. Fix Repeat the rules you want inside that crawler's own User-agent group, or remove the empty named group, so the crawler reads the rules you intended. Share of scans measured in answer runs, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=BOT_RULES_SHADOWED_NOT_CITED)

#### NO_LLMS_TXT_NOT_CITED An answer engine does not cite the site and there is no llms.txt info — rev 1

Why it matters The engine can reach the site; the site gives it no summary to start from. Fix Publish /llms.txt: a short Markdown file naming the business, what it does, and the pages worth reading, served as text/plain or text/markdown. Share of scans measured in answer runs, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=NO_LLMS_TXT_NOT_CITED)

## Capabilities

Code Title Level Scans Revision

#### CAPABILITY_DECLARED_PUBLIC_OBSERVED_AUTH Declared capability declared public observed auth not scored — rev 1

Fix the finding in your report names what to change Share of scans listed in the record's capabilities section, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=CAPABILITY_DECLARED_PUBLIC_OBSERVED_AUTH)

#### CAPABILITY_DECLARED_ENDPOINT_MISSING Declared capability declared endpoint missing not scored — rev 1

Fix the finding in your report names what to change Share of scans listed in the record's capabilities section, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=CAPABILITY_DECLARED_ENDPOINT_MISSING)

#### CAPABILITY_DECLARED_NO_REQUIRED_INPUT_OBSERVED_REJECTED Declared capability declared no required input observed rejected not scored — rev 1

Fix the finding in your report names what to change Share of scans listed in the record's capabilities section, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=CAPABILITY_DECLARED_NO_REQUIRED_INPUT_OBSERVED_REJECTED)

#### CAPABILITY_DECLARED_MEDIA_MISMATCH Declared capability declared media mismatch not scored — rev 1

Fix the finding in your report names what to change Share of scans listed in the record's capabilities section, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=CAPABILITY_DECLARED_MEDIA_MISMATCH)

#### CAPABILITY_DECLARED_TOOL_NOT_LISTED Declared capability declared tool not listed not scored — rev 1

Fix the finding in your report names what to change Share of scans listed in the record's capabilities section, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=CAPABILITY_DECLARED_TOOL_NOT_LISTED)

#### CAPABILITY_LISTED_TOOL_NOT_DECLARED Declared capability listed tool not declared not scored — rev 1

Fix the finding in your report names what to change Share of scans listed in the record's capabilities section, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=CAPABILITY_LISTED_TOOL_NOT_DECLARED)

## Self-audit: defects in our own record

Code Title Level Scans Revision

#### FETCHER_DISAGREEMENT Two of our fetch layers disagree about a machine file medium — rev 1

Why it matters A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site. Fix the finding in your report names what to change Share of scans about our own record, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=FETCHER_DISAGREEMENT)

#### BENCH_POOL_MISMATCH The benchmark's pool size contradicts its scope medium — rev 1

Why it matters A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site. Fix the finding in your report names what to change Share of scans about our own record, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=BENCH_POOL_MISMATCH)

#### COHORT_BELOW_THRESHOLD Ranked against a cohort under the minimum low — rev 1

Why it matters A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site. Fix the finding in your report names what to change Share of scans about our own record, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=COHORT_BELOW_THRESHOLD)

#### SECTION_SCORE_OUT_OF_RANGE A section score outside 0 to 100 high — rev 1

Why it matters A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site. Fix the finding in your report names what to change Share of scans about our own record, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=SECTION_SCORE_OUT_OF_RANGE)

#### RANKED_VALUE_NOT_REPRODUCIBLE The ranked number does not follow from the printed table high — rev 1

Why it matters A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site. Fix the finding in your report names what to change Share of scans about our own record, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=RANKED_VALUE_NOT_REPRODUCIBLE)

#### SECTION_WEIGHT_UNDECLARED A section scored without a declared weight medium — rev 1

Why it matters A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site. Fix the finding in your report names what to change Share of scans about our own record, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=SECTION_WEIGHT_UNDECLARED)

#### PATH_NOT_MEASURED A deep path was asked for, the home page was measured low — rev 1

Why it matters A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site. Fix the finding in your report names what to change Share of scans about our own record, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=PATH_NOT_MEASURED)

#### PLACEHOLDER_IN_RECORD One of our placeholders reached the record high — rev 1

Why it matters A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site. Fix the finding in your report names what to change Share of scans about our own record, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=PLACEHOLDER_IN_RECORD)

#### GRADE_OFF_ITS_OWN_SCALE The letter does not follow the record's own thresholds high — rev 1

Why it matters A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site. Fix the finding in your report names what to change Share of scans about our own record, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=GRADE_OFF_ITS_OWN_SCALE)

#### GRADED_THROUGH_A_WALL A grade on a site that refused CrawlCheck critical — rev 1

Why it matters A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site. Fix The reading was taken through a block page. Allow CrawlCheck and re-scan for a real grade.
effort: config Share of scans about our own record, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=GRADED_THROUGH_A_WALL)

#### EXTRACTORS_READ_NOTHING Our extractors read nothing from a head that has tags medium — rev 1

Why it matters A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site. Fix the finding in your report names what to change Share of scans about our own record, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=EXTRACTORS_READ_NOTHING)

#### GRADE_MISSING A graded record with no grade medium — rev 1

Why it matters A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site. Fix the finding in your report names what to change Share of scans about our own record, not counted per scan Links [JSON](https://crawlcheck.io/api/rules?code=GRADE_MISSING)

## MCP servers

Code Title Level Scans Revision

#### MCP_TOOL_POISONING An MCP server's tool descriptions carry text aimed at the model high — rev 1

Why it matters Text a client hands the model from tools/list hides characters a person cannot see, tells the model to ignore its instructions or keep something from the user, asks for secrets or the conversation, or tells the model how to use other tools. A model reads it as an instruction; the person approving the server usually never sees it. Fix Rewrite the tool descriptions so they describe the tool and nothing else; remove any text that addresses the model or tells it what to do. Share of scans measured on remote MCP servers in the official MCP Registry, not per site scan: see https://crawlcheck.io/api/mcp/index (tool_poisoning) Links [JSON](https://crawlcheck.io/api/rules?code=MCP_TOOL_POISONING)

## High-risk field watch

Code Title Level Scans Revision

#### HIGH_RISK_FIELD_CHANGED A high-risk field changed and the owner has not confirmed it high — rev 1

Why it matters One of the fields an attacker changes first moved since CrawlCheck last read it: the payment processors loaded on the homepage or checkout page, the host checkout links go to, a cryptocurrency wallet address printed on the page, the OAuth authorization server the site's MCP endpoint names, the domain's MX hosts, or its registry-listed MCP endpoints. Watched fields: payment_processors, checkout_host, wallet_addresses, oauth_authorization_server, mx_hosts, mcp_endpoints. The first reading is the baseline and never alerts. Fix If you made the change, confirm it from the owner's session. If you did not, treat it as a compromise: restore the previous value and find out who changed the site. Share of scans measured by the daily high-risk field watch, not per site scan: see https://crawlcheck.io/api/v1/high-risk Links [JSON](https://crawlcheck.io/api/rules?code=HIGH_RISK_FIELD_CHANGED)

Generated 2026-11-22 01:33 UTC. A rule you think is wrong? Write to [hello@crawlcheck.io](mailto:hello@crawlcheck.io) with the report id; corrections are published as a new revision.

We read your site the way crawlers do, from our own network, and every report says exactly what was asked for and what came back. [Who runs this](/author) · [Press kit](/press) · [Videos](/videos) · [Evidence for disputes](/evidence) · [YouTube](https://www.youtube.com/@crawlcheck) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io)

#### How we measure, and our own machine files

CrawlCheck reads every site as itself. Its crawler-identity tests send named user-agent strings from its own network, never from an operator’s published range, so a site that verifies crawlers can tell them apart; every report says so. Results reflect what an anonymous non-browser client receives. [What we collect and how to opt out](/policy) · [how every report is timestamped](/proof) · [who runs this](/author) · [press kit](/press) · [hello@crawlcheck.io](mailto:hello@crawlcheck.io).

**Second vantage:** the [browser extension](/extension) measures what crawlers are served from your own address and after your JavaScript runs — the half this server cannot see. From 1.0.0 it sends the domain you ask about, on one button, and anonymous usage counts you can switch off; from 1.0.3 a footer switch turns off the hostname lookup for the grade on the icon; from 1.0.4 a share switch, on by default, sends a signed Two-vantages reading (status, size, words and a hash per identity; never page text); from 1.0.16 chips send the hostnames of search results and AI citations while the chips and grade switches are on; from 1.0.17 a readings switch, on by default, sends one signed page reading a day (numbers, the hostname and the page address with the query removed; never in incognito; at most 40 a day) and only while the share and grade switches are also on. Earlier versions send nothing.

**Our own machine layer for the agentic web:** [llms.txt](/llms.txt) · [llms-full.txt](/llms-full.txt) · [agents.md](/agents.md) · [SKILL.md](/SKILL.md) · [entitymap.json](/entitymap.json) · [media kit](/.well-known/media-kit.json) · [MCP server card](/.well-known/mcp/server-card.json) · [API catalog](/.well-known/api-catalog) · [OpenAPI](/openapi.json) · [robots.txt](/robots.txt) · [sitemap.xml](/sitemap.xml) · [security.txt](/.well-known/security.txt) — every one named in robots.txt, which is exactly what we check for on yours.

**Registry:** [MCP](/registry/mcp) · [AI policy](/registry/aipolicy) | **Pages:** [IP ranges](/data/ip-ranges) · [Keys](/keys) · [Conduct: Googlebot](/agents/conduct/googlebot) | **Datasets:** [Technologies: Google fonts](/datasets/technologies/google-fonts) | **Guides:** [How to check an MCP server before an agent connects](/guides/how-to-check-an-mcp-server-before-an-agent-connects) | **Glossary:** [entity map](/glossary/entity-map) · [Parent-child retrieval](/glossary/parent-child-retrieval) · [GET request](/glossary/get-request) · [Inbound link](/glossary/inbound-link) · [user-triggered fetch exemption](/glossary/user-triggered-fetch-exemption)

[Contact](/contact) · [Partnerships](/partnerships) · [Terms of service](/terms) · [Refund policy](/refunds) · [Trust center](/trust) · [Subprocessors](/subprocessors) · [Changelog](/changelog) · [Report an error](/report-error)
