RulebookThe rules behind every answer an agent receives.
Every finding in a Resolve answer names its rule code and the revision that decided it, and every decision receipt names the engine and a digest of its rule tables. This page is what each of those means, so an agent, or the person behind it, can read the reason as well as the verdict.
111
rules a finding can cite
82
graded on every site scan
10
revised, old revisions kept
9,304
counted scans behind each share
1 · A finding
Code and revision
Each item in an answer’s findings carries code and rule_revision. /rules/<CODE> opens the rule here; the revision says which wording decided it, and a replay uses that one, never today’s.
The rules/api/rules
2 · A decision
Engine and rule digest
A decision receipt names engine.version and engine.rules_sha256. The tables that digest covers, the action templates, the ranking and how long a receipt holds, are published so anyone can recompute it.
/rules/preflight.jsonThe receipt format
3 · A grade
Level and effect
Every graded finding lowers its section’s score; an open finding at medium or above also caps the letter. A self-audit rule is a defect in our own record, never in the scanned site.
Self-audit rulesSchema
/api/rules (JSON) · schema · decision rule tables · compatibility policy · trust
Each finding has a level from info to critical. Every graded finding lowers the score of its section; an open finding at medium or above also caps the letter grade, so fix those first. A rule that changes what it reports gets a new revision; a record keeps the revision that decided it, and a replay uses that revision, never today's.
Measured
How often each decision rule was right
1 outcome scored in 90 days — too few to estimate a rate; the rows are the raw reports, and one report counts under every rule that fired in its receipt.
Agents that act on a decision receipt can report what happened. Each report is scored against the rules that fired in that receipt: for a warn, confirmation or block, hit is the share of acted-on decisions where acting went wrong; for allow, held is the share where it went fine. A block that stopped the agent leaves no outcome, so cautionary rates cover the agents that acted anyway. No blocklist publishes this.
| Rule | Effect | Outcomes | Not ok | Rate |
|---|
| decision | require_confirmation | 1 | 1 | 100% hit |
| resolve.actions.transact | require_confirmation | 1 | 1 | 100% hit |
/api/v1/decisions/accuracy (JSON, 90 days) · every new receipt carries the same figures next to each rule that fired (steps[].measured) · report outcomes at /api/v1/outcome with the receipt’s decision_sha256.
Every rule
111 rules in 17 families
Open a rule for what it means, how to fix it and how many scans carried it.
Site, hosts and page 18 Headers 4 Mostly code 3 Machine files 12 Other 9 Agentic resource discovery (ARD) 4 WebMCP form tools 2 Trust path 2 NAP and claims 3 Robots 17 Delivery parity 9 JSON-LD and entities 3 Answer-engine citations 5 Capabilities 6 Self-audit: defects in our own record 12 MCP servers 1 High-risk field watch 1
Site, hosts and page
CodeTitleLevelScansRevision
CSP_REPORTING_BROKENCSP violation reporting is misconfiguredinfo0.3%rev 1
- Why it matters
- The Content-Security-Policy header declares where to send violation reports, and the declaration is wrong. `report-to` takes the NAME of a group defined in a separate header — given a URL it is silently ignored, so nothing is reported at all. And `report-uri` aimed at your own origin turns every violation into another request to the page you are already trying to keep fast. Neither breaks the page; both mean the reporting you think you have is either absent or self-inflicted.
- Fix
- The Content-Security-Policy report endpoint does not accept reports. Fix the report-uri / report-to target or remove the directive.
effort: config - Share of scans
- 0.3% — 24 of 9304 counted scans
- Links
- JSON · glossary: content-security-policy
BROKEN_INTERNAL_LINKLinks on this site point at pages that do not answermedium2.7%rev 1
- Why it matters
- A link on this site was followed and the target answered 404 or 410. A crawler spends budget on every one of these and learns nothing; a reader following one gets an error page. The usual causes are a route renamed without updating the links to it, and a nav or footer that renders links to authenticated-only routes for everyone — those answer 404 when signed out, which is what a crawler always is. Only 404 and 410 count here. A link answering 401, 403 or 429 means something declined to answer CrawlCheck — bot protection, a login wall, a rate limit — which is a fact about the request, not about the link, so those are counted and never reported as broken. Links are sampled rather than exhaustively crawled, so this is a floor and not a full link audit.
- Fix
- Fix or remove the internal links that return 404 so crawlers and visitors are not sent to dead pages.
effort: content - Share of scans
- 2.7% — 255 of 9304 counted scans
- Links
- JSON · glossary: broken-link · glossary: internal-link · glossary: dead-anchor
BROKEN_EMBEDAn embedded frame on the homepage cannot loadlow0.2%rev 1
- Why it matters
- A frame, map or player on the page was fetched the way a browser loads it, and its source answered an error or forbade being shown inside this site (X-Frame-Options or Content-Security-Policy frame-ancestors). Visitors see an empty box or a browser error where it should be, and a crawler that renders the page sees the same gap. The usual causes are a deleted video or form, and a map or page address pasted as a frame instead of the provider's embed address.
- Fix
- Replace each listed source with the provider's embed address (Google Maps: Share, then Embed a map; YouTube: youtube.com/embed/<id>) or remove the frame. A source that forbids framing cannot be fixed from your page: link to it instead.
effort: content - Share of scans
- 0.2% — 22 of 9304 counted scans
- Links
- JSON
AD_DENSITY_HIGHAds take too much of the page on a phonelow0%rev 1
- Why it matters
- The homepage was rendered at phone width and its ads measured: more than 30% of the page height, more than 30% of the first screen, or a sticky ad over 30% of the screen. Those are the limits of the Coalition for Better Ads mobile standard, which Chrome's ad filter enforces, and a page over them reads as made for ads. Visitors scroll past ads to reach the content, and search engines that judge page experience see the same layout.
- Fix
- Cut the ad slots on the mobile layout until ads stay under 30% of the page height and of the first screen, and keep any sticky ad under 30% of the screen.
effort: build - Share of scans
- 0% — 3 of 9304 counted scans
- Links
- JSON
MOBILE_INTERSTITIALA pop-up covers the page on a phone when it loadsmedium0.3%rev 1
- Why it matters
- The homepage was rendered at phone width and, on load, a fixed overlay covered at least half of the first screen. Google treats a pop-up that covers the main content as soon as a visitor arrives from search as an intrusive interstitial, and visitors who wanted the answer leave. Cookie consent and age-verification notices are exempt and are never counted here.
- Fix
- Show the pop-up later (after a scroll or on exit), make it a banner that leaves most of the screen free, or remove it on mobile. Consent and age notices may stay.
effort: build - Share of scans
- 0.3% — 31 of 9304 counted scans
- Links
- JSON
SECRETS_EXPOSEDPrivate files or live keys are readable by anyonehigh0.3%rev 1
- Why it matters
- A file the site should never serve (a .git or .svn folder, a .env file, a config backup) answered with its own contents, or a live API key or private key is written into the homepage or the site's own scripts. Anyone who requests the same address gets the same bytes: database passwords, keys and source history are the first things an attacker tries. Each path was requested once and counts only when it answers with the file itself, never with a page.
- Fix
- Deny the listed paths at the server (or delete the files), then treat every password and key they contained as public: change the passwords and revoke and reissue each key shown. Remove keys from page code and scripts; a key a browser needs must be a publishable one.
effort: config - Share of scans
- 0.3% — 30 of 9304 counted scans
- Links
- JSON
IMAGES_BLOCKED_GOOGLEBOT_IMAGErobots.txt keeps Google's image crawler off the site's imageslow0.8%rev 1
- Why it matters
- Images on the homepage are under a path the robots.txt group for Googlebot-Image disallows (that crawler obeys its own group, else Googlebot's, else the * group). Those images cannot appear in Google Images or as image results, and a page whose pictures are blocked loses them in visual search. The usual cause is a Disallow on an uploads or assets folder written to save crawl budget.
- Fix
- Remove the Disallow that covers the image folder, or add an Allow for it in the group Googlebot-Image follows.
effort: config - Share of scans
- 0.8% — 76 of 9304 counted scans
- Links
- JSON
IMAGE_FORMAT_MISMATCHImages are not the format their file names sayinfo1.8%rev 1
- Why it matters
- An image's first bytes say one format while its file name says another (a .jpg that is a PNG, or an image address that returns an HTML page). It was fetched without asking for WebP or AVIF, so a CDN that converts on request is not counted. Browsers usually cope; image indexers and some clients trust the name, and an image address that returns a page is a broken image.
- Fix
- Save each listed image under the extension of its real format (or re-export it in the format its name says), and fix image addresses that return a page.
effort: content - Share of scans
- 1.8% — 168 of 9304 counted scans
- Links
- JSON
DATE_MOVED_NO_CHANGEThe page's date changed but its words did notinfo1.4%rev 1
- Why it matters
- Between two of CrawlCheck's reads, the page's declared modification date (dateModified or article:modified_time) moved forward while the main content, read with navigation, ads and dates removed, stayed the same. A date that moves without new words claims freshness the page does not have; engines that compare the two learn to ignore the date.
- Fix
- Update the modification date only when the content changes; turn off plugins or templates that stamp the current date on every build.
effort: config - Share of scans
- 1.4% — 132 of 9304 counted scans
- Links
- JSON
CHANGED_DATE_STAYEDThe page's words changed but its date did notinfo0.8%rev 1
- Why it matters
- Between two of CrawlCheck's reads, the page's main content changed (read with navigation, ads and dates removed) while the modification date it states (dateModified or article:modified_time), and the sitemap's date for the page when one was read, stayed the same. An engine that decides what to re-read from those dates is told nothing changed, so it keeps the old copy.
- Fix
- When the content changes, update dateModified (and article:modified_time) and the sitemap lastmod for the page in the same publish; most CMSs do this when the page is saved rather than edited in place by a widget or a cache.
effort: config - Share of scans
- 0.8% — 78 of 9304 counted scans
- Links
- JSON
PAGES_SNIPPET_RESTRICTEDPages meant to be found hide their own search snippetlow0.1%rev 1
- Why it matters
- Pages that allow indexing also tell search engines to show little or nothing of them: nosnippet, a max-snippet under 50 characters, max-image-preview:none, or an unavailable_after date that has passed. The page can rank but appears as a bare title, with no description and no image, and answer engines that respect nosnippet will not quote it. It is usually a theme or SEO-plugin setting applied site-wide by mistake, or a header added at the server.
- Fix
- Remove nosnippet and max-snippet / max-image-preview limits from pages that should show in search (check the SEO plugin's advanced robots settings and any X-Robots-Tag header), or noindex the page if it should not appear at all.
effort: config - Share of scans
- 0.1% — 7 of 9304 counted scans
- Links
- JSON
REDIRECT_CHAINThe homepage is reached through a chain of redirectsinfo22.6%rev 1
- Why it matters
- Reaching the homepage takes more than one redirect, for example http to https and then the domain to www. Each hop is another request and round trip before a crawler or visitor gets the page, temporary hops (302, 307) tell search engines to keep the old address, and Google stops following after ten.
- Fix
- Send every old address straight to the final one in a single permanent (301 or 308) redirect: in the host, CDN or plugin rules, redirect http://domain and http://www straight to the final https address instead of chaining scheme, host and path fixes.
effort: config - Share of scans
- 22.6% — 2102 of 9304 counted scans
- Links
- JSON
HOST_DUPLICATE_200www and the apex both serve the site instead of one redirectingmedium5.5%rev 1
- Why it matters
- Both www and the bare domain answer 200 with the same pages, so every URL on this site exists at two addresses and a crawler fetches the whole site twice. A canonical tag does not prevent this: it is read after the response arrives, so the second fetch has already been spent. Nothing looks broken from a browser, which is why this survives for years. The fix is a 301 from one host to the other so there is one address per page.
- Fix
- Pick one host (apex or www) and redirect the other to it with a 301 so there is one copy of every page.
effort: config - Share of scans
- 5.5% — 515 of 9304 counted scans
- Links
- JSON · glossary: canonical · glossary: canonical-url · glossary: host-duplicate
ORIGIN_REFUSED_SCANNERThe homepage and robots.txt both refused this client, so the site was never measuredcritical0.2%rev 1
- Fix
- Allow CrawlCheck's address through the security layer, or scan from the browser extension, which uses your own address.
effort: config - Share of scans
- 0.2% — 14 of 9304 counted scans
- Links
- JSON · glossary: refusal · glossary: datacenter-ip · glossary: vantage-point
DOMAIN_IS_ALIASThis domain is an alias: every path redirects to another sitelow1.1%rev 1
- Why it matters
- Every request to this hostname is redirected to a different site, so nothing measured here describes a site at this address. Scan the destination instead; the redirect itself is the only fact about this domain.
- Fix
- Nothing to fix on this domain: it forwards to another site. Scan the destination host instead.
- Share of scans
- 1.1% — 102 of 9304 counted scans
- Links
- JSON · glossary: alias-domain
HOMEPAGE_REDIRECTS_OFF_HOSTThe homepage sends crawlers to a different hostnamemedium3.4%rev 1
- Why it matters
- A request for this domain's homepage answers with a redirect to a hostname outside its own www/apex pair - a regional storefront, a checkout host, a vendor platform. This scan stopped at the redirect; the destination was not evaluated as part of this report, so nothing on the other host counts for this domain here. A deliberate forward (a brand domain, a migration) is a choice this scan cannot distinguish from a misrouted one; what it can say is that a crawler reading this hostname alone finds no page.
- Fix
- Point the domain at the site it is meant to serve, or make the redirect a single permanent hop to the canonical host.
effort: config - Share of scans
- 3.4% — 317 of 9304 counted scans
- Links
- JSON · glossary: off-host-redirect
ORIGIN_BLOCKED_BEHIND_CACHEA security layer refused this file — source unconfirmedmedium0.2%rev 1
- Why it matters
- The cached copy is fine; behind it the origin refused this request. IMPORTANT LIMIT: a scanner runs from one IP, and application firewalls block by IP reputation — so this may mean the site refuses crawlers, or it may only mean the site refuses CrawlCheck specifically. The two are indistinguishable from outside. Confirm in the server access logs, grouped by source IP, before acting on it.
- Fix
- The cached copy works but the origin refuses. Fix the origin block before the cache expires, or the next miss serves the block to everyone.
effort: config - Share of scans
- 0.2% — 20 of 9304 counted scans
- Links
- JSON · glossary: edge-cache · glossary: cache-buster
STAGING_LEAKA staging hostname is exposed in the page sourcelow0%rev 1
- Why it matters
- A staging hostname appears in the page source. If that hostname is reachable and indexable it competes with the production site for the same content.
- Fix
- Remove staging or development hostnames, paths and comments from the production HTML, and block the staging site from crawlers.
effort: config - Share of scans
- 0% — 3 of 9304 counted scans
- Links
- JSON · glossary: staging-leak
CodeTitleLevelScansRevision
HSTS_MISSINGHTTPS is served without a Strict-Transport-Security headerinfo15.9%rev 1
- Why it matters
- Strict-Transport-Security tells a browser that has visited once to use HTTPS for every later request without trying HTTP first. Without it, a typed or linked http:// address makes one plaintext request before the redirect. It does not change what crawlers receive or how the site ranks; it is a small, reversible hardening step, and it is listed at the lowest severity for that reason.
- Fix
- Send Strict-Transport-Security: max-age=15552000; includeSubDomains on HTTPS responses (Cloudflare: SSL/TLS, Edge Certificates, HSTS). Leave preload off unless you intend a change that takes months to undo.
generate the fix: /api/fix/hsts?domain={domain}
effort: config - Share of scans
- 15.9% — 1479 of 9304 counted scans
- Links
- JSON · glossary: hsts
HSTS_PRELOAD_INELIGIBLEThe HSTS header asks for browser preloading but does not qualify for itlow5%rev 1
- Why it matters
- The header carries the `preload` token, which is a request to be compiled into the HTTPS-only list shipped inside browsers. That list only accepts a header with max-age of at least 31536000 (one year) AND includeSubDomains AND preload. As sent, the token is ignored: the site is declaring an intent it does not meet, and if it ever was on the list, this header is what gets it removed. The rest of the HSTS header still works.
- Fix
- Either raise max-age to 31536000 and keep includeSubDomains so the preload token qualifies, or remove the preload token. As sent it does nothing.
generate the fix: /api/fix/hsts?domain={domain}
effort: config - Share of scans
- 5% — 465 of 9304 counted scans
- Links
- JSON · glossary: hsts-preload-list
HSTS_PRELOAD_REMOVEDThis domain was removed from the browser HSTS preload listlow0.8%rev 1
- Why it matters
- The browser preload list (hstspreload.org) records this domain as removed. Domains are removed when the header stops meeting the requirements, or on request. Browsers that shipped with the old list still force HTTPS on every subdomain; newer ones do not, so behaviour now differs by browser version. Either the removal was intended and the `preload` token should go, or it was not and the header needs to qualify again before resubmitting.
- Fix
- If preloading was never intended, remove the preload token. If it was, restore a qualifying header (max-age at least 31536000, includeSubDomains, preload) and resubmit at hstspreload.org.
effort: config - Share of scans
- 0.8% — 74 of 9304 counted scans
- Links
- JSON
STALE_CACHE_SERVEDA cached copy of this page older than an hour was servedlow13.5%rev 3 · 2026-09-30
- Why it matters
- The page handed to us carried a cache age older than an hour. The detail says whether the copy still matches what the origin produces now: if it matches, nothing is broken today; if it differs, an edit is already waiting behind the cache. It matters the moment you change something: an edit, a new price, a corrected phone number stays invisible to every visitor and every crawler for about that long, and nothing in your dashboard says so.
- Fix
- Lower the cache lifetime on HTML (an hour or less), or purge the edge and origin caches after every publish so crawlers see the current copy.
generate the fix: /api/fix/cache?domain={domain}
effort: config - Share of scans
- 13.5% — 1256 of 9304 counted scans
- Links
- JSON · glossary: stale-response · glossary: age-header · the fix end to end on a real site
Mostly code
CodeTitleLevelScansRevision
CONTENT_NEEDS_JAVASCRIPTThe page’s content only exists after JavaScript runsmedium2.1%rev 1
- Why it matters
- The HTML the server sends is an application shell: a framework mount point and a script bundle, with almost no readable text. A browser runs the bundle and the page appears. A crawler that does not execute JavaScript — which is most AI crawlers and every training-data fetch — receives the shell and quotes nothing. This is measured on the bytes the server delivered, not on a rendered view; a framework by itself is not the defect, an empty delivery is.
- Fix
- Render the page on the server or at build time (Next.js/Nuxt/SvelteKit SSR or SSG, or a prerender step) so the delivered HTML already contains the text. Check by fetching the page with curl: the words a crawler needs must be in that response.
effort: build - Share of scans
- 2.1% — 196 of 9304 counted scans
- Links
- JSON · glossary: render-dependence · glossary: render-dependence-gap · glossary: render-gap
PAGE_IS_MOSTLY_CODEAlmost nothing a machine receives from this page is readable textmedium28.4%rev 1
- Why it matters
- A crawler pays for every byte it fetches and can only quote the text. Under 5% readable content on the homepage means an answer engine downloads the whole page and comes away with almost nothing it can quote. The content-ratio row above passes at 20%, so without this a page at 2% and a page at 19% were scored identically.
- Fix
- Move inline CSS and JavaScript into external files, drop unused page-builder styling, and make sure the words a visitor reads are in the HTML itself.
generate the fix: /api/fix/code?id={report_id}
effort: build - Share of scans
- 28.4% — 2642 of 9304 counted scans
- Links
- JSON · glossary: payload-text-share · the fix end to end on a real site
PLACEHOLDER_CONTENTUnfinished placeholder text is live on this pagehigh0.4%rev 1
- Why it matters
- Template text that was never replaced is being served to visitors and to crawlers. An answer engine cannot tell draft copy from real copy — it reads and may quote whatever is there. Placeholder in a title or meta description is worse than in body copy, because that is the text a search result shows and an assistant repeats.
- Fix
- Replace the placeholder text (lorem ipsum, 'test content', template copy) with the real page, or unpublish the page until it is written.
effort: content - Share of scans
- 0.4% — 34 of 9304 counted scans
- Links
- JSON · glossary: placeholder-content
Machine files
CodeTitleLevelScansRevision
SITEMAP_NO_LASTMODThe sitemap does not say when anything changedlow7.4%rev 3 · 2026-11-14
- Why it matters
- The sitemap lists URLs but carries no usable lastmod dates: either none at all, or one date copied onto every URL, which says everything changed at once and so says nothing. Either way it tells a crawler what exists and not what changed. A crawler with a budget uses lastmod to decide what to re-fetch; without it, the choice is made for you — usually by re-crawling little and late. This is the change signal every crawler already understands, and unlike a push protocol it needs no key, no account and no participation from the search engine.
- Fix
- Give each sitemap entry its own <lastmod>: the date that page last changed, not the date the sitemap was built. Then a crawler can prioritise the pages that changed.
generate the fix: /api/fix/sitemap?domain={domain}
effort: content - Share of scans
- 7.4% — 684 of 9304 counted scans
- Links
- JSON · glossary: sitemap · glossary: sitemap-lastmod
MACHINE_CHAIN_HEAVYThe machine files cost more than the pageinfo22.7%rev 4 · 2026-11-14
- Why it matters
- A crawler reads robots.txt, the sitemap, llms.txt and the agent files before the first page, and pays for every 404 that answers with a full page. Keep machine files as small as their job allows and serve absent paths with a short 404 body.
- Fix
- Answer absent machine paths (/agents.md, /.well-known/...) with a short plain 404 body instead of a full HTML page: that is the part that is pure waste. Then keep robots.txt, llms.txt and the sitemap index as small as their job allows; a large sitemap index is normal on a large site.
generate the fix: /api/fix/chain?id={report_id} - Share of scans
- 22.7% — 2108 of 9304 counted scans
- Links
- JSON · glossary: machine-layer
MACHINE_FILE_CACHE_SPLITThis file answers differently depending on whether the cache is hithigh0%rev 1
- Fix
- Make the apex and www hosts serve the same machine files, or redirect one host to the other so there is only one copy to keep current.
effort: config - Share of scans
- 0% — 2 of 9304 counted scans
- Links
- JSON · glossary: edge-cache-pinning · glossary: cache-split
MACHINE_FILE_CACHE_STALEThe cache is still serving a file the origin no longer haslow0.4%rev 1
- Fix
- Give robots.txt, sitemap.xml and llms.txt a short cache lifetime (five minutes is plenty) so an edit reaches crawlers the same day.
effort: config - Share of scans
- 0.4% — 41 of 9304 counted scans
- Links
- JSON
SITEMAP_IS_HTMLThe sitemap returns HTML, not XMLcritical0.3%rev 1
- Why it matters
- A crawler that cannot parse the sitemap cannot enumerate the site. It falls back to following links, so deep and newly published pages go undiscovered.
- Fix
- The sitemap URL is returning an HTML page. Serve XML at that path, or update robots.txt to point at the sitemap that actually exists.
effort: config - Share of scans
- 0.3% — 28 of 9304 counted scans
- Links
- JSON · glossary: soft-404-machine-file
SITEMAP_BLOCKEDThe sitemap is refusedhigh1.9%rev 1
- Why it matters
- SAME LIMIT AS ABOVE: if a named application firewall appears in the response, this may be blocking CrawlCheck's IP rather than crawlers. Confirm in server logs grouped by source IP. Otherwise: the server refuses this sitemap rather than returning it or a clean 404. Crawlers cannot enumerate the site and fall back to link-following, so deep and newly published pages go undiscovered.
- Fix
- Allow the sitemap URL through the security layer so crawlers can read it.
effort: config - Share of scans
- 1.9% — 178 of 9304 counted scans
- Links
- JSON
SITEMAP_ORIGIN_ERRORThe origin returned a server error on the sitemap pathlow0.2%rev 1
- Fix
- The sitemap URL returned a server error. Regenerate it (most CMSs have a setting) and check the URL loads in a private window.
effort: config - Share of scans
- 0.2% — 20 of 9304 counted scans
- Links
- JSON · glossary: sitemap-origin-error
SITEMAP_UNREADABLEWhether a sitemap exists could not be determined — every path was refusedlow0.5%rev 1
- Why it matters
- A refusal is not an absence. The paths where a sitemap would live answered with a block, so this scan cannot say whether one exists — and it does not guess. Confirm from a second network, and check server logs grouped by source IP: if the block is aimed at CrawlCheck's address, the finding is about our access; if named crawlers see the same refusal, the sitemap is unreachable to them too.
- Fix
- Find out why the sitemap paths were refused (security layer, bot rule, rate limit) and confirm from a second network or your server logs; validate the XML only once the file can be fetched.
effort: build - Share of scans
- 0.5% — 49 of 9304 counted scans
- Links
- JSON · glossary: sitemap-index
SITEMAP_DECLARES_NOTHINGA sitemap answers 200 and lists nothinglow1.8%rev 1
- Why it matters
- The file parses as a sitemap and contains no URLs. A crawler that follows the declaration fetches it on every visit and learns nothing, and Search Console reports 0 discovered URLs for it. The usual cause is a generator with an empty source — an entity-map, image or news sitemap whose feed returned nothing — or a rewrite rule left behind. Either populate it or stop declaring it.
- Fix
- Populate the empty sitemap or remove it from robots.txt and from the sitemap index. A file that lists nothing should not be declared.
effort: build - Share of scans
- 1.8% — 170 of 9304 counted scans
- Links
- JSON
NO_SITEMAP_FOUNDNo XML sitemap foundhigh9.8%rev 1
- Why it matters
- Without a sitemap, discovery depends entirely on internal linking. Orphaned and recently published pages may never be found.
- Fix
- Publish a sitemap at /sitemap.xml (most CMSs generate one) and name it in robots.txt.
effort: build - Share of scans
- 9.8% — 911 of 9304 counted scans
- Links
- JSON
LLMS_LINK_BROKENllms.txt links to pages that do not loadlow1.4%rev 1
- Why it matters
- llms.txt is the map an AI agent reads to find your key pages. Some of the links in it answer 404, 410 or a server error, so an agent following the map lands on a dead page or gives up on it.
- Fix
- Fix or remove each dead link in llms.txt so every entry points at a page that answers 200.
effort: content - Share of scans
- 1.4% — 132 of 9304 counted scans
- Links
- JSON · glossary: llms-txt
NO_LLMS_TXTNo llms.txtinfo34.1%rev 1
- Why it matters
- Not a defect. llms.txt is an emerging convention for telling AI systems what a site is and which pages matter.
- Fix
- Publish /llms.txt: a short Markdown file naming the business, what it does, and the pages worth reading, served as text/plain or text/markdown.
generate the fix: /api/tool/llms?domain={domain}&format=txt
effort: content - Share of scans
- 34.1% — 3173 of 9304 counted scans
- Links
- JSON · the fix end to end on a real site
Other
CodeTitleLevelScansRevision
SCRIPT_ONLY_LINKSSome pages can only be reached by a click handler, not a linklow0.1%rev 1
- Why it matters
- Search and AI crawlers follow <a href> links. They do not click. A page that is reached only through a data-href attribute or an onclick that changes the address is, to a crawler, not linked at all: it is found only if the sitemap lists it, and it receives none of the link signals that tell a search engine or an AI system how important it is. This is the commonest way a site builds part of the “invisible web” by accident: product tiles, listing cards and menus wired in JavaScript. Only internal targets with no plain link anywhere on the pages read are counted, so a scripted card that duplicates a real link is not reported.
- Fix
- Make each scripted element a real link: wrap it in <a href="/the-page"> (keep the click handler if you need it), so crawlers can follow it. Until then, list those pages in the sitemap.
effort: content - Share of scans
- 0.1% — 13 of 9304 counted scans
- Links
- JSON
FORM_ONLY_CONTENTSome listings can only be reached by filling in a forminfo0.1%rev 1
- Why it matters
- Crawlers do not fill in forms. A results page reached only by picking options in a filter form (a location, a category, a price band) and pressing submit is part of the “deep web”: the listings exist for people but no search engine or AI crawler can find them unless a plain link or the sitemap leads there. A lone site-search box is not counted, because search results are meant to stay out of indexes. Reported only when no plain link to the results page exists on the pages read.
- Fix
- Give the main results a crawlable route: plain links to category, location or filter pages (for example /services/denver), and list those pages in the sitemap. Keep the form for people.
effort: content - Share of scans
- 0.1% — 10 of 9304 counted scans
- Links
- JSON
FRONTIER_BLOATLinks send crawlers to the same pages under many different addresseslow1.2%rev 1
- Why it matters
- A crawler keeps a queue of every address it has found and still has to fetch (its crawl frontier). Each distinct URL is a separate entry and a separate fetch, so one page linked under several query strings, a session id in every link, “add to cart” and “reply to comment” links, or a calendar that links one month further forever all fill that queue with addresses that teach it nothing. Search engines and AI crawlers spend a limited budget per site; what goes on duplicates is not spent on the pages that matter, and the duplicates compete with each other. Counted from real links on the pages read, not guessed from the sitemap.
- Fix
- Link every page by one address: drop session ids and tracking parameters from internal links, add rel="nofollow" (or a robots.txt Disallow) to add-to-cart, reply-to and calendar-paging links, and give sort/filter variants a canonical tag pointing at the plain page.
effort: content - Share of scans
- 1.2% — 116 of 9304 counted scans
- Links
- JSON
SCANNER_REFUSED_BROWSER_READThe site refuses plain automated requests; it was measured through a real browserlow0.8%rev 1
- Why it matters
- The site’s security layer refused CrawlCheck’s plain HTTP request but served the page to a real browser. Most AI crawlers (GPTBot, ClaudeBot, PerplexityBot and others) make plain HTTP requests and do not run JavaScript, so a shield that tells the two apart this way may be refusing them too. This scan cannot verify that: it measured its own request from its own network, not a verified crawler. Confirm with your server logs or the crawler operators' own diagnostics before treating the site as blocked. The rest of this report was measured through the browser, so its findings describe the site, not the wall.
- Fix
- Decide which automated readers you want. If AI and search crawlers should reach the site, allow verified bots in the bot-protection settings (Cloudflare, Akamai and others have a verified-bot or signed-agent allow list) instead of challenging every client that does not run JavaScript.
effort: config - Share of scans
- 0.8% — 77 of 9304 counted scans
- Links
- JSON
SHOPPING_BLOCKED_STOREBOTrobots.txt keeps Google's Shopping crawler off the storehigh0%rev 1
- Why it matters
- The site sells online, and robots.txt disallows Storebot-Google on its homepage or product pages (that crawler obeys its own group, else the * group). Google says rules for Storebot-Google affect all surfaces of Google Shopping, so those products cannot be listed in the Shopping tab or other Shopping results from the crawl. Googlebot being allowed does not cover it. The usual cause is a Disallow: / under User-agent: * written to keep unknown bots out, or a block-all-bots preset.
- Fix
- Add a User-agent: Storebot-Google group that allows the product pages (Allow: /), or remove the Disallow that covers them from the group it follows.
effort: config - Share of scans
- 0% — 0 of 9304 counted scans
- Links
- JSON
MACHINE_FILE_ACCEPT_DEPENDENTA machine file is served to browsers but not to crawlersmedium0%rev 1
- Why it matters
- A browser asks for text/html and gets the file; a crawler asks with Accept: */* or a text type and gets an error page, so the file it was written for is invisible to it. The cause is usually a CDN, framework or redirect rule that sends non-HTML requests to the 404 handler. Serve the file whatever the Accept header says, then check with: curl -H 'Accept: */*' on the file's URL.
- Fix
- Serve the file the same way whatever Accept header the request carries: the same status and body for Accept: */* as for a browser.
- Share of scans
- 0% — 4 of 9304 counted scans
- Links
- JSON
SITEMAP_LASTMOD_INVALIDA sitemap lastmod is not a valid datelow0.1%rev 1
- Why it matters
- A lastmod value in the sitemap is not in W3C date format (2026-11-06 or 2026-11-06T09:00:00Z). Crawlers ignore a date they cannot parse, so those pages lose the one change signal a sitemap carries.
- Fix
- Write every lastmod as a W3C date: YYYY-MM-DD, or a full timestamp with a time zone.
effort: build - Share of scans
- 0.1% — 6 of 9304 counted scans
- Links
- JSON
SITEMAP_URL_OFF_HOSTThe sitemap lists URLs on another hostlow0.4%rev 1
- Why it matters
- The sitemap names URLs on a different host from the sitemap itself. The sitemap protocol allows only URLs on its own host, so crawlers drop them; if the other host is a staging or old domain, it also advertises that copy.
- Fix
- List only URLs on this host. Give another host its own sitemap, declared in that host's robots.txt.
effort: build - Share of scans
- 0.4% — 37 of 9304 counted scans
- Links
- JSON
SITEMAP_CHILD_BROKENA child sitemap named in the sitemap index does not loadmedium0.2%rev 1
- Why it matters
- The sitemap index lists a child sitemap that answers 404, an error or an HTML page. Every page listed only in that child is missing from the sitemap a crawler reads, so new and deep pages there are found late or not at all.
- Fix
- Publish the missing child sitemap at the URL the index names, or remove that <sitemap> entry from the index. Then confirm every child loads as XML.
effort: build - Share of scans
- 0.2% — 14 of 9304 counted scans
- Links
- JSON
Agentic resource discovery (ARD)
CodeTitleLevelScansRevision
ARD_SOURCE_BROKENThe site advertises an ARD entry source that does not answermedium0%rev 1
- Why it matters
- Agentic Resource Discovery (ARD) lets an agent registry find a site's MCP servers, agent cards and skills. The site points at an entry source - an Agentmap line in robots.txt, a <link rel="ard"> tag or a Link header - and that address does not return a JSON document. A consumer that honours the pointer (the specification says it must) gets nothing; a registry that follows it indexes nothing.
- Fix
- Make the advertised address answer 200 with the JSON manifest, or remove the Agentmap line, <link rel="ard"> or Link header that points at it. Publish the manifest at /.well-known/ard.json, which every consumer reads.
effort: config - Share of scans
- 0% — 1 of 9304 counted scans
- Links
- JSON
ARD_MANIFEST_INVALIDThe ARD manifest breaks the specificationmedium0.8%rev 1
- Why it matters
- The site's ARD manifest (/.well-known/ard.json or the source it advertises) fails the rules every entry must meet: a urn:air:<publisher>:<name> identifier, a displayName, a media type, and exactly one of url or data. A registry rejects the entry, so the resource it describes is not discoverable; where the trust manifest names an identity that does not belong to the URN's publisher, the entry is rejected as namespace squatting.
- Fix
- Give every entry an identifier of the form urn:air:<your-domain>:<name>, a displayName, a type (an IANA media type such as application/mcp-server-card+json) and exactly one of url or data. If an entry carries a trustManifest, its identity must belong to the same domain as the URN. Validate with the official tester: npx ajv-cli validate -s ard-entry.schema.json -d ard.json.
effort: build - Share of scans
- 0.8% — 74 of 9304 counted scans
- Links
- JSON
ARD_MANIFEST_WARNINGSThe ARD manifest is valid but its entries will not be found by searchinfo0.4%rev 1
- Why it matters
- The ARD manifest validates, but entries are missing representativeQueries (or carry fewer than 2 or more than 5), use a renamed media type, or carry non-standard parameters. The semantic index a registry builds comes from representativeQueries: an entry without them is a valid catalogue row that search never returns.
- Fix
- Add 2 to 5 representativeQueries to every entry: the natural-language questions a person would ask that this resource answers. Rename application/mcp-server+json to application/mcp-server-card+json. Use the standard media types without extra parameters.
effort: content - Share of scans
- 0.4% — 41 of 9304 counted scans
- Links
- JSON
ARD_LEGACY_PATHThe ARD manifest is served only under its old nameinfo0.3%rev 1
- Why it matters
- The manifest is served only at /.well-known/ai-catalog.json or with rel="ai-catalog", the names ARD's predecessor used. Consulting those names is optional for consumers; a conformant consumer that reads only /.well-known/ard.json and rel="ard" does not find this site.
- Fix
- Serve the same document at /.well-known/ard.json and change the link relation to rel="ard". The old name can stay for older readers; the new one is the only one consumers must consult.
effort: config - Share of scans
- 0.3% — 29 of 9304 counted scans
- Links
- JSON
WebMCP form tools
CodeTitleLevelScansRevision
WEBMCP_SCHEMA_INVALIDA WebMCP form tool is missing what Chrome needs to register itlow0%rev 1
- Why it matters
- WebMCP lets a page offer its forms to an AI agent as tools: a form with toolname and tooldescription becomes a tool, and each field a parameter. This page has a form with only one of the two attributes, or a required field with no name, so Chrome reports the tool as invalid and an agent cannot call it as written. Lighthouse's Agentic Browsing category scores this as a failure.
- Fix
- Give every annotated form both toolname (a short verb-noun name such as book_appointment) and tooldescription (one sentence saying what submitting it does), and give every required field a name attribute.
effort: content - Share of scans
- 0% — 0 of 9304 counted scans
- Links
- JSON
WEBMCP_SCHEMA_WARNINGSWebMCP form tools have fields an agent cannot identifyinfo0%rev 1
- Why it matters
- The page's WebMCP form tools register, but some fields have no name or no description (toolparamdescription, a <label> or aria-description). An agent sees a parameter it cannot identify, or none at all, and fills it by guessing.
- Fix
- Give every field a name and a description: toolparamdescription="..." on the field, or a <label> tied to it with for=, or aria-description.
effort: content - Share of scans
- 0% — 0 of 9304 counted scans
- Links
- JSON
Trust path
CodeTitleLevelScansRevision
TRUST_PATH_LOOKALIKEThe site sends agents' data to a domain that imitates its ownmedium0%rev 1
- Why it matters
- Following this site the way an agent does (its forms, the MCP server and agent card it advertises, and the login service those name), a place that receives data is on a domain one or two characters off the site's own, written with look-alike characters, or made of the site's name plus a word such as -login or -secure. That is how a phishing or hijacked integration looks; it can also be a domain the owner registered on purpose. An agent cannot tell the difference and trusts it because the site pointed there.
- Fix
- If you own the domain, move the endpoint onto your own domain (or a subdomain of it) so nothing an agent follows looks like an imitation; if you do not, remove the link from the form, card or manifest that names it now and find out how it got there.
effort: config - Share of scans
- 0% — 1 of 9304 counted scans
- Links
- JSON
TRUST_PATH_CLEARTEXTThe site sends agents' data over plain httplow0%rev 1
- Why it matters
- A form, tool endpoint or login service that this site points agents to is addressed with http:// rather than https://. Anything an agent submits there (customer details, a login, a token) travels unencrypted and can be read or altered by anyone on the network path.
- Fix
- Change the address to https:// in the form action, server card, agent card or protected-resource metadata, and serve that endpoint over TLS.
effort: config - Share of scans
- 0% — 0 of 9304 counted scans
- Links
- JSON
NAP and claims
CodeTitleLevelScansRevision
PHONE_SHARED_BY_OTHER_BUSINESSESDirectories list this phone number for other businesseslow0.8%rev 1
- Why it matters
- The site's phone number was searched in directories, and listings under other business names came back (names that are not a spelling of the site's own). One number behind several businesses is a known spam pattern in local search: it can merge the businesses, split their reviews, or make an engine distrust all of them. Old names and sister brands are the usual cause; a number reused after a business closed is the other.
- Fix
- Give each brand its own number, or update the old listings to the current name; if a listed business is not yours, ask the directory to correct the number.
effort: content - Share of scans
- 0.8% — 73 of 9304 counted scans
- Links
- JSON
NAP_NAME_DRIFTA directory prints a different business name than the site declareslow0.6%rev 1
- Why it matters
- Two names on one phone number read as two entities, or as one entity with an unsettled name; an engine reconciling them may print either.
- Fix
- Correct the business name on the directory listing (claim the listing if you have to) so it matches the name the site declares.
- Share of scans
- 0.6% — 58 of 9304 counted scans
- Links
- JSON · glossary: nap · glossary: nap-consistency · glossary: name-drift
NAP_UNDECLARED_LISTINGOff-site listings carry this phone number and the site does not declare theminfo3.6%rev 1
- Why it matters
- A listing the site does not declare is one an engine corroborates without you: whatever it prints becomes your name, hours and category in the answer.
- Fix
- Add the listings you own to sameAs in the business's structured data, and claim or correct the ones you do not.
- Share of scans
- 3.6% — 331 of 9304 counted scans
- Links
- JSON
Robots
CodeTitleLevelScansRevision
ROBOTS_RULES_SHADOWEDrobots.txt rules do not apply to named agentsmedium14%rev 2 · 2026-09-30
- Why it matters
- A crawler reads only the User-agent group that matches it best and ignores every other group, including `*`. Naming an agent and giving it only `Allow: /` therefore deletes all of your `*` Disallow rules for that agent. The file still parses and the agent still reaches your homepage, so this is invisible on inspection — but the paths you meant to keep out of search are open, and search engines will crawl and may index them.
- Fix
- Every User-agent group that names a crawler must repeat the Disallow rules you want applied. A named group with no rules means 'allow everything' for that crawler.
generate the fix: /api/fix/robots?domain={domain}
effort: config - Share of scans
- 14% — 1303 of 9304 counted scans
- Links
- JSON · glossary: robots-txt · glossary: robots-group-merging
INSTRUCTION_CONFLICTThe instruction files contradict each otherlow0.6%rev 1
- Why it matters
- A crawler obeys only its own User-agent group. A Content-Signal or Disallow written under * says nothing to an agent that has its own group, and a group that both invites and forbids leaves the crawler to pick. Put the signal in every group it is meant for, and let the rules under it agree.
- Fix
- Decide the rule once and state the same thing everywhere it is stated: robots.txt, llms.txt, the meta robots tag and the response headers.
- Share of scans
- 0.6% — 59 of 9304 counted scans
- Links
- JSON · glossary: x-robots-tag
ROBOTS_IS_HTMLrobots.txt is HTML, not textcritical0.1%rev 1
- Why it matters
- robots.txt must be plain text. Served as HTML it cannot be parsed, so no crawl directives and no sitemap reference are read.
- Fix
- robots.txt is returning an HTML page. Serve a real text file at /robots.txt, and check that no catch-all route is answering instead.
effort: config - Share of scans
- 0.1% — 9 of 9304 counted scans
- Links
- JSON · glossary: robots-parse-error
ROBOTS_IS_CATCHALLrobots.txt answers with the catch-all page - there is no robots.txtlow0.6%rev 1
- Why it matters
- The site answers robots.txt with the same page it serves for a path that does not exist. Crawlers cannot parse HTML as robots rules, so they proceed as if no file existed: everything allowed. Publish a real text/plain robots.txt.
- Fix
- A catch-all route returns the homepage for any path, including robots.txt. Add a real robots.txt and return 404 for paths that do not exist.
effort: config - Share of scans
- 0.6% — 53 of 9304 counted scans
- Links
- JSON
ROBOTS_LEGACY_AGENT_ONLYrobots.txt rules name a retired crawler, not the one that visitsmedium0.2%rev 1
- Why it matters
- The file writes rules for a crawler name its operator no longer lists (anthropic-ai or Claude-Web for Anthropic, FacebookBot for Meta). The crawler that actually visits uses its current name and obeys only a group naming it, so it follows the `*` group instead and the rule you wrote never reaches it. Reported only when that changes what the crawler may fetch.
- Fix
- Add the current name to the same group: `User-agent: ClaudeBot` under the anthropic-ai / Claude-Web line, `User-agent: meta-externalagent` under FacebookBot. Keep the old line or remove it; it no longer matters.
effort: config - Share of scans
- 0.2% — 21 of 9304 counted scans
- Links
- JSON
ROBOTS_SITEMAP_RELATIVErobots.txt declares a sitemap with a relative pathlow0.1%rev 1
- Why it matters
- A Sitemap line in robots.txt gives a path (/sitemap.xml) instead of a full URL. The standard requires a full URL, and crawlers drop a relative one, so the sitemap is not declared to them through robots.txt.
- Fix
- Write the Sitemap line as a full URL: Sitemap: https://yourdomain.com/sitemap.xml.
effort: config - Share of scans
- 0.1% — 13 of 9304 counted scans
- Links
- JSON
ROBOTS_NOINDEX_LINErobots.txt uses a Noindex line no crawler obeyslow0.2%rev 1
- Why it matters
- robots.txt carries a Noindex: (or Nofollow:) line. Those were never robots.txt directives; Google stopped reading its unofficial Noindex on 1 September 2019 and no other crawler supports it, so the pages it names stay in the index.
- Fix
- Remove the Noindex line and put <meta name="robots" content="noindex"> on those pages (or send X-Robots-Tag: noindex). Do not also Disallow them, or crawlers never see the tag.
effort: config - Share of scans
- 0.2% — 14 of 9304 counted scans
- Links
- JSON
ROBOTS_GROUP_NAME_TYPOrobots.txt has a group for a misspelt crawler namemedium1.9%rev 1
- Why it matters
- A group is written for a name no crawler sends, one spelling away from a real one (Perplexity-Bot for PerplexityBot, GPT-Bot for GPTBot). A crawler obeys only a group naming its own token, so the real crawler follows the `*` group instead and the rules you wrote for it never reach it. Reported only when that changes what the crawler may fetch.
- Fix
- Correct the User-agent line to the crawler's exact token as its operator documents it (PerplexityBot, GPTBot, ClaudeBot). Matching ignores case but not hyphens or spaces.
effort: config - Share of scans
- 1.9% — 180 of 9304 counted scans
- Links
- JSON
ROBOTS_BLOCKEDrobots.txt is unreadablehigh0.4%rev 1
- Why it matters
- A 5xx on robots.txt tells a standards-following crawler to treat the whole site as disallowed (RFC 9309). A 401 or 403 is a 4xx, which the standard treats as no file and so allows everything, though some crawlers read it as a refusal. Either way the rules you wrote are not the rules a crawler read. This is the single most expensive file on the site to get wrong.
- Fix
- robots.txt could not be fetched. Allow it through the security layer; it is the one file every crawler reads first.
effort: config - Share of scans
- 0.4% — 37 of 9304 counted scans
- Links
- JSON · glossary: robots-unavailable
ROBOTS_DISALLOW_ALLrobots.txt blocks every crawler it does not namecritical1.5%rev 2 · 2026-09-30
- Why it matters
- robots.txt instructs every crawler to stay off the entire site.
- Fix
- robots.txt disallows everything. Remove the blanket Disallow: / rule (or restrict it to the paths you actually mean).
effort: config - Share of scans
- 1.5% — 137 of 9304 counted scans
- Links
- JSON
DECLARED_SITEMAP_REFUSEDThe sitemap robots.txt declares refused this clientlow0.8%rev 1
- Fix
- The sitemap named in robots.txt refuses crawlers. Allow it through the security layer.
effort: config - Share of scans
- 0.8% — 77 of 9304 counted scans
- Links
- JSON
DECLARED_SITEMAP_BROKENrobots.txt points at a sitemap that failshigh0.8%rev 1
- Why it matters
- robots.txt advertises a sitemap URL that does not resolve. Crawlers that trust the declaration and fail get nothing.
- Fix
- robots.txt names a sitemap that does not resolve. Fix the URL in robots.txt or publish the sitemap at the path it names.
effort: build - Share of scans
- 0.8% — 75 of 9304 counted scans
- Links
- JSON
ROBOTS_NOT_200robots.txt does not return 200medium4.7%rev 1
- Why it matters
- The response code on robots.txt determines how crawlers treat the whole site. Anything other than 200 or a clean 404 is risky.
- Fix
- Serve robots.txt with status 200 as text/plain. A 404 or 410 means no file: under RFC 9309 a crawler may then fetch everything, so any rule you meant to publish is not in force. A 429 is read by Google as a server error, which pauses crawling: exempt robots.txt from rate limits.
effort: config - Share of scans
- 4.7% — 440 of 9304 counted scans
- Links
- JSON
ROBOTS_NO_SITEMAProbots.txt declares no sitemapmedium9.2%rev 1
- Why it matters
- Adding one Sitemap: line lets crawlers enumerate the site instead of guessing at it.
- Fix
- Add a Sitemap: line to robots.txt pointing at the full sitemap URL, so crawlers find it without guessing.
generate the fix: /api/fix/robots?domain={domain}
effort: config - Share of scans
- 9.2% — 857 of 9304 counted scans
- Links
- JSON
AI_OPTOUT_SETAn AI opt-out signal is setmedium1.8%rev 1
- Why it matters
- A Content-Signal directive is telling AI systems not to use this site. CDNs inject this by default on some plans, so it is frequently set without the owner deciding to set it.
- Fix
- This is a choice, not a defect. If you want answer engines to quote the site, remove the AI opt-out directives; if you do not, leave it.
effort: config - Share of scans
- 1.8% — 165 of 9304 counted scans
- Links
- JSON · glossary: ai-opt-out · glossary: machine-readable-opt-out
ROBOTS_CTYPErobots.txt has an unexpected content typelow0.2%rev 1
- Why it matters
- robots.txt should be served as text/plain. Some parsers are strict about this.
- Fix
- Serve robots.txt as text/plain. Some validators refuse any other content type.
effort: config - Share of scans
- 0.2% — 15 of 9304 counted scans
- Links
- JSON
RENDER_RESOURCE_BLOCKEDThe homepage depends on files this site's own robots.txt disallowsmedium3.1%rev 2 · 2026-09-30
- Why it matters
- A crawler that obeys robots.txt cannot load those files, so it renders a different page from the one visitors see.
- Fix
- Allow crawlers to fetch the CSS and JavaScript the page depends on. Remove the Disallow rules covering those paths, or serve the content without them.
effort: config - Share of scans
- 3.1% — 290 of 9304 counted scans
- Links
- JSON · glossary: blocked-render-resource
Delivery parity
CodeTitleLevelScansRevision
CHALLENGE_SERVED_200A verification page is being served instead of the filecritical0.2%rev 1
- Why it matters
- The server answered with HTTP 200 — so uptime monitors, Search Console and your host all report this as healthy — but the body is a JavaScript verification page, not the file. Crawlers do not run JavaScript on these requests. They receive the interstitial and move on.
- Fix
- A verification page must not answer with status 200. Return 403 or 503 for challenges so crawlers do not index the challenge as the page.
effort: config - Share of scans
- 0.2% — 23 of 9304 counted scans
- Links
- JSON · glossary: challenge-page-at-200
ANSWER_ENGINE_REFUSEDAn answer engine's crawler was refused while a browser was servedhigh5.8%rev 2 · 2026-11-07
- Why it matters
- The same URL, the same minute: an ordinary client was served the page and a named answer-engine crawler was not. That is the edge rather than a robots.txt decision. One caveat this site's own guides state and this finding must too: the scan sent the crawler's name from an address that operator does not publish, and an edge that verifies identity by IP is right to refuse it. So this proves the edge treats the identity differently - not that it refuses the real crawler. Your own logs, checked against the operator's published ranges, settle which. If the edge is refusing the real crawler, nothing else in this report reaches that engine, however good it is.
- Fix
- Allow the named answer-engine crawlers through the security layer (Cloudflare bot rules, WAF, rate limits), or verify their published IP ranges instead of blocking by name.
effort: config - Share of scans
- 5.8% — 535 of 9304 counted scans
- Links
- JSON · glossary: delivery-comparison
UNIFORM_REFUSALThis origin refused CrawlCheck, so the site was never measuredcritical16.1%rev 1
- Fix
- The site refused every request from CrawlCheck. Allow CrawlCheck's address, or run the browser extension from your own network, then re-scan.
effort: config - Share of scans
- 16.1% — 1501 of 9304 counted scans
- Links
- JSON · glossary: uniform-refusal
CRAWLER_SERVED_LESSA named crawler was served materially less text than an ordinary clienthigh1.1%rev 3 · 2026-11-07
- Why it matters
- Same URL, same minute, both answered 200 — but the named crawler received materially fewer words than an unnamed client did. A refusal is loud and easy to find; being served a thinner page is silent, and it is the version an answer engine quotes from.
- Fix
- Serve every crawler the same HTML a browser gets. Check bot-management rules, 'lite' templates and user-agent conditions in the CMS or CDN.
effort: config - Share of scans
- 1.1% — 100 of 9304 counted scans
- Links
- JSON
CRAWLER_REDIRECTED_AWAYA named crawler was sent somewhere the ordinary client was nothigh0.2%rev 3 · 2026-11-07
- Why it matters
- The ordinary client stayed on the URL and the named crawler was sent elsewhere. Whatever is at the destination is what that engine will read and quote instead of this page.
- Fix
- Remove the user-agent based redirect so named crawlers land on the page a visitor lands on.
effort: config - Share of scans
- 0.2% — 18 of 9304 counted scans
- Links
- JSON
CHALLENGE_PINNED_AT_EDGEThe verification page is cached at the CDN edgecritical0.2%rev 1
- Why it matters
- The origin marked this response private and uncacheable. A cache rule with a TTL override stored it anyway, so the verification page is now being handed to every visitor and every crawler from the edge until it expires or is purged.
- Fix
- Purge the edge cache: a challenge page was cached in place of the real file. Then exclude machine files from HTML caching rules.
effort: config - Share of scans
- 0.2% — 15 of 9304 counted scans
- Links
- JSON
HOMEPAGE_REFUSEDThe homepage refused this client, so every homepage section is unmeasuredlow1.2%rev 1
- Fix
- The homepage was refused. Check bot-protection and WAF rules; a challenge page served with a 200 status reads as the site to a crawler.
effort: config - Share of scans
- 1.2% — 107 of 9304 counted scans
- Links
- JSON · glossary: homepage-refused
HOMEPAGE_IS_INTERSTITIALThe homepage answered with an interstitial, not a pagecritical0.5%rev 1
- Fix
- Remove or lower the interstitial (age gate, consent wall, JS challenge) for crawlers, or make sure the real content is in the first response.
effort: config - Share of scans
- 0.5% — 45 of 9304 counted scans
- Links
- JSON · glossary: 202-interstitial
JSON-LD and entities
CodeTitleLevelScansRevision
ENTITY_COLLISIONTwo different entities share the same identifying datamedium2.6%rev 1
- Why it matters
- Two entities on the page resolve to the same identifying data (e.g. name, phone, or place ID), so they cannot both be correct at once.
- Fix
- Merge the duplicate business nodes into one, keep a single @id, and remove any plugin that emits a second empty LocalBusiness record.
generate the fix: /api/fix/entitymap?domain={domain}
effort: build - Share of scans
- 2.6% — 246 of 9304 counted scans
- Links
- JSON · glossary: entity-collision
ENTITY_NO_COORDINATESAn entity is missing geographic coordinatesinfo6.2%rev 1
- Why it matters
- An entity is missing latitude/longitude coordinates, so it cannot be verified against a real-world location.
- Fix
- Add latitude and longitude to the LocalBusiness node's geo property so an engine can place the business on a map without guessing.
generate the fix: /api/fix/entitymap?domain={domain}
effort: content - Share of scans
- 6.2% — 573 of 9304 counted scans
- Links
- JSON · glossary: geocoordinates · glossary: geocoding · glossary: postaladdress
GEO_COUNTRY_MISMATCHPublished coordinates are not in the country the address nameslow0.5%rev 1
- Why it matters
- The latitude/longitude on the page falls outside the country its own address names. A range check cannot catch this: a missing minus sign is still a legal coordinate. Coordinates are read only by machines, so the error produces no visible symptom.
- Fix
- Correct the latitude and longitude in the page's structured data so the point falls at the address. The usual causes are a missing minus sign (western longitudes and southern latitudes are negative) or the two values swapped; the evidence names which.
- Share of scans
- 0.5% — 49 of 9304 counted scans
- Links
- JSON
Answer-engine citations
CodeTitleLevelScansRevision
CITED_WITH_WRONG_FACTSAn answer engine cites the site with the wrong phone or addressmedium—rev 1
- Why it matters
- The engine read the business facts from somewhere other than the site, usually a directory, and is repeating them.
- Fix
- Find the source that prints the wrong phone or address (an old directory listing, a stale page) and correct it, and make the site's own structured data state the right one.
- Share of scans
- measured in answer runs, not counted per scan
- Links
- JSON
REACHABLE_NOT_CITEDAn answer engine can reach the site and does not cite itinfo—rev 1
- Why it matters
- Access is not the problem: the content and authority are.
- Fix
- Nothing blocks this engine. Put the answer to the question in the page's visible text and keep the business facts in structured data, then ask again after the engine has re-read the site.
- Share of scans
- measured in answer runs, not counted per scan
- Links
- JSON
BOT_BLOCKED_NOT_CITEDAn answer engine does not cite the site and robots.txt blocks its crawlermedium—rev 1
- Why it matters
- An engine cannot quote a page its crawler is told not to read.
- Fix
- If you want this engine to cite the site, allow its crawler in robots.txt and at the firewall.
- Share of scans
- measured in answer runs, not counted per scan
- Links
- JSON
BOT_RULES_SHADOWED_NOT_CITEDAn answer engine does not cite the site and its crawler group is shadowedlow—rev 1
- Why it matters
- The crawler's own group in robots.txt does not carry the rules the site meant it to follow.
- Fix
- Repeat the rules you want inside that crawler's own User-agent group, or remove the empty named group, so the crawler reads the rules you intended.
- Share of scans
- measured in answer runs, not counted per scan
- Links
- JSON
NO_LLMS_TXT_NOT_CITEDAn answer engine does not cite the site and there is no llms.txtinfo—rev 1
- Why it matters
- The engine can reach the site; the site gives it no summary to start from.
- Fix
- Publish /llms.txt: a short Markdown file naming the business, what it does, and the pages worth reading, served as text/plain or text/markdown.
- Share of scans
- measured in answer runs, not counted per scan
- Links
- JSON
Capabilities
CodeTitleLevelScansRevision
CAPABILITY_DECLARED_PUBLIC_OBSERVED_AUTHDeclared capability declared public observed authnot scored—rev 1
- Fix
- the finding in your report names what to change
- Share of scans
- listed in the record's capabilities section, not counted per scan
- Links
- JSON
CAPABILITY_DECLARED_ENDPOINT_MISSINGDeclared capability declared endpoint missingnot scored—rev 1
- Fix
- the finding in your report names what to change
- Share of scans
- listed in the record's capabilities section, not counted per scan
- Links
- JSON
CAPABILITY_DECLARED_NO_REQUIRED_INPUT_OBSERVED_REJECTEDDeclared capability declared no required input observed rejectednot scored—rev 1
- Fix
- the finding in your report names what to change
- Share of scans
- listed in the record's capabilities section, not counted per scan
- Links
- JSON
CAPABILITY_DECLARED_MEDIA_MISMATCHDeclared capability declared media mismatchnot scored—rev 1
- Fix
- the finding in your report names what to change
- Share of scans
- listed in the record's capabilities section, not counted per scan
- Links
- JSON
CAPABILITY_DECLARED_TOOL_NOT_LISTEDDeclared capability declared tool not listednot scored—rev 1
- Fix
- the finding in your report names what to change
- Share of scans
- listed in the record's capabilities section, not counted per scan
- Links
- JSON
CAPABILITY_LISTED_TOOL_NOT_DECLAREDDeclared capability listed tool not declarednot scored—rev 1
- Fix
- the finding in your report names what to change
- Share of scans
- listed in the record's capabilities section, not counted per scan
- Links
- JSON
Self-audit: defects in our own record
CodeTitleLevelScansRevision
FETCHER_DISAGREEMENTTwo of our fetch layers disagree about a machine filemedium—rev 1
- Why it matters
- A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site.
- Fix
- the finding in your report names what to change
- Share of scans
- about our own record, not counted per scan
- Links
- JSON
BENCH_POOL_MISMATCHThe benchmark's pool size contradicts its scopemedium—rev 1
- Why it matters
- A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site.
- Fix
- the finding in your report names what to change
- Share of scans
- about our own record, not counted per scan
- Links
- JSON
COHORT_BELOW_THRESHOLDRanked against a cohort under the minimumlow—rev 1
- Why it matters
- A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site.
- Fix
- the finding in your report names what to change
- Share of scans
- about our own record, not counted per scan
- Links
- JSON
SECTION_SCORE_OUT_OF_RANGEA section score outside 0 to 100high—rev 1
- Why it matters
- A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site.
- Fix
- the finding in your report names what to change
- Share of scans
- about our own record, not counted per scan
- Links
- JSON
RANKED_VALUE_NOT_REPRODUCIBLEThe ranked number does not follow from the printed tablehigh—rev 1
- Why it matters
- A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site.
- Fix
- the finding in your report names what to change
- Share of scans
- about our own record, not counted per scan
- Links
- JSON
SECTION_WEIGHT_UNDECLAREDA section scored without a declared weightmedium—rev 1
- Why it matters
- A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site.
- Fix
- the finding in your report names what to change
- Share of scans
- about our own record, not counted per scan
- Links
- JSON
PATH_NOT_MEASUREDA deep path was asked for, the home page was measuredlow—rev 1
- Why it matters
- A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site.
- Fix
- the finding in your report names what to change
- Share of scans
- about our own record, not counted per scan
- Links
- JSON
PLACEHOLDER_IN_RECORDOne of our placeholders reached the recordhigh—rev 1
- Why it matters
- A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site.
- Fix
- the finding in your report names what to change
- Share of scans
- about our own record, not counted per scan
- Links
- JSON
GRADE_OFF_ITS_OWN_SCALEThe letter does not follow the record's own thresholdshigh—rev 1
- Why it matters
- A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site.
- Fix
- the finding in your report names what to change
- Share of scans
- about our own record, not counted per scan
- Links
- JSON
GRADED_THROUGH_A_WALLA grade on a site that refused CrawlCheckcritical—rev 1
- Why it matters
- A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site.
- Fix
- The reading was taken through a block page. Allow CrawlCheck and re-scan for a real grade.
effort: config - Share of scans
- about our own record, not counted per scan
- Links
- JSON
GRADE_MISSINGA graded record with no grademedium—rev 1
- Why it matters
- A contradiction inside CrawlCheck's own record: a defect in CrawlCheck, never in the scanned site.
- Fix
- the finding in your report names what to change
- Share of scans
- about our own record, not counted per scan
- Links
- JSON
MCP servers
CodeTitleLevelScansRevision
MCP_TOOL_POISONINGAn MCP server's tool descriptions carry text aimed at the modelhigh—rev 1
- Why it matters
- Text a client hands the model from tools/list hides characters a person cannot see, tells the model to ignore its instructions or keep something from the user, asks for secrets or the conversation, or tells the model how to use other tools. A model reads it as an instruction; the person approving the server usually never sees it.
- Fix
- Rewrite the tool descriptions so they describe the tool and nothing else; remove any text that addresses the model or tells it what to do.
- Share of scans
- measured on remote MCP servers in the official MCP Registry, not per site scan: see https://crawlcheck.io/api/mcp/index (tool_poisoning)
- Links
- JSON
High-risk field watch
CodeTitleLevelScansRevision
HIGH_RISK_FIELD_CHANGEDA high-risk field changed and the owner has not confirmed ithigh—rev 1
- Why it matters
- One of the fields an attacker changes first moved since CrawlCheck last read it: the payment processors loaded on the homepage or checkout page, the host checkout links go to, a cryptocurrency wallet address printed on the page, the OAuth authorization server the site's MCP endpoint names, the domain's MX hosts, or its registry-listed MCP endpoints. Watched fields: payment_processors, checkout_host, wallet_addresses, oauth_authorization_server, mx_hosts, mcp_endpoints. The first reading is the baseline and never alerts.
- Fix
- If you made the change, confirm it from the owner's session. If you did not, treat it as a compromise: restore the previous value and find out who changed the site.
- Share of scans
- measured by the daily high-risk field watch, not per site scan: see https://crawlcheck.io/api/v1/high-risk
- Links
- JSON
Generated 2026-11-22 01:25 UTC. A rule you think is wrong? Write to hello@crawlcheck.io with the report id; corrections are published as a new revision.