CrawlCheck
Rules · fixes by platform

How to fix crawler findings on Shopify

Where each fix is made on Shopify, including where Shopify does not let you make it and what to do instead. Every report on a Shopify site shows these steps under the finding they fix.

WordPress · Shopify · Webflow · Squarespace · Wix · Next.js · Drupal · Cloudflare

Edit robots.txt

  1. Online Store → Themes → … → Edit code → Templates → Add a new template → robots.txt. Shopify creates robots.txt.liquid.
  2. Add your lines inside the existing loop over robots.default_groups instead of replacing it, so Shopify's own rules (checkout, cart, admin) stay.
  3. Delete robots.txt.liquid to go back to Shopify's default.

Fixes these findings: ROBOTS_RULES_SHADOWED, ROBOTS_NO_SITEMAP, ROBOTS_NOT_200, ROBOTS_IS_HTML, ROBOTS_IS_CATCHALL, ROBOTS_DISALLOW_ALL, ROBOTS_CTYPE, ROBOTS_BLOCKED, RENDER_RESOURCE_BLOCKED, INSTRUCTION_CONFLICT, DECLARED_SITEMAP_REFUSED, DECLARED_SITEMAP_BROKEN, SITEMAP_BLOCKED, AI_OPTOUT_SET, BOT_BLOCKED_NOT_CITED, BOT_RULES_SHADOWED_NOT_CITED

Publish a sitemap

  1. Shopify writes /sitemap.xml itself (with lastmod) and lists it in its default robots.txt; it cannot be edited.
  2. If a scan says the sitemap is missing or not declared, check whether a custom robots.txt.liquid dropped the Sitemap line.

Fixes these findings: NO_SITEMAP_FOUND, SITEMAP_NO_LASTMOD, SITEMAP_IS_HTML, SITEMAP_DECLARES_NOTHING, SITEMAP_UNREADABLE, SITEMAP_ORIGIN_ERROR, ROBOTS_NO_SITEMAP, DECLARED_SITEMAP_BROKEN, DECLARED_SITEMAP_REFUSED

Publish llms.txt

  1. Shopify cannot serve a file you upload at the root.
  2. Upload llms.txt under Content → Files, copy its URL, then add a URL redirect from /llms.txt to it (search "URL redirects" in the admin). Readers that follow redirects get the file; some agents do not follow a redirect to another host.

Fixes these findings: NO_LLMS_TXT, NO_LLMS_TXT_NOT_CITED

Send HSTS

  1. Shopify sets response headers itself; you cannot add HSTS on Shopify. If the domain runs through Cloudflare, set it there.

Fixes these findings: HSTS_MISSING, HSTS_PRELOAD_INELIGIBLE, HSTS_PRELOAD_REMOVED

Let crawlers through

  1. Shopify's own bot protection has no setting. Look for an app that blocks bots, and for Cloudflare or another proxy in front of the store.

Fixes these findings: CHALLENGE_SERVED_200, CHALLENGE_PINNED_AT_EDGE, ANSWER_ENGINE_REFUSED, UNIFORM_REFUSAL, HOMEPAGE_REFUSED, HOMEPAGE_IS_INTERSTITIAL, ORIGIN_REFUSED_SCANNER, CRAWLER_SERVED_LESS, CRAWLER_REDIRECTED_AWAY, BOT_BLOCKED_NOT_CITED

Serve the text, not just code

  1. Uninstalled apps often leave script tags in the theme: remove them from theme.liquid and snippets.
  2. Turn off app embeds (Theme editor → App embeds) on pages that do not use them.

Fixes these findings: PAGE_IS_MOSTLY_CODE, CONTENT_NEEDS_JAVASCRIPT

Fix the structured data

  1. The theme prints Organization and Product JSON-LD in theme.liquid or a snippet: edit it there, and do not stack an SEO app's schema on top of the theme's.

Fixes these findings: ENTITY_COLLISION, ENTITY_NO_COORDINATES, GEO_COUNTRY_MISMATCH, NAP_NAME_DRIFT, NAP_UNDECLARED_LISTING

Clear stale copies

  1. Shopify's CDN refreshes on its own after a change; if Cloudflare is in front, purge there.

Fixes these findings: STALE_CACHE_SERVED, MACHINE_FILE_CACHE_STALE, MACHINE_FILE_CACHE_SPLIT

Check the fix

Scan the site again on the homepage: a fix counts when the finding is gone from the new report. JSON for tools: /api/v1/fix/platform?platform=shopify.