CrawlCheck
Rules · fixes by platform

How to fix crawler findings on Cloudflare

Many crawler problems come from Cloudflare settings in front of the site, whatever runs behind it. These are the Cloudflare steps for each kind of finding.

WordPress · Shopify · Webflow · Squarespace · Wix · Next.js · Drupal · Cloudflare

Edit robots.txt

  1. If Cloudflare's managed robots.txt is on (Security → Bots, or AI Crawl Control → robots.txt), Cloudflare adds its own lines above yours. Turn it off, or keep it knowing those lines are there.
  2. Robots.txt is cached at the edge: purge it (Caching → Configuration → Custom purge → the robots.txt URL) after you change it.

Fixes these findings: ROBOTS_RULES_SHADOWED, ROBOTS_NO_SITEMAP, ROBOTS_NOT_200, ROBOTS_IS_HTML, ROBOTS_IS_CATCHALL, ROBOTS_DISALLOW_ALL, ROBOTS_CTYPE, ROBOTS_BLOCKED, RENDER_RESOURCE_BLOCKED, INSTRUCTION_CONFLICT, DECLARED_SITEMAP_REFUSED, DECLARED_SITEMAP_BROKEN, SITEMAP_BLOCKED, AI_OPTOUT_SET, BOT_BLOCKED_NOT_CITED, BOT_RULES_SHADOWED_NOT_CITED

Send HSTS

  1. SSL/TLS → Edge Certificates → HTTP Strict Transport Security (HSTS) → Enable: max-age 12 months; include subdomains and preload only when every subdomain is on HTTPS for good.

Fixes these findings: HSTS_MISSING, HSTS_PRELOAD_INELIGIBLE, HSTS_PRELOAD_REMOVED

Let crawlers through

  1. Security → Bots: Bot Fight Mode challenges crawlers and cannot be bypassed with WAF rules. Turn it off, or on a paid plan use Super Bot Fight Mode with verified bots allowed.
  2. Check "Block AI bots" and AI Crawl Control for the crawlers you want to let in.
  3. Security → WAF → Custom rules: a rule with action Skip for Known Bots (cf.client.bot) keeps verified crawlers out of your other challenges.

Fixes these findings: CHALLENGE_SERVED_200, CHALLENGE_PINNED_AT_EDGE, ANSWER_ENGINE_REFUSED, UNIFORM_REFUSAL, HOMEPAGE_REFUSED, HOMEPAGE_IS_INTERSTITIAL, ORIGIN_REFUSED_SCANNER, CRAWLER_SERVED_LESS, CRAWLER_REDIRECTED_AWAY, BOT_BLOCKED_NOT_CITED

Clear stale copies

  1. Caching → Configuration → Purge Cache (custom purge for the exact URLs) after each change.
  2. Caching → Cache Rules: bypass the cache or set a short edge TTL for robots.txt, sitemaps and llms.txt.

Fixes these findings: STALE_CACHE_SERVED, MACHINE_FILE_CACHE_STALE, MACHINE_FILE_CACHE_SPLIT

Check the fix

Scan the site again on the homepage: a fix counts when the finding is gone from the new report. JSON for tools: /api/v1/fix/platform?platform=cloudflare.