{"kind":"crawlcheck-platform-fix","v":1,"platform":"cloudflare","platform_name":"Cloudflare","code":null,"groups":[{"group":"robots","title":"Edit robots.txt","codes":["ROBOTS_RULES_SHADOWED","ROBOTS_NO_SITEMAP","ROBOTS_NOT_200","ROBOTS_IS_HTML","ROBOTS_IS_CATCHALL","ROBOTS_DISALLOW_ALL","ROBOTS_CTYPE","ROBOTS_BLOCKED","RENDER_RESOURCE_BLOCKED","INSTRUCTION_CONFLICT","DECLARED_SITEMAP_REFUSED","DECLARED_SITEMAP_BROKEN","SITEMAP_BLOCKED","AI_OPTOUT_SET","BOT_BLOCKED_NOT_CITED","BOT_RULES_SHADOWED_NOT_CITED"],"steps":["If Cloudflare's managed robots.txt is on (Security → Bots, or AI Crawl Control → robots.txt), Cloudflare adds its own lines above yours. Turn it off, or keep it knowing those lines are there.","Robots.txt is cached at the edge: purge it (Caching → Configuration → Custom purge → the robots.txt URL) after you change it."],"page":"https://crawlcheck.io/fix/cloudflare#robots"},{"group":"hsts","title":"Send HSTS","codes":["HSTS_MISSING","HSTS_PRELOAD_INELIGIBLE","HSTS_PRELOAD_REMOVED"],"steps":["SSL/TLS → Edge Certificates → HTTP Strict Transport Security (HSTS) → Enable: max-age 12 months; include subdomains and preload only when every subdomain is on HTTPS for good."],"page":"https://crawlcheck.io/fix/cloudflare#hsts"},{"group":"bots","title":"Let crawlers through","codes":["CHALLENGE_SERVED_200","CHALLENGE_PINNED_AT_EDGE","ANSWER_ENGINE_REFUSED","UNIFORM_REFUSAL","HOMEPAGE_REFUSED","HOMEPAGE_IS_INTERSTITIAL","ORIGIN_REFUSED_SCANNER","CRAWLER_SERVED_LESS","CRAWLER_REDIRECTED_AWAY","BOT_BLOCKED_NOT_CITED"],"steps":["Security → Bots: Bot Fight Mode challenges crawlers and cannot be bypassed with WAF rules. Turn it off, or on a paid plan use Super Bot Fight Mode with verified bots allowed.","Check \"Block AI bots\" and AI Crawl Control for the crawlers you want to let in.","Security → WAF → Custom rules: a rule with action Skip for Known Bots (cf.client.bot) keeps verified crawlers out of your other challenges."],"page":"https://crawlcheck.io/fix/cloudflare#bots"},{"group":"cache","title":"Clear stale copies","codes":["STALE_CACHE_SERVED","MACHINE_FILE_CACHE_STALE","MACHINE_FILE_CACHE_SPLIT"],"steps":["Caching → Configuration → Purge Cache (custom purge for the exact URLs) after each change.","Caching → Cache Rules: bypass the cache or set a short edge TTL for robots.txt, sitemaps and llms.txt."],"page":"https://crawlcheck.io/fix/cloudflare#cache"}],"rules":"https://crawlcheck.io/rules","reading":"Where to make each fix on Cloudflare. The general fix for each finding is in its report and at /rules/<CODE>; these steps say where that change is made on this platform, and say so when the platform cannot do it."}