CrawlCheck
Rules · fixes by platform

How to fix crawler findings on WordPress

Where each fix is made on WordPress, including where WordPress does not let you make it and what to do instead. Every report on a WordPress site shows these steps under the finding they fix.

WordPress · Shopify · Webflow · Squarespace · Wix · Next.js · Drupal · Cloudflare

Edit robots.txt

  1. WordPress serves a virtual robots.txt. Edit it in your SEO plugin: Yoast SEO → Tools → File editor; Rank Math → General Settings → Edit robots.txt; All in One SEO → Tools → Robots.txt Editor.
  2. A real robots.txt file in the site root (the folder with wp-config.php) overrides every plugin. If one exists, edit that file instead, through your host's file manager or SFTP.
  3. Keep one group per crawler: a crawler that has its own User-agent group ignores the * group entirely, so repeat any Disallow lines it still needs.

Fixes these findings: ROBOTS_RULES_SHADOWED, ROBOTS_NO_SITEMAP, ROBOTS_NOT_200, ROBOTS_IS_HTML, ROBOTS_IS_CATCHALL, ROBOTS_DISALLOW_ALL, ROBOTS_CTYPE, ROBOTS_BLOCKED, RENDER_RESOURCE_BLOCKED, INSTRUCTION_CONFLICT, DECLARED_SITEMAP_REFUSED, DECLARED_SITEMAP_BROKEN, SITEMAP_BLOCKED, AI_OPTOUT_SET, BOT_BLOCKED_NOT_CITED, BOT_RULES_SHADOWED_NOT_CITED

Publish a sitemap

  1. WordPress 5.5 and later serve /wp-sitemap.xml; Yoast and Rank Math replace it with /sitemap_index.xml. Keep exactly one turned on.
  2. Core's wp-sitemap.xml has no lastmod dates; Yoast and Rank Math sitemaps do. If the finding is missing lastmod, use the plugin sitemap.
  3. Add the sitemap's address to robots.txt as Sitemap: https://yourdomain/sitemap_index.xml.

Fixes these findings: NO_SITEMAP_FOUND, SITEMAP_NO_LASTMOD, SITEMAP_IS_HTML, SITEMAP_DECLARES_NOTHING, SITEMAP_UNREADABLE, SITEMAP_ORIGIN_ERROR, ROBOTS_NO_SITEMAP, DECLARED_SITEMAP_BROKEN, DECLARED_SITEMAP_REFUSED

Publish llms.txt

  1. Upload llms.txt to the site root, the folder with wp-config.php, through your host's file manager or SFTP.
  2. Recent versions of Yoast SEO and Rank Math can generate llms.txt for you; use one source only.

Fixes these findings: NO_LLMS_TXT, NO_LLMS_TXT_NOT_CITED

Send HSTS

  1. Set it at the web server or CDN, not in WordPress: Apache .htaccess Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"; Nginx add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
  2. Add includeSubDomains only when every subdomain serves HTTPS, and preload only when you mean to stay on HTTPS for good.

Fixes these findings: HSTS_MISSING, HSTS_PRELOAD_INELIGIBLE, HSTS_PRELOAD_REMOVED

Let crawlers through

  1. Security plugins challenge unknown clients. Check each one you run: Wordfence (Firewall → Rate limiting, fake-crawler blocking), MalCare (its bot protection serves a challenge page with status 200), Sucuri firewall, and your host's own bot filter.
  2. Allow the crawlers you want, or turn the challenge off for verified crawlers, then rescan to confirm each crawler gets the real page.

Fixes these findings: CHALLENGE_SERVED_200, CHALLENGE_PINNED_AT_EDGE, ANSWER_ENGINE_REFUSED, UNIFORM_REFUSAL, HOMEPAGE_REFUSED, HOMEPAGE_IS_INTERSTITIAL, ORIGIN_REFUSED_SCANNER, CRAWLER_SERVED_LESS, CRAWLER_REDIRECTED_AWAY, BOT_BLOCKED_NOT_CITED

Serve the text, not just code

  1. Page builders add most of the code: in Elementor turn on the optimized DOM output and improved asset loading features; remove plugins that inject scripts on every page.
  2. Let a cache or optimization plugin defer JavaScript and remove unused CSS, then rescan to see the text share move.

Fixes these findings: PAGE_IS_MOSTLY_CODE, CONTENT_NEEDS_JAVASCRIPT

Fix the structured data

  1. Keep one source of business markup: Rank Math → Titles & Meta → Local SEO, or Yoast Local SEO. Turn off the same block in your theme or any second plugin.
  2. Fill name, address, phone, geo coordinates and opening hours there, matching the Business Profile exactly.

Fixes these findings: ENTITY_COLLISION, ENTITY_NO_COORDINATES, GEO_COUNTRY_MISMATCH, NAP_NAME_DRIFT, NAP_UNDECLARED_LISTING

Clear stale copies

  1. Purge the server cache first (your host's Varnish or page cache), then the caching plugin, then the CDN; purging in the other order lets the CDN refill from the stale layer.
  2. Give robots.txt, sitemaps and llms.txt a short cache lifetime.

Fixes these findings: STALE_CACHE_SERVED, MACHINE_FILE_CACHE_STALE, MACHINE_FILE_CACHE_SPLIT

Check the fix

Scan the site again on the homepage: a fix counts when the finding is gone from the new report. JSON for tools: /api/v1/fix/platform?platform=wordpress.