CrawlCheck
Rules · fixes by platform

How to fix crawler findings on Drupal

Where each fix is made on Drupal, including where Drupal does not let you make it and what to do instead. Every report on a Drupal site shows these steps under the finding they fix.

WordPress · Shopify · Webflow · Squarespace · Wix · Next.js · Drupal · Cloudflare

Edit robots.txt

  1. Drupal ships robots.txt in the web root next to index.php; edit it there, or use the RobotsTxt module to edit it in the admin.
  2. With Composer-managed core, exclude robots.txt from drupal/core-composer-scaffold (file-mapping "[web-root]/robots.txt": false) so a core update does not overwrite your edits.

Fixes these findings: ROBOTS_RULES_SHADOWED, ROBOTS_NO_SITEMAP, ROBOTS_NOT_200, ROBOTS_IS_HTML, ROBOTS_IS_CATCHALL, ROBOTS_DISALLOW_ALL, ROBOTS_CTYPE, ROBOTS_BLOCKED, RENDER_RESOURCE_BLOCKED, INSTRUCTION_CONFLICT, DECLARED_SITEMAP_REFUSED, DECLARED_SITEMAP_BROKEN, SITEMAP_BLOCKED, AI_OPTOUT_SET, BOT_BLOCKED_NOT_CITED, BOT_RULES_SHADOWED_NOT_CITED

Publish a sitemap

  1. Install Simple XML Sitemap (or XML Sitemap); it serves /sitemap.xml with lastmod and can add itself to robots.txt.

Fixes these findings: NO_SITEMAP_FOUND, SITEMAP_NO_LASTMOD, SITEMAP_IS_HTML, SITEMAP_DECLARES_NOTHING, SITEMAP_UNREADABLE, SITEMAP_ORIGIN_ERROR, ROBOTS_NO_SITEMAP, DECLARED_SITEMAP_BROKEN, DECLARED_SITEMAP_REFUSED

Publish llms.txt

  1. Put llms.txt in the web root next to index.php.

Fixes these findings: NO_LLMS_TXT, NO_LLMS_TXT_NOT_CITED

Send HSTS

  1. Security Kit (seckit) module → SSL/TLS → HTTP Strict Transport Security, or set the header in the web server config.

Fixes these findings: HSTS_MISSING, HSTS_PRELOAD_INELIGIBLE, HSTS_PRELOAD_REMOVED

Let crawlers through

  1. Check modules that block or rate-limit clients (Antibot, CrowdSec, Perimeter, Honeypot rules on full pages) and the host's firewall.

Fixes these findings: CHALLENGE_SERVED_200, CHALLENGE_PINNED_AT_EDGE, ANSWER_ENGINE_REFUSED, UNIFORM_REFUSAL, HOMEPAGE_REFUSED, HOMEPAGE_IS_INTERSTITIAL, ORIGIN_REFUSED_SCANNER, CRAWLER_SERVED_LESS, CRAWLER_REDIRECTED_AWAY, BOT_BLOCKED_NOT_CITED

Serve the text, not just code

  1. Turn on CSS and JavaScript aggregation (Configuration → Performance) and remove modules that add scripts to every page.

Fixes these findings: PAGE_IS_MOSTLY_CODE, CONTENT_NEEDS_JAVASCRIPT

Fix the structured data

  1. Schema.org Metatag module: set the Organization or LocalBusiness fields once, site-wide.

Fixes these findings: ENTITY_COLLISION, ENTITY_NO_COORDINATES, GEO_COUNTRY_MISMATCH, NAP_NAME_DRIFT, NAP_UNDECLARED_LISTING

Clear stale copies

  1. Clear caches (drush cr or Configuration → Performance), then purge the CDN or Varnish.

Fixes these findings: STALE_CACHE_SERVED, MACHINE_FILE_CACHE_STALE, MACHINE_FILE_CACHE_SPLIT

Check the fix

Scan the site again on the homepage: a fix counts when the finding is gone from the new report. JSON for tools: /api/v1/fix/platform?platform=drupal.