CrawlCheck
Rules · fixes by platform

How to fix crawler findings on Next.js

Where each fix is made on Next.js, including where Next.js does not let you make it and what to do instead. Every report on a Next.js site shows these steps under the finding they fix.

WordPress · Shopify · Webflow · Squarespace · Wix · Next.js · Drupal · Cloudflare

Edit robots.txt

  1. App Router: add app/robots.ts that returns a MetadataRoute.Robots object (rules per userAgent, sitemap URL). Pages Router or static export: put robots.txt in public/.
  2. Ship one of the two, not both: the route handler and a public/robots.txt collide.

Fixes these findings: ROBOTS_RULES_SHADOWED, ROBOTS_NO_SITEMAP, ROBOTS_NOT_200, ROBOTS_IS_HTML, ROBOTS_IS_CATCHALL, ROBOTS_DISALLOW_ALL, ROBOTS_CTYPE, ROBOTS_BLOCKED, RENDER_RESOURCE_BLOCKED, INSTRUCTION_CONFLICT, DECLARED_SITEMAP_REFUSED, DECLARED_SITEMAP_BROKEN, SITEMAP_BLOCKED, AI_OPTOUT_SET, BOT_BLOCKED_NOT_CITED, BOT_RULES_SHADOWED_NOT_CITED

Publish a sitemap

  1. App Router: app/sitemap.ts returning MetadataRoute.Sitemap, one entry per URL with lastModified set to the content's real change date.
  2. Reference it from app/robots.ts (sitemap field) so crawlers find it.

Fixes these findings: NO_SITEMAP_FOUND, SITEMAP_NO_LASTMOD, SITEMAP_IS_HTML, SITEMAP_DECLARES_NOTHING, SITEMAP_UNREADABLE, SITEMAP_ORIGIN_ERROR, ROBOTS_NO_SITEMAP, DECLARED_SITEMAP_BROKEN, DECLARED_SITEMAP_REFUSED

Publish llms.txt

  1. Put llms.txt in public/ for a static file, or add app/llms.txt/route.ts that returns it with content-type text/plain.

Fixes these findings: NO_LLMS_TXT, NO_LLMS_TXT_NOT_CITED

Send HSTS

  1. next.config.js headers(): source '/(.*)' with Strict-Transport-Security: max-age=63072000; includeSubDomains. Some hosts send it already; check the live response before adding a second copy.

Fixes these findings: HSTS_MISSING, HSTS_PRELOAD_INELIGIBLE, HSTS_PRELOAD_REMOVED

Let crawlers through

  1. Check middleware.ts for user-agent checks, and your host's firewall: on Vercel, Firewall → Bot protection and Attack Challenge Mode challenge crawlers.

Fixes these findings: CHALLENGE_SERVED_200, CHALLENGE_PINNED_AT_EDGE, ANSWER_ENGINE_REFUSED, UNIFORM_REFUSAL, HOMEPAGE_REFUSED, HOMEPAGE_IS_INTERSTITIAL, ORIGIN_REFUSED_SCANNER, CRAWLER_SERVED_LESS, CRAWLER_REDIRECTED_AWAY, BOT_BLOCKED_NOT_CITED

Serve the text, not just code

  1. Render content on the server: keep it in Server Components (the App Router default) or use getStaticProps / getServerSideProps; avoid "use client" at the page root and content fetched only in useEffect.
  2. Check View Source, not the browser inspector: the text must be in the HTML the server sends.

Fixes these findings: PAGE_IS_MOSTLY_CODE, CONTENT_NEEDS_JAVASCRIPT

Fix the structured data

  1. Render one <script type="application/ld+json"> per entity in the layout or page (JSON.stringify of the object), with a stable @id.

Fixes these findings: ENTITY_COLLISION, ENTITY_NO_COORDINATES, GEO_COUNTRY_MISMATCH, NAP_NAME_DRIFT, NAP_UNDECLARED_LISTING

Clear stale copies

  1. Revalidate (revalidatePath / revalidateTag or a shorter revalidate time) and redeploy; on a CDN, purge the paths that changed.

Fixes these findings: STALE_CACHE_SERVED, MACHINE_FILE_CACHE_STALE, MACHINE_FILE_CACHE_SPLIT

Check the fix

Scan the site again on the homepage: a fix counts when the finding is gone from the new report. JSON for tools: /api/v1/fix/platform?platform=nextjs.