# robots.txt for frase.io — corrected by CrawlCheck, 2026-09-03
# Source: the file the site served at the time. Every change is listed here; nothing else was touched.
# Changes:
#   - Read from https://www.frase.io/robots.txt (the apex redirected there); publish at the host you serve.
#   - 11 named groups lacked Disallow lines the * group carries. A crawler obeys only its own most specific group and ignores *, so those rules never applied to it. Copied in.
# Verify after publishing: curl -sI https://frase.io/robots.txt must show text/plain, on the bare URL, with no cache-buster.

User-Agent: *
Allow: /
Disallow: /sentry-test
Content-Signal: search=yes, ai-input=yes, ai-train=no

User-Agent: GPTBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /sentry-test

User-Agent: OAI-SearchBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /sentry-test

User-Agent: ChatGPT-User
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /sentry-test

User-Agent: ClaudeBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /sentry-test

User-Agent: Claude-Web
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /sentry-test

User-Agent: anthropic-ai
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /sentry-test

User-Agent: PerplexityBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /sentry-test

User-Agent: Perplexity-User
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /sentry-test

User-Agent: Google-Extended
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /sentry-test

User-Agent: Applebot-Extended
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /sentry-test

User-Agent: CCBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /sentry-test

Host: https://www.frase.io
Sitemap: https://www.frase.io/sitemap.xml