# Block training and corpus crawlers, keep search and answer engines — caddy # 36 user agents, generated from https://crawlcheck.io/ai-crawlers # A user agent string can be forged; this stops crawlers that name themselves. # Inside a site block. @cc_block header_regexp User-Agent (?i)(img2dataset|TurnitinBot|archive\.org_bot|ia_archiver|omgili|cohere-training-data-crawler|PanguBot|Ai2Bot-Dolma|FriendlyCrawler|MyCentralAIScraperBot|DeepSeekBot|ICC-Crawler|GoogleOther|Amazonbot|GPTBot|ClaudeBot|anthropic-ai|Claude-Web|CCBot|Bytespider|meta-externalagent|FacebookBot|cohere-ai|Diffbot|Omgilibot|ImagesiftBot|Timpibot|AI2Bot|Applebot-Extended-Ads|TikTokSpider|QuillBot|Webzio-Extended|ProRataInc|MistralAI-Training|GoogleOther-Image|GoogleOther-Video) respond @cc_block 403