# Block SEO, advertising and monitoring crawlers — caddy # 27 user agents, generated from https://crawlcheck.io/ai-crawlers # A user agent string can be forged; this stops crawlers that name themselves. # Inside a site block. @cc_block header_regexp User-Agent (?i)(KlaviyoAIBot|Mediapartners-Google|APIs-Google|BingPreview|Magpie-crawler|AwarioRssBot|AwarioSmartBot|AhrefsBot|SemrushBot|MJ12bot|DotBot|rogerbot|DataForSeoBot|BLEXBot|VelenPublicWebCrawler|Google-CloudVertexBot|CloudflareBrowserRenderingCrawler|Cloudflare-AutoRAG|Cloudflare-AI-Search-External|Peer39_Crawler|AdsBot-Google|AmazonAdBot|AdIdxBot|Scrapy|SemrushBot-OCOB|AwarioBot|OAI-AdsBot) respond @cc_block 403