# Block SEO, advertising and monitoring crawlers — apache # 27 user agents, generated from https://crawlcheck.io/ai-crawlers # A user agent string can be forged; this stops crawlers that name themselves. # .htaccess or a virtual host; needs mod_rewrite. RewriteEngine On RewriteCond %{HTTP_USER_AGENT} (KlaviyoAIBot|Mediapartners-Google|APIs-Google|BingPreview|Magpie-crawler|AwarioRssBot|AwarioSmartBot|AhrefsBot|SemrushBot|MJ12bot|DotBot|rogerbot|DataForSeoBot|BLEXBot|VelenPublicWebCrawler|Google-CloudVertexBot|CloudflareBrowserRenderingCrawler|Cloudflare-AutoRAG|Cloudflare-AI-Search-External|Peer39_Crawler|AdsBot-Google|AmazonAdBot|AdIdxBot|Scrapy|SemrushBot-OCOB|AwarioBot|OAI-AdsBot) [NC] RewriteRule .* - [F,L]