robots.txt asks AI crawlers to stay away. A server rule makes them. This guide gives you working configs to block AI crawlers in nginx, Apache and Cloudflare, shows how to test each one, and covers the scrapers that hide their name.

Which AI crawlers to block

Most sites block AI training crawlers and keep AI search crawlers, so ChatGPT search, Perplexity and Claude can still cite and link to them. The configs below use this training list:

User-Agent tokenOperator
GPTBotOpenAI
ClaudeBotAnthropic
CCBotCommon Crawl
BytespiderByteDance
Meta-ExternalAgentMeta
AmazonbotAmazon
cohere-aiCohere
AI2BotAllen Institute for AI

To leave AI search as well, add OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User.

Two names you will see in robots.txt guides do not belong here: Google-Extended and Applebot-Extended. They are robots.txt tokens that control AI training use, not crawlers, and they never appear in a User-Agent header. Disallow them in robots.txt.

None of these tokens match Googlebot or Bingbot, so search engines are unaffected.

nginx

Add a map in the http block, usually in /etc/nginx/nginx.conf or a file in /etc/nginx/conf.d/:

map $http_user_agent $block_ai_crawler {
    default 0;
    "~*(gptbot|claudebot|ccbot|bytespider|meta-externalagent|amazonbot|cohere-ai|ai2bot)" 1;
}

~* makes it a case-insensitive regular expression. Then, in each server block you want to protect:

server {
    # ...your existing config...

    if ($block_ai_crawler) {
        return 403;
    }
}

Test and reload:

sudo nginx -t && sudo systemctl reload nginx

The map is evaluated only when the variable is used, so it costs almost nothing per request.

Apache (.htaccess)

Add this near the top of the .htaccess file in your site root. It needs mod_rewrite, which most hosts enable:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|Meta-ExternalAgent|Amazonbot|cohere-ai|AI2Bot) [NC]
RewriteRule .* - [F,L]
</IfModule>

[NC] ignores case and [F] answers 403 Forbidden. On WordPress, put it above the # BEGIN WordPress block so WordPress never sees these requests.

If you manage the server config rather than .htaccess, Apache 2.4 can do the same without rewrites:

SetEnvIfNoCase User-Agent "(GPTBot|ClaudeBot|CCBot|Bytespider|Meta-ExternalAgent|Amazonbot|cohere-ai|AI2Bot)" block_ai_crawler

<Location "/">
    <RequireAll>
        Require all granted
        Require not env block_ai_crawler
    </RequireAll>
</Location>

Cloudflare

If your site runs through Cloudflare, block AI crawlers at the edge so they never reach your server.

Option A: a custom WAF rule

  1. In the Cloudflare dashboard, open your domain, then Security > WAF > Custom rules, and create a rule.

  2. Choose Edit expression and paste:

    (lower(http.user_agent) contains "gptbot")
    or (lower(http.user_agent) contains "claudebot")
    or (lower(http.user_agent) contains "ccbot")
    or (lower(http.user_agent) contains "bytespider")
    or (lower(http.user_agent) contains "meta-externalagent")
    or (lower(http.user_agent) contains "amazonbot")
    or (lower(http.user_agent) contains "cohere-ai")
    or (lower(http.user_agent) contains "ai2bot")
  3. Set the action to Block and deploy.

This works on the free plan. Use contains as shown: the regex operator matches needs a higher plan.

Option B: Cloudflare’s AI bot setting

Cloudflare also has its own one-click setting to block AI bots, and AI Crawl Control for managing them. Both use Cloudflare’s list of AI crawlers. Use the custom rule when you want the exact same list on Cloudflare and on your own servers. Our comparison with AI Crawl Control covers the differences.

Test your block

Send a request with an AI crawler’s User-Agent and check for 403:

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://yoursite.com/

Then confirm normal visitors and search engines still get 200:

curl -s -o /dev/null -w "%{http_code}\n" https://yoursite.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://yoursite.com/

Over the next days, check your access logs for 403s to AI crawlers:

grep -iE "gptbot|claudebot|ccbot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c

Scrapers that hide their name

Every rule above matches the name a crawler sends. The large AI companies send their real names, so these rules stop them. Scrapers that copy a Chrome User-Agent pass straight through, along with any crawler you haven’t listed yet.

That is the gap WebDecoy closes:

  • Up-to-date crawler list. WebDecoy recognises over 200 crawlers and AI agents by name, and the list updates without you editing configs.
  • Detection beyond the name. Hidden trap links and decoy pages that only bots follow, plus request fingerprints, catch scrapers that pretend to be a browser.
  • Fake crawlers caught. Requests claiming to be Googlebot or Bingbot are checked against the operator’s published addresses.
  • Runs where you do. A one-click Cloudflare edge sensor, a WordPress plugin and Next.js middleware.

Quick checklist

  • Disallow AI crawlers in robots.txt, including Google-Extended and Applebot-Extended.
  • Add the server rule for nginx, Apache or Cloudflare.
  • Decide on AI search crawlers: keep them to stay in AI answers, or add them to the list.
  • Test with curl: 403 for GPTBot, 200 for browsers and Googlebot.
  • Add detection for scrapers that hide their name.

Frequently Asked Questions

How do I block AI crawlers in nginx? +

Add a map block in the http section that sets a variable when the User-Agent contains an AI crawler name such as GPTBot, ClaudeBot or CCBot, then add if ($block_ai_crawler) { return 403; } to your server block and reload nginx.

How do I block GPTBot with .htaccess? +

Add a mod_rewrite rule: RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot) [NC] followed by RewriteRule .* - [F,L]. Apache then answers those crawlers with 403 Forbidden.

Can Cloudflare block AI crawlers for free? +

Yes. Cloudflare's free plan includes custom WAF rules. Create a rule that matches the User-Agent against AI crawler names and set the action to Block. Use contains rather than regex matching, which needs a higher plan.

Should I block Google-Extended in nginx or Apache? +

No. Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers. They never appear in a User-Agent header, so a server rule cannot match them. Disallow them in robots.txt instead.

Does blocking by User-Agent stop all AI scrapers? +

It stops crawlers that send their real name, which includes the large AI companies' crawlers. Scrapers that pretend to be a browser need detection based on behaviour and traps, which is what WebDecoy adds.

Continue your investigation

  1. The robots.txt rules to block AI crawlers

    Ask AI crawlers to stay out before you refuse them.

  2. One-click Cloudflare edge sensor

    See every crawler at the edge, including ones that hide their name.

  3. Cloudflare AI Crawl Control vs WebDecoy

    What each one sees and blocks.

Want to see WebDecoy in action?

Get a personalized demo from our team.

Request Demo