Block AI Crawlers in WordPress Without Editing robots.txt
Keep GPTBot, ClaudeBot, CCBot and other AI training crawlers off WordPress with an automatic robots.txt, and refuse the ones that ignore it.
Blocking AI training crawlers by hand means maintaining a robots.txt list of user-agent names that keeps growing. New crawlers appear every few months, some vendors run several, and a list copied from a blog post last year is already missing some.
This guide shows how to keep AI training crawlers off a WordPress site by category, without editing robots.txt, and what to do about crawlers that ignore it.
Decide what you are blocking
“AI crawler” covers several different things, and blocking all of them has side effects:
- AI training crawlers collect pages to train models: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance), Meta-ExternalAgent and others. Google-Extended and Applebot-Extended are robots.txt names only: they control whether Google and Apple use your content for AI, without being separate crawlers.
- AI search crawlers index pages so AI search products can cite them: OAI-SearchBot, PerplexityBot, Claude-SearchBot and similar. Blocking these keeps you out of those answers.
- AI assistants and agents fetch a page because a person asked: ChatGPT-User, Perplexity-User, Claude-User. Blocking these stops an assistant reading your page for the person who asked about it.
Most sites that want to opt out of training still want to appear in AI search. Block the training category and leave the others alone unless you have a reason.
Automatic robots.txt in WordPress
The free WebDecoy plugin builds these rules from a registry of more than 230 known crawlers, AI agents and bot tools:
- In WordPress Admin, go to Plugins > Add New, search for WebDecoy, then choose Install Now and Activate.
- Open WebDecoy > Settings and the Good Bots tab.
- Under Automatic robots.txt, tick AI training crawler. Each category shows how many crawlers it covers.
- Save, then follow View your live robots.txt to check the result.
WebDecoy adds one marked group to the end of the robots.txt WordPress already serves:
# WebDecoy: AI and bot rules (32 agents, updated with plugin version 2.13.0)
User-agent: Reflectionbot
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic
User-agent: Claude-Web
User-agent: CCBot
User-agent: Google-Extended
User-agent: Bytespider
...
User-agent: ChatGLM-Spider
Disallow: /
# End of WebDecoy rulesYour sitemap lines and other plugins’ rules stay as they are. When a plugin update adds newly named crawlers to the registry, they join the list without you touching anything. The same settings offer other categories, such as SEO tools, commercial scrapers and archivers, if you want to keep those out too.
When the rules do not appear
- A physical robots.txt file. If your site root has a real
robots.txt, the web server sends it and WordPress never builds its own. The settings page tells you and shows the rules, so you can copy them into that file or remove the file. - WordPress in a subfolder. Crawlers only read robots.txt at the root of a domain. If your site lives at
example.com/blog, copy the rules into the robots.txt atexample.com. - A cached robots.txt. Some hosts and CDNs cache robots.txt. Purge it after saving if the live file looks unchanged.
Crawlers that ignore robots.txt
robots.txt is a request. The large AI vendors say their training crawlers honour it, but nothing makes a crawler comply, and anyone can send a request that claims to be GPTBot.
Two things help:
- Watch the counts. WebDecoy’s Statistics page lists every known crawler that visited in the last 7 days. If a crawler you disallowed keeps showing up, it is not following your robots.txt.
- Refuse at the door. Under Good Bots, Block AI crawlers refuses AI training crawlers, AI agents and AI assistants when they request a page. AI search crawlers are treated like search engines and stay allowed. The plugin starts in monitor mode, which logs what it would block without blocking it. Turn monitor mode off when you are ready to enforce.
Note that this switch also refuses assistants such as ChatGPT-User, which fetch a page when a person asks about it. If you want AI assistants to read your pages for their users, rely on robots.txt for the training category and leave the switch off.
Summary
Pick the categories you want out, let the plugin keep the names current, and check your crawler counts to see who follows the rules. Install WebDecoy from WordPress.org, then open Settings > Good Bots > Automatic robots.txt.
Frequently Asked Questions
How do I block AI crawlers in WordPress without editing robots.txt? +
Install the free WebDecoy plugin, open WebDecoy Settings, Good Bots, Automatic robots.txt, tick AI training crawlers and save. WebDecoy adds a rule for every AI training crawler it knows by name to the robots.txt WordPress serves, and keeps the list current with plugin updates.
Does blocking GPTBot remove my site from ChatGPT search? +
OpenAI documents that GPTBot (training) and OAI-SearchBot (ChatGPT search) are controlled separately. Disallowing training crawlers does not by itself disallow AI search crawlers. Choose the AI search crawler category only if you also want out of AI search results.
Do AI crawlers obey robots.txt? +
The major vendors say their training crawlers do. robots.txt is still a request, and a crawler that ignores it is not stopped by it. Use request-time blocking for those, and watch your crawler counts to see whether a disallowed crawler keeps visiting.
Why doesn't my robots.txt change after I save? +
Usually because a physical robots.txt file sits in the site root. The web server sends that file and WordPress never builds its own, so nothing can be added to it. WebDecoy warns about this on the settings page and shows the rules so you can copy them into the file.
Continue your investigation
- Install the free WordPress plugin
Automatic robots.txt, AI crawler counts and AI referral counts, with no account.
- See how much traffic ChatGPT sends you
Count visits from AI assistants by product and landing page.
Share this post
Like this post? Share it with your friends!
Want to see WebDecoy in action?
Get a personalized demo from our team.