Cloudflare AI Crawl Control vs WebDecoy
Cloudflare AI Crawl Control vs the free WebDecoy WordPress plugin: what each shows about AI crawlers and AI referrals, where each blocks, and why to use both.
If your site already runs through Cloudflare, you have AI Crawl Control (it used to be called AI Audit) on every plan. So the honest question is not “which one”, it is “what does each one show, and is there a gap worth filling?” This page answers that using Cloudflare’s own documentation and the WebDecoy plugin’s source.
Short version: Cloudflare is the better place to block AI crawlers, because it stops them at the edge. WebDecoy is a free WordPress plugin that shows what reaches WordPress: which crawlers came over the last week or month, whether the search engine crawlers are real, and which AI assistants send you visitors. If you use Cloudflare, use both.
What each one is
Cloudflare AI Crawl Control
AI Crawl Control is part of Cloudflare’s dashboard for any domain proxied through Cloudflare. According to Cloudflare’s docs it:
- Shows AI crawler activity: a crawler table with operator, requests and data transfer, request trends, status codes and the most requested paths, with CSV export and a GraphQL API.
- Blocks per crawler: pick Block on a crawler and Cloudflare creates a WAF custom rule on your zone, so the request never reaches your server.
- Tracks robots.txt violations: a table of AI crawlers that requested paths your robots.txt disallows.
- Can manage robots.txt for you: managed robots.txt (available on all plans) prepends Disallow rules for AI training crawlers and Cloudflare’s Content Signals Policy to your existing file.
- Pay per crawl: charging crawlers for access, documented as a private beta.
What depends on your plan, per Cloudflare’s docs:
- On the Free plan, AI crawlers are identified by their user agent strings. Detection using Bot Management detection IDs is for Enterprise plans with Bot Management.
- On free plans the Metrics tab shows the past 24 hours.
- Top referrers and referral totals (for example visits from chatgpt.com or perplexity.ai) are for paid plans.
- Custom block responses (403 or 402 with your own message) are for paid plans.
WebDecoy WordPress plugin
WebDecoy is a free WordPress plugin that runs on your own server with no account or API key. For AI traffic it:
- Counts known crawlers and AI agents on an AI Traffic & Analytics page, with a 7- or 30-day view, each bot’s operator and type (AI training crawler, AI search crawler, AI agent, search engine, SEO tool and more), named from a registry of more than 230 crawlers, AI agents and bot tools.
- Checks the big crawlers are real: Googlebot, Bingbot, Applebot, DuckDuckBot, Yandex, Baidu and the major social crawlers are checked with reverse and forward DNS. A visitor using one of those names from someone else’s address is flagged as an impersonator. Other bots, including AI crawlers such as GPTBot, are labelled as identified by name only, so you know how far to trust them.
- Counts visits from AI assistants: ChatGPT, Perplexity, Claude, Gemini, Copilot, Meta AI, DeepSeek, Grok and others, by AI product and landing page. Only a daily count is kept, nothing about the visitor.
- Writes robots.txt rules by category: choose categories such as AI training crawlers and it adds a rule for every known crawler in them, alongside what WordPress and other plugins already write.
- Refuses AI crawlers that ignore robots.txt, with the Block AI crawlers setting. Search engines and AI search crawlers such as PerplexityBot and OAI-SearchBot stay allowed.
Side by side
| Cloudflare AI Crawl Control | WebDecoy WordPress plugin | |
|---|---|---|
| Where it runs | Cloudflare’s edge, in front of your server | Inside WordPress, on your server |
| Needs | A domain proxied through Cloudflare | WordPress 6.1+ and PHP 7.4+ |
| Account | Cloudflare account | None |
| Price | Available on all Cloudflare plans; some features paid | Free |
| How crawlers are identified (free) | User agent strings | User agent, plus reverse and forward DNS for search and social crawlers |
| History you can see (free) | Metrics tab: past 24 hours | 7 or 30 days |
| Visits sent by AI assistants | Paid plans | Free, by AI product and landing page |
| Blocking | At the edge, through a WAF rule | Inside WordPress, for requests that reach it |
| robots.txt | Managed robots.txt plus a violations table | Rules by category, kept current with plugin updates |
| Cached pages | Seen, Cloudflare serves them | Not seen unless crawlers skip the page cache (optional) |
| Pay per crawl | Private beta | No |
Where Cloudflare is better
- Blocking before your server. A crawler blocked by AI Crawl Control is stopped by a WAF rule at Cloudflare’s edge. It costs your server nothing. The WebDecoy plugin can only refuse a request after it reaches WordPress.
- Cached traffic. Cloudflare sits in front of your site, so it sees requests it answers from its own cache. A WordPress plugin never sees a page served from a cache before WordPress loads.
- robots.txt compliance reporting. The violations table tells you which AI crawlers fetched paths your robots.txt disallows.
- Pay per crawl, if you want to charge crawlers rather than block them, is a Cloudflare feature (in private beta) with no WebDecoy equivalent.
What WebDecoy adds inside WordPress
- A longer free history. On Cloudflare’s free plans the Metrics tab covers the past 24 hours. The plugin keeps 7- and 30-day views on your own server.
- AI referrals at no cost. Seeing which AI assistants send people to which pages is a paid-plan feature in AI Crawl Control. The plugin counts it for free.
- A second opinion on identity. Cloudflare’s free tier identifies crawlers by user agent. The plugin’s DNS check catches a visitor claiming to be Googlebot or Bingbot from an address that is not Google’s or Microsoft’s, and labels it an impersonator.
- No Cloudflare required. Many WordPress sites are not on Cloudflare. The plugin needs no DNS change, proxy or account.
One honest limit: if you use a page cache, crawler visits to cached pages never reach WordPress. Since version 2.15.0 the plugin tells you when that is happening, and an optional setting lets known crawlers skip WP Super Cache, W3 Total Cache, LiteSpeed Cache or WP Rocket so they are counted (people still get cached pages). It is off by default, because those crawler requests then use your server.
Using both
A sensible setup for a WordPress site on Cloudflare:
- Block in Cloudflare. Use AI Crawl Control to block the AI training crawlers you do not want. That stops them at the edge.
- Watch in WordPress. Install the WebDecoy plugin to see the crawlers you allowed, check that search crawlers are genuine, and count the visitors AI assistants send you.
- Keep robots.txt in one place. If Cloudflare’s managed robots.txt is on, you probably do not need the plugin’s automatic robots.txt as well; it is off until you choose categories.
If you would rather see crawler traffic at the edge inside WebDecoy, there is also the Cloudflare edge sensor: a Worker WebDecoy deploys to your Cloudflare zone that reports crawlers that never run JavaScript, including requests WordPress never sees. It needs a WebDecoy account, and enforcement at the edge is on paid plans.
Which to choose
- Your site is on Cloudflare and you only want AI crawlers gone: AI Crawl Control is enough.
- You want to know what the crawlers you allow are doing, which ones are fake, and what AI assistants send you: add WebDecoy.
- Your WordPress site is not on Cloudflare: WebDecoy gives you crawler counts, AI referrals, robots.txt rules and blocking with nothing to change at your host.
Related reading: Understand your AI traffic in WordPress, without an account and Block AI crawlers in WordPress without editing robots.txt.
Get started
On WordPress: Install WebDecoy free from WordPress.org. It runs alongside Cloudflare with no account and no configuration.
Not on WordPress? The Cloudflare edge sensor reports crawler traffic from your Cloudflare zone. Create a free WebDecoy account to deploy it.
Frequently Asked Questions
Is WebDecoy a replacement for Cloudflare AI Crawl Control?
Not if your site already runs through Cloudflare. AI Crawl Control blocks AI crawlers at Cloudflare's edge, before a request reaches your server, which the WebDecoy WordPress plugin cannot do. WebDecoy adds what happens inside WordPress: a 7- or 30-day history of AI crawler visits, DNS checks on search and social crawlers, and counts of visitors sent by AI assistants, free and with no account. Most Cloudflare users are best served by running both.
Is Cloudflare AI Crawl Control free?
Cloudflare documents AI Crawl Control as available on all plans. On the Free plan it identifies AI crawlers by their user agent strings and the Metrics tab shows the past 24 hours. Referral data and custom block responses are documented as paid-plan features, and detection using Bot Management detection IDs is for Enterprise plans with Bot Management.
Can I see which AI assistants send visitors to my WordPress site without paying?
Yes, with the WebDecoy plugin. It counts visits that arrive from links in ChatGPT, Perplexity, Claude, Gemini, Copilot and other AI products, by AI product and landing page, with no account. Cloudflare documents top referrers in AI Crawl Control as a paid-plan feature.
Which one sees cached pages?
Cloudflare does, because it sits in front of your server and answers cached requests itself. The WebDecoy plugin only sees requests that reach WordPress, so a page cache hides some crawler visits. Since version 2.15.0 the plugin tells you when that is happening, and an optional setting lets known crawlers skip WP Super Cache, W3 Total Cache, LiteSpeed Cache or WP Rocket so their visits are counted.
Do I need a Cloudflare account to use WebDecoy?
No. The WordPress plugin runs on your own server with no Cloudflare account, no DNS change and no WebDecoy account. WebDecoy's separate Cloudflare edge sensor does need a Cloudflare-proxied site and a WebDecoy account.
Need help choosing a bot protection solution?
Our team can help you compare options and find the right fit for your needs.