
Content Scraping Protection: Stop AI Training Scrapers
Content scraping protection for the AI era: robots.txt rules, TDM headers, crawler detection, and enforcement that stop AI scrapers from taking your site.
securityWhy Arcjet's SDK-based protection falls short and how WebDecoy's network-layer honeypot detection is more effective.
Published · Updated
When evaluating bot protection solutions, many teams initially choose application-layer tools like Arcjet because they’re easy to integrate. But there’s a critical architectural flaw that limits their effectiveness against sophisticated scrapers and attackers. Let’s explore why.
Arcjet operates through an SDK that integrates directly into your application code. While this approach seems convenient, it has a critical limitation: the bot request has already been received and is consuming your server resources by the time Arcjet can make a decision.
When a bot makes a request to your application:
aj.protect() is invokedBy the time Arcjet returns its decision, the damage is already done.
By the time your application calls aj.protect(), the bot has:
Let’s say you have an e-commerce site with product catalog pages. A scraper makes 10,000 requests per day:
With Arcjet:
The scraper’s perspective:
Arcjet adds a “speed bump,” but doesn’t actually prevent the scraping. It just makes your servers work harder while processing requests you ultimately reject.
Arcjet relies on fingerprinting and pattern analysis to identify bots. But bots can:
More fundamentally, Arcjet can only block after your server has already done the work.
Traditional WAFs and CDN-based protections (Cloudflare, Akamai) address this by intercepting traffic before it reaches your origin:
But most organizations don’t have enterprise WAF budgets. This is where WebDecoy’s approach becomes powerful.
WebDecoy takes a fundamentally different approach that combines the best of application-layer visibility with network-level efficiency:
WebDecoy gives you invisible decoy links to place in your HTML (the WordPress plugin and the Express and Fastify adapters can add a hidden honeytoken link for you) that:
When a bot crawls your honeypot link, you know exactly which visitor is a scraper, before they access your real content.
Advantage over Arcjet: You identify scrapers on first contact, before they’ve consumed significant resources.
Alongside your real API endpoints, WebDecoy serves fake ones:
Real endpoint: /api/v1/users
Decoy endpoint: /api/v1/admin-loginWhen bots probe for vulnerabilities:
Advantage over Arcjet: Bots that probe for hidden endpoints reveal themselves on a decoy WebDecoy hosts, instead of on your servers.
WebDecoy’s biggest advantage: it’s not just telling your application to deny a request. It feeds detections into your WAF and security infrastructure:
Once a block is in your WAF, that bot’s future requests stop at your edge instead of reaching your servers. (Automatic blocking from detections is currently held off in production until scores are calibrated; you can block manually today.)
Advantage over Arcjet: Your entire security stack can respond in real-time, not just your application.
For high-traffic sites, a flat-rate model can be much more cost-effective.
Arcjet acknowledges this limitation in their documentation, stating: “It’s impossible to create a system that can block all bots” and recommends combining bot protection with rate limiting to minimize impact.
But this is exactly the problem. You’re left implementing multiple layers of protection, each adding complexity and cost.
| Scenario | Arcjet | WebDecoy |
|---|---|---|
| Bot makes first request | Processed by server | Detected by honeypot |
| Bot scrapes 10 pages | 10 server requests processed | Identified after 1st honeypot hit |
| Sophisticated scraper | Bypasses with IP rotation | Rotating non-browser clients grouped into one actor that can be blocked at your WAF |
| Cost on high traffic | Grows with bot activity | Flat monthly rate |
| Response time | 403 after processing | Blocked at your WAF once a block is in place |
Arcjet is appropriate if:
WebDecoy excels when:
Arcjet is a speed bump. WebDecoy is a barrier.
Arcjet makes scraping slightly harder and more detectable, but it doesn’t actually prevent determined bots from accessing your content. The request has already reached your servers, consumed your resources, and potentially retrieved data.
WebDecoy works to keep bots away from your real content by:
For organizations serious about content protection and infrastructure efficiency, honeypot-based detection with WAF enforcement moves the fight to your edge instead of your application.
Ready to protect your content from sophisticated bots?

Content scraping protection for the AI era: robots.txt rules, TDM headers, crawler detection, and enforcement that stop AI scrapers from taking your site.
security
Compare WebDecoy's honeypot detection with CleanTalk's blacklist approach. Learn why behavioral analysis beats reputation lists for spam bot protection.
security
Compare honeypot detection vs CAPTCHA for bot prevention. Learn effectiveness rates, implementation, user experience, and when to use each.
securityLike this post? Share it with your friends!
Get a personalized demo from our team.