Netlify Bot Protection: From Monitoring to Enforcement

WebDecoy’s Netlify extension now supports clearance enforcement. You can see automated traffic and require verification on the paths you choose, using the same extension that already reports crawler activity.

This completes our Netlify protection milestone. The release adds the clearance validator, crawler verification, and request outcome reporting, and fixes the extension’s public installation page. Netlify joins Cloudflare Workers and AWS Lambda@Edge as a deployment option for WebDecoy’s clearance checks.

Existing monitoring installs keep their current behavior. Protection is an explicit opt-in, and you can review what it would do before refusing requests.

What changes for your site

A browser tag cannot report a request from a client that never runs it. The Netlify extension observes those requests at the edge: AI crawlers, command-line tools, and other automated clients. It is injected during your site’s build, without editing your application source.

With enforcement enabled, that function also checks access to your protected paths. A browser can earn a signed wd_clearance cookie. A request that satisfies your policy passes; a request that does not is asked to verify, or receives a 403 if it is a non-HTML client. Verified crawlers and service credentials are handled according to your configured access exceptions.

For example, you can start by watching /account/* or an expensive /search/* route while leaving your public articles available for indexing. After reviewing the results, turn on enforcement for the paths you intend to protect. Choosing a path matters: enabling the validator alone does not make every URL protected.

1. Install the Netlify extension

Install the WebDecoy Crawler Sensor on Netlify on your team. The extension is public and unlisted, so use this direct link rather than searching Netlify’s extension directory.

In WebDecoy, select your site, open Setup, and choose Netlify. Copy the three values shown there into your Netlify site’s Site configuration → Environment variables, with the Functions scope:

Keep the sensor key in Netlify’s environment settings. Do not put it in browser code or commit it to your repository.

Trigger a new deploy. Adding the extension to a team does not activate it on every site; the site’s variables and its next build determine where it runs.

For the complete instructions, see the Netlify installation guide.

2. Verify that monitoring works

After the deploy finishes, send a test request to your own site:

curl -A 'WebDecoy-Test/1.0' https://YOUR-SITE.netlify.app/

Replace the hostname with the deployed site you connected. WebDecoy labels this as a test, not a bot finding. Return to Setup and confirm that the sensor has reported. A successful build proves that the function was deployed; the report proves that traffic reaches WebDecoy.

You can stop here if you only want visibility. With just the three sensor variables, the extension reports traffic without blocking or challenging it.

3. Add clearance enforcement

Add one more Netlify environment variable, also with the Functions scope:

WEBDECOY_ENFORCEMENT=on

Redeploy the site. The extension now includes the clearance validator. This variable enables the validator; WebDecoy’s policy determines whether it monitors or enforces.

In WebDecoy:

1. Open Enforcement → Setup. Complete browser verification using the setup instructions, and verify the deployed validator for your site’s hostname. Keep the browser detection script if you use it for browser behavior and fingerprinting; the edge sensor covers a different set of traffic.
2. Open Enforcement → Policy and add the protected paths you actually want to gate. Review the verification requirement and access exceptions for each use case. Leave public pages unscoped when they should stay openly crawlable.
3. Start in Monitor. Review request activity and the outcomes that would have required verification. Check legitimate browsers, integrations, and crawlers that your site depends on.
4. Address the readiness checks in Enforcement → Overview, then switch to Enforce when the policy is ready.

If you use decoy hits to revoke browser clearance, the decoy must be on the same origin as your pages. A decoy on the shared WebDecoy domain records detections but does not receive your site’s clearance cookie. See the clearance enforcement guide for browser setup, path policies, service credentials, and the revocation loop.

Know which crawlers can pass

The Netlify validator verifies supported search engines through forward-confirmed reverse DNS. That means checking the requesting IP’s hostname and confirming that the hostname resolves back to that IP. It also accepts valid Web Bot Auth signatures under your policy.

Netlify does not supply Cloudflare’s verified-bot signal. A crawler category you allow may therefore be allowed only if signed when the validator cannot verify it through DNS. An unsigned client in that category still has to satisfy the protected path’s verification requirement. A familiar user-agent string alone is not proof.

Review those exceptions before protecting pages needed by search engines, link previews, monitoring services, or AI crawlers. The dashboard exposes the deployed validator’s capabilities so you can see which settings it can apply.

A Next.js detail that affects coverage

On Netlify, Next.js middleware runs before the injected WebDecoy function. If middleware returns a response itself, that request never reaches the validator. If it rewrites a path, WebDecoy evaluates the rewritten path. Site edge functions, Netlify redirects, and the origin run after the injected function.

Test your actual routes, especially middleware redirects, early responses, and rewrites. The Netlify guide documents this boundary.

See results, and undo the rollout

The validator reports request outcomes with the hostname and deployed build identifier. That gives you evidence of what ran on this site, rather than treating a configured policy as proof that requests were refused. Monitor outcomes remain distinct from enforcement outcomes.

The function is designed to fail open on operational failures such as unavailable configuration or verification infrastructure. That keeps a reporting or infrastructure failure from turning into an automatic site-wide refusal, but also makes deployment verification and ongoing request activity worth checking.

To disable the validator while keeping monitoring, remove WEBDECOY_ENFORCEMENT and redeploy. To remove the sensor as well, remove the three sensor variables and redeploy. Environment changes take effect through a new build.

Netlify usage charges and limits still apply. This release uses the injected edge function; WebDecoy does not manage Netlify Firewall Traffic Rules, and no Netlify firewall connection is implied by installing the extension.

Choose the setup for your platform

Start with the Netlify platform page for this integration, or compare Cloudflare, Vercel, AWS CloudFront and Lambda@Edge, WordPress, and Shopify. Their installation and enforcement options differ; the integration directory explains the available paths.

Ready to set it up? Open WebDecoy, install the Netlify extension, and follow the Netlify documentation.