Bot protection for Publishers & Media

Your content. Your crawl policy.

See who collects your content, decide who gets access, and measure the visits AI platforms send back.

Start Free Trial

The business impact

Publishing for readers means managing machines.

Content is collected outside your analytics

HTTP crawlers can fetch article after article without running your analytics tag. Edge observation makes that traffic visible, so decisions about AI access start with requests you can actually inspect.

A familiar name is easy to impersonate

A scraper can call itself Googlebot or GPTBot in a request header. Supported network and signature checks help distinguish an operator’s traffic from an unproven claim before it earns a place on your allowlist.

Crawling and reader value are different measures

A platform can collect many pages while sending few identifiable visits. Compare observed crawls with attributable referrals, and keep shared search-and-AI operators separate when interpreting the exchange.

Inside WebDecoy

A crawler name comes with evidence.

The real AI Crawlers & Agents dashboard separates proven identities, unproven claims, and forged identities. It gives your team a starting point for crawler policy without treating a familiar user-agent string as permission.

WebDecoy AI Crawlers and Agents dashboard separating proven, unproven, and forged identities

Actual WebDecoy interface. Product views shown with recorded traffic.

How it fits

Build an access policy from observed traffic.

  1. Make the crawl pass visible

    Deploy Edge Sensor on Cloudflare to observe requests from non-JavaScript clients. Combine it with browser detection and decoy links for automated browsers that interact with the site.

    Explore Edge Sensor

  2. Verify before you allow

    Review supported IP-range, reverse-DNS, and signature evidence. Set per-agent access decisions based on the identities you can establish, with separate treatment for unverified claims.

    Explore crawler verification

  3. Compare access with return traffic

    Use AI Traffic Exchange to compare crawled pages and known AI referrals over the same window. Share a dated report with editorial, audience, and commercial teams before revising policy.

    Explore AI Traffic Exchange

Deployment

Start where
you have control.

Start with a section of your publication and observe traffic before enforcing changes. WordPress publishers can add browser detection with the plugin. Cloudflare sites can add Edge Sensor to see HTTP crawlers that a page tag misses.

Choose a plan for your deployment. View platform pricing.

Before you start

Questions from
publishers teams.

Will blocking an AI crawler affect search visibility?

It depends on the operator and the access policy. Some operators use infrastructure for both search and AI. Review the crawler’s documented purpose and observed identity before denying it, and keep desired search access explicitly allowed.

Does the report capture every AI referral?

No. WebDecoy attributes a referral when the visit includes a recognized AI platform in its Referer header. Platforms and browsers can omit that header, so measured referrals are a floor and the crawl-to-referral ratio is an upper bound.

Does WebDecoy license content or recover copies already collected?

WebDecoy helps you observe traffic and enforce access decisions through your integration. It does not negotiate content licenses, remove existing copies from external datasets, or control what happens to material already retrieved.

Go deeper

Explore all industries

WebDecoy for Publishers & Media

Give your content an informed access policy.

Start Free Trial