A User-Agent Is a Claim. A Signature Is Proof.
WebDecoy verifies AI agents cryptographically with Web Bot Auth, turns failed verification into a tripwire-grade detection, and grades how human a session actually is — so allowing an agent stops meaning trusting a header.
GET /pricing HTTP/1.1
User-Agent: GPTBot/1.0
Signature-Agent: "https://chatgpt.com"
Signature-Input: sig1=("@authority" "signature-agent");
keyid="poqkLGiymh_W0uP6..."; tag="web-bot-auth"
// verified · signature · chatgpt.com
— versus —
GET /pricing HTTP/1.1
User-Agent: GPTBot/1.0
// agent_impersonation · no signature
// source IP outside published ranges
// verified_status: spoofedEvery allowlist was built on an honor system
Ask any bot detection product to "allow AI crawlers" and, until recently, it did the only thing it could: match a substring in the User-Agent header and wave the request through. That is not an allowlist. That is a password printed on the door.
Scrapers have known this for years. Wearing the GPTBot or Googlebot user agent is one of the cheapest evasions available, and it works precisely because so many sites reward it. Catching the fakes meant maintaining IP ranges and reverse-DNS checks per operator, and hoping the lists stayed current.
That changed when major operators started signing their traffic. Google began signing crawler requests in June 2026; OpenAI signs today. A signature collapses the whole problem into one question: does the math check out?
Three tiers of evidence, never guessed
Every agent WebDecoy sees is assigned the strongest verification method it actually earned. A user-agent match alone never earns "verified" again.
SignatureStrongest
RFC 9421 Web Bot Auth signature verified against the operator’s key directory
Cryptographic proof. Survives IP rotation, cannot be replayed at another origin.
IP + rDNSStrong
Published IP ranges, or a reverse DNS record forward-confirmed back to the source IP
Proves the request came from the operator’s infrastructure.
ClaimedNo evidence
A User-Agent string that matches a known agent, and nothing else
Proves nothing. Anyone can send this header.
The reverse-DNS check most implementations get wrong
Checking whether a PTR record contains "googlebot.com" is not verification — an attacker who controls their own reverse zone can publish fake.googlebot.com.evil.net and pass. WebDecoy matches on dot boundaries only and forward-confirms: the hostname in the PTR record must resolve back to the IP that sent the request. That is the procedure Google and Bing document, and it is the only version that means anything.
Agent impersonation
Verification creates a detection that was impossible without it. When a request claims an identity that can be checked and the check fails, that request isn't merely suspicious — it is caught in a lie.
- A signing agent's user agent arriving with an invalid signature
- Googlebot from a residential IP, outside every published range
- A crawler user agent whose reverse DNS points somewhere else entirely
Each of these floors the actor's threat score at tripwire grade — the same weight WebDecoy gives a honeypot hit, because it carries the same kind of certainty. No one accidentally signs a request wrong while wearing someone else's name.
Verified Agents · this week
Someone pretended to be GPTBot
47 times
Before verification, all 47 of those requests looked exactly like the real thing — and on a site with "allow AI crawlers" enabled, all 47 were let through.
Grade the humans too
Verifying agents solves half the problem. The other half is the browser you can't dismiss as a bot and can't quite trust either — a real Chrome, driven by a script, from a residential IP.
WebDecoy's session clearance token used to answer exactly one question: has this session tripped a decoy? It now carries a trust level and the evidence that earned it, and a protected route can require a minimum grade. Your checkout can demand more proof than your homepage.
attested-humanEarned by: A third-party attestation the visitor already carries — today, a Cloudflare Turnstile verdict.
Reserve for checkout, password reset, or anything where a scripted browser must never reach.
human-likelyEarned by: Browser-integrity checks at the interstitial, or interaction cadence that reads as a real hand on a real device.
A sensible minimum for login, signup, and the endpoints scrapers actually want.
cleanEarned by: The session hasn’t tripped a decoy. The old binary token, and still the default.
Ordinary browsing. Nothing is gated on evidence the visitor never had a chance to produce.
Evidence only grades up
Sparse interaction, keyboard-only navigation, and assistive technology are unscored, never scored badly. Behavioral signals must never become an accessibility barrier.
The server decides
The browser reports observations, never a verdict, and a high threat score caps the grade. Fabricated cadence cannot out-vote real bot evidence.
No session replay
Interaction cadence means timing and movement aggregates. No keystroke content, no recorded sessions. What we collect is published in the docs.
What ships in the trust layer
Signature Verification
RFC 9421 Web Bot Auth signatures verified against the operator’s published key directory. Cryptographic proof of identity that no user agent string can fake.
IP Range + Reverse DNS
For crawlers that don’t sign yet: published IP ranges plus forward-confirmed reverse DNS. Not a substring match — the PTR record must resolve back to the source IP.
Agent Impersonation
A request that claims a verifiable identity and fails verification becomes a tripwire-grade detection. Fake GPTBot traffic is now a named threat class, not background noise.
Verified Agents Dashboard
Every agent that touched your site, how its identity was established, and who tried to impersonate it — with per-agent allow and deny overrides.
Verified-Only Allowlists
Allowing AI crawlers no longer means trusting a header. Only agents that prove their identity pass the allowlist — on any zone, not just Cloudflare.
Graded Human Trust
Session clearance stops being a yes/no flag. Tokens carry a trust level and the evidence behind it, and routes can require a minimum grade.
Verify agents in your own middleware
@webdecoy/node verifies Web Bot Auth signatures locally — no round trip, no network on the warm path, under 5ms at p95. It runs on Node and on the Vercel Edge runtime.
- Same verdict taxonomy as the pipeline: verified, claimed, or impersonation
- Curated key directories only — a
Signature-Agentheader can't point us at an arbitrary host - Zero dependencies
import { WebDecoy, webBotAuth } from '@webdecoy/node'
const wd = new WebDecoy({ apiKey: process.env.WD_KEY })
const { agent } = await wd.detectBot(request)
// agent.verdict: 'verified' | 'claimed'
// | 'impersonation' | 'none'
if (agent.verdict === 'impersonation') {
return new Response('Forbidden', { status: 403 })
}
if (agent.verdict === 'verified') {
// agent.name → 'ChatGPT-User'
return allowCrawler(agent)
}Web Bot Auth for Go
The verification engine behind all of this is public. github.com/WebDecoy/web-bot-auth is an Apache-2.0 Go implementation of Web Bot Auth with zero third-party dependencies — stdlib crypto, a hand-written structured-fields parser, and an SSRF-guarded directory client.
Cross-validated
Passes every vector in Cloudflare's reference implementation, both draft revisions, both algorithms.
Live interop
Tested in both directions against Cloudflare Research's production deployment, re-run weekly in CI.
Fuzzed
Roughly 10.8M fuzz executions across the parsers with zero crashes; fuzzing runs in CI.
There was no canonical Go implementation of Web Bot Auth — Cloudflare ships TypeScript and Rust — and Go is the language the proxies, gateways and edge infrastructure of the internet are written in. So we published one.
Frequently Asked Questions
Common questions about agent identity verification and graded trust.
What is Web Bot Auth?
Web Bot Auth is a profile of RFC 9421 HTTP Message Signatures for automated clients, defined in IETF drafts authored at Cloudflare and Google. An operator publishes its public keys at a well-known path on a domain it controls, and its bots sign each request with a tag="web-bot-auth" signature plus a Signature-Agent header pointing at that directory. The receiving server verifies the signature against the published key and learns cryptographically — not heuristically — which operator sent the request.
Why isn’t a User-Agent header good enough?
Because a User-Agent is a claim, and claims are free. Copying the GPTBot or Googlebot user agent takes one line of code, and it works on any site that treats that string as permission. That is exactly why impersonating a verified crawler is a staple of scraper evasion. WebDecoy now treats a bare user-agent match as claimed — an unproven assertion that earns nothing.
What is agent impersonation detection?
It is a detection class that only exists once verification does. When a request claims an identity that can be checked — a signing agent’s user agent, or a crawler with published IP ranges — and that check fails, the request is not merely suspicious, it is caught in a lie. WebDecoy scores it as tripwire-grade evidence and marks the actor spoofed, the same weight it gives a honeypot hit.
Does a valid signature mean the agent is allowed?
No — identity and permission are separate decisions. A valid signature tells you exactly who is calling; it says nothing about whether you want them. The point is that your policy can finally target real identities: allow Googlebot, rate-limit a verified AI crawler, deny another outright, all without guessing from a spoofable header.
What are graded trust levels?
WebDecoy’s session clearance token used to answer one question: has this session tripped a decoy? It now carries a trust level — clean, human-likely, or attested-human — along with the evidence that earned it. A protected route can require a minimum level, so your checkout can demand stronger proof than your homepage. Evidence only ever raises a grade, and a high threat score caps it.
What happens to visitors who produce no evidence?
Nothing. An absent claim reads as clean, which is exactly what every visitor got before graded trust existed. Sessions with sparse interaction — keyboard-only navigation, assistive technology, a browser that suppresses canvas — are unscored rather than scored badly. Evidence can only lift a token, never lower it, so no one is worse off for lacking it.
Do I have to run WebDecoy on Cloudflare for this to work?
No. Cloudflare exposes verified-bot categories on its own zones, and relying on that would limit verification to Cloudflare customers. WebDecoy verifies Web Bot Auth signatures itself — in the ingest pipeline, in the edge validator, and locally inside the Node SDK — so the same verification runs wherever you deploy.
Is WebDecoy’s Web Bot Auth implementation open source?
Yes. github.com/WebDecoy/web-bot-auth is an Apache-2.0 Go implementation with zero third-party dependencies, cross-validated against Cloudflare’s reference vectors and tested against their live reference deployment in both directions. It is the same library that runs in production behind WebDecoy’s verification.
Related: Rules & Enforcement, SDKs, and Web Bot Auth: Google Now Signs Its Crawlers.
Stop trusting user-agent strings
Verify the agents you allow, catch the ones pretending, and require real proof on the routes that matter.
Talk to Us