Shipped: Agent Identity & Human Trust Layer

For as long as bot detection has existed, “allow AI crawlers” has meant one thing: match a substring in the User-Agent header and wave the request through. Every product did it, including ours. It is not an allowlist. It is a password printed on the door.

Over the last stretch we replaced that with cryptography. WebDecoy now verifies agent identity with Web Bot Auth, turns failed verification into a new high-confidence detection class, and grades how human a session actually is instead of answering yes or no. This post is what shipped, what it does, and — because it matters more than the feature list — what is still missing.

The hole we were closing

A user agent is a claim, and claims are free. Copying the GPTBot or Googlebot string takes one line of code, and it works on any site that treats that string as permission. Impersonating a verified crawler is a staple of scraper evasion precisely because the payoff is so reliable.

Our own allow_ai_crawlers toggle was exactly this hole. A customer turning it on to be a good citizen for legitimate AI crawlers was, in practice, publishing a bypass.

The fix arrived from the ecosystem. Google began signing its crawler traffic in June 2026; OpenAI signs today. The IETF chartered a webbotauth working group with Cloudflare, Amazon, Akamai and OpenAI at the table. A signature collapses the whole problem into one question: does the math check out?

Phase 1: we wrote the Go implementation, and open-sourced it

Web Bot Auth is a profile of RFC 9421 HTTP Message Signatures. An operator publishes an Ed25519 or RSA-PSS public key set at /.well-known/http-message-signatures-directory on a domain it controls; its bots sign each request with a tag="web-bot-auth" signature and a Signature-Agent header pointing back at that directory.

Cloudflare ships reference implementations in TypeScript and Rust. There was no canonical Go one — and Go is the language the proxies, gateways and edge infrastructure of the internet are written in. So we wrote it and published it:

github.com/WebDecoy/web-bot-auth — Apache 2.0, zero third-party dependencies.

That last part was a deliberate call. The available Go RFC 9421 libraries were tiny single-maintainer projects, which is an unacceptable supply-chain and correctness surface underneath a security library. The crypto is all standard library anyway (crypto/ed25519, crypto/rsa, crypto/sha256), so what remained was a hand-written RFC 8941 structured-fields parser, signature-base construction, JWK thumbprints, and an SSRF-guarded directory client. We scoped it tightly to the Web Bot Auth profile and documented the rest of RFC 9421 as an explicit non-goal, so nobody “completes” it later.

Correctness got the weight that decision demanded:

• Every vector from Cloudflare’s reference implementation passes — both draft revisions, both algorithms, including structured-field member extraction.
• Live interop in both directions against Cloudflare Research’s production deployment: our signer’s requests verify on their endpoint, and our verifier resolves keys from their real directory over the network. CI re-runs it weekly, so drift surfaces within seven days.
• Roughly 10.8 million fuzz executions across the parsers, zero crashes, fuzzing wired into CI.

The interop requirement immediately earned its keep. Cloudflare’s live directory publishes the nbf key field in milliseconds, where RFC 7517 specifies seconds. A strict reading rejects that key as “not yet valid” somewhere in the year 57217. Vectors alone would never have caught it; a deployed system did.

Verification, three tiers deep

Signatures are the ceiling, not the whole story — most crawlers don’t sign yet. WebDecoy assigns every agent the strongest method it actually earned:

For the middle tier we load the range lists Google, Bing, OpenAI, Perplexity, Apple and DuckDuckGo publish — refreshed every 12 hours, with a hard rule that data older than seven days degrades every verdict to unknown. Stale ranges must never false-verify, and must never condemn a crawler that added IPs since our last fetch.

The reverse-DNS path also fixed a real vulnerability in our own code. The old check used a substring match and never forward-confirmed, which means an attacker controlling their IP’s reverse zone could publish fake.googlebot.com.evil.net and be identified as Googlebot. It now matches on dot boundaries only and requires the PTR hostname to resolve back to the source IP — the procedure Google and Bing document, and the only version that means anything.

The new detection class: agent impersonation

Here is the part that only becomes possible once verification exists.

When a request claims an identity that can be checked and the check fails, that request is not merely suspicious. It is caught in a lie. WebDecoy scores it as agent_impersonation, floors the actor’s unified threat score at 85, and marks the actor spoofed — the same tripwire grade we give a honeypot hit, because it carries the same kind of certainty. Nobody accidentally signs a request wrong while wearing someone else’s name.

Three evidence paths produce it:

• A signing agent’s user agent arriving with a missing or invalid signature
• Googlebot from a residential IP, outside every published range
• A crawler user agent whose forward-confirmed reverse DNS points somewhere else entirely

And one deliberate restraint: a legitimate crawler that simply doesn’t sign yet degrades to claimed. Absence of proof only becomes evidence of a lie once we have a source of truth to contradict it.

Enforcement follows the same logic. allow_ai_crawlers now honors verified agents only — a user-agent string grants nothing. We put signature verification directly into the edge validator with WebCrypto rather than leaning on Cloudflare’s verified-bot categories, so this works on any zone rather than only Cloudflare’s. The fail-open invariant is unchanged: any config or crypto error forwards the request untouched, and the rollout is monitor-first, annotating would-be denials before enforcing them.

The visible surface is a Verified Agents dashboard: every agent seen, its verification badge, per-agent allow and deny overrides, and impersonation attempts surfaced as first-class detections. Signature-verified reads strong; claimed-only reads visibly weaker, which is the entire point.

Phase 3: grading the humans

Verifying agents solves half the problem. The other half is the browser you can’t dismiss as a bot and can’t quite trust either — a real Chrome, driven by a script, from a residential IP.

Our session clearance token used to answer exactly one question: has this session tripped a decoy? It now carries a trust level and the evidence behind it, and a route scope can require a minimum:

• clean — hasn’t tripped a decoy. The old binary token, still the default.
• human-likely — browser-integrity checks passed at the interstitial, or interaction cadence read as a real hand on a real device.
• attested-human — a third-party attestation the visitor already carries.

Two evidence producers ship for human-likely. The interstitial integrity challenge finally owns a step we used to delegate to the customer’s WAF: it checks permission/Notification coherence, navigator.webdriver, window.chrome under a Chrome UA, and canvas/WebGL liveness — invisibly, in a few milliseconds, with no user interaction. Crucially the server decides, gating only on low-false-positive tells, so a stealth headless that spoofs navigator.webdriver still can’t reach human-likely on its residual tells.

The behavioral human-likelihood score is the second. The browser reports privacy-preserving interaction aggregates — pointer and scroll cadence, timing — and ingest scores them 0–100 on the actor, alongside the threat score and never folded into it. “How bot-like is this actor” and “how human-like is this actor” are different questions, and an actor scoring high on both — real hands driving an automation tool — is exactly the case a single blended number would hide.

The test vector we’re proudest of: Playwright’s page.mouse.move(x, y, {steps: n}) is exact linear interpolation on a uniform beat, which is wrong in four independent ways at once — no micro-tremor, perfectly collinear, isochronous, constant velocity. It scores 0 against a real session’s 100.

Three properties are non-negotiable in that design, and they are worth stating plainly:

It only grades up. Sparse, absent, or machine-like interaction yields no evidence and no refusal. Keyboard-only and assistive-technology sessions are unscored, not scored badly. Behavioral data must never become an accessibility barrier.

The server decides. The browser reports observations, never verdicts, and a high threat score caps any grade. Fabricated cadence cannot out-vote real bot evidence.

It is explainable. Every weight names the automation tell it encodes. “The model said so” is not a defensible answer to a customer whose checkout route enforces a minimum trust level — which is also why we don’t use machine learning for detection.

What doesn’t work yet

Two things in this release are built and merged but not yet delivering value, and saying so is cheaper than letting you discover it.

Private Access Tokens don’t work. We implemented the full Privacy Pass type-0x0002 blind-RSA verifier, and the flow is correct end to end — but the TokenChallenge carries an origin_info naming where the token may be redeemed, and one ingest serves every tenant. A single static origin is wrong for everyone except one customer, so Apple clients silently ignore the challenge. We can’t simply trust an origin passed in by the worker either: a malicious site could then harvest PATs from its own visitors and redeem them against us. Registering validator origins per organization is the fix, and it’s the open blocker.

So today, attested-human is reachable through Cloudflare Turnstile, which has the advantage of working on every browser rather than only recent Apple devices. If you don’t configure it, nothing is lost — sessions still reach human-likely on browser integrity and interaction cadence.

Minimum trust is enforced at the edge validator, not in the SDK. @webdecoy/node forwards the clearance cookie but doesn’t yet verify it or gate on its level. For customers whose enforcement point is the SDK, per-route minimum trust isn’t reachable yet. That work is filed and next.

We also do not sign our own outbound traffic yet. Publishing WebDecoy’s key directory is real work we haven’t finished, and we’d rather say that than imply otherwise.

Ship it in your own middleware

@webdecoy/node 0.5.0 is on npm with local Web Bot Auth verification — no API round trip, no network on the warm path, under 5ms at p95, running on both Node and the Vercel Edge runtime:

import { WebDecoy, webBotAuth } from '@webdecoy/node'

const wd = new WebDecoy({ apiKey: process.env.WEBDECOY_API_KEY })

const { agent } = await wd.detectBot(request)

// 'verified' | 'claimed' | 'impersonation' | 'none'
if (agent.verdict === 'impersonation') {
  return new Response('Forbidden', { status: 403 })
}

if (agent.verdict === 'verified') {
  return allowVerifiedCrawler(agent) // agent.name → 'ChatGPT-User'
}

Key directories are curated rather than followed from the Signature-Agent header, which is attacker-controlled — so verification can never be steered at an arbitrary host.

Where this goes

The webbotauth working group is moving fast enough that we automated tracking it: a weekly job checks all ten drafts against the IETF datatracker and stays silent unless something actually moved.

One draft deserves flagging now rather than later. draft-rescorla-anonymous-webbotauth proposes anonymous bot authentication, where an origin learns that an authenticated bot from an acceptable anchor is calling but never which one. It is explicit that anonymity precludes allow-lists, deny-lists, and per-bot auditing — which is precisely what our enforcement is built on. If that direction gains traction, “deny this actor” stops being expressible for anonymous agents, and enforcement has to be re-expressed as rate limiting per credential. That’s an architecture conversation worth having before it’s urgent.

For now, the shift that matters is simpler. Identity and permission were always separate decisions, but until recently you could only guess at the first one. Now you can check.

See it in the product: Agent Identity & Trust Layer · Use the library: github.com/WebDecoy/web-bot-auth · Background reading: Signed Agents and the Coming Web Identity Schism