Web Bot Auth: Google Now Cryptographically Signs Crawlers

In June 2026, Google started doing something bot detection has wanted from good bots for a decade: cryptographically signing its crawler traffic. A request from Google’s fetcher infrastructure can now carry a verifiable Ed25519 signature instead of just a User-Agent string that anyone can copy.

That header is the first large-scale deployment of Web Bot Auth — and it changes the shape of a problem every site operator knows: a user agent is a claim, and claims are free. Impersonating Googlebot is a staple of scraper evasion precisely because so many sites give anything wearing that string a pass. Until now, catching the fakes required maintaining IP-range lists and reverse-DNS checks per operator. A signature collapses all of that into one question: does the math check out?

We looked at where this ecosystem is heading in Signed Agents and the Coming Web Identity Schism; this post is the deployment update — what Google shipped, and what to do about it operationally.

What Web Bot Auth is

Web Bot Auth is a profile of RFC 9421 HTTP Message Signatures for automated clients, defined in IETF drafts authored at Cloudflare and Google — which makes Google’s deployment less a surprise than a co-author shipping its own spec. The design has three parts:

1. Key discovery. A bot operator publishes its public keys (Ed25519 or RSA-PSS) as a JSON Web Key Set at a well-known path on a domain it controls:

https://<operator-domain>/.well-known/http-message-signatures-directory

2. Request signing. The bot signs each request over a defined set of components — including the target authority — and attaches the signature with a tag="web-bot-auth" parameter, plus a Signature-Agent header that tells the receiver where to find the keys:

GET /pricing HTTP/1.1
Host: example.com
Signature-Agent: "https://agent.example-crawler.com"
Signature-Input: sig1=("@authority" "signature-agent");
  created=1750000000;expires=1750000300;
  keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";
  tag="web-bot-auth"
Signature: sig1=:jdq0SqOwHdyHr9+r5jw3iYZH6aNGKijYp/EstF4RQTQdi5N5YYKrD+mCT1HA1nZDsi6nJKuHxUi/5Syp3rLWBA==:

3. Verification. The server fetches the operator’s key directory (and caches it), matches the signature’s keyid against the RFC 7638 thumbprints of the published keys, and verifies the signature. If it validates and the timestamp window is current, the request provably came from that operator — no matter what IP it arrived from or what its user agent says.

What Google actually shipped

As of June 2026, Google’s fetcher traffic includes Signature-Agent headers pointing to Google’s key directory at agent.bot.goog, with requests signed per the Web Bot Auth drafts. Two caveats matter:

• It’s explicitly experimental. Google itself recommends keeping IP and reverse-DNS verification as a fallback.
• Coverage is partial. Not all Google crawl traffic is signed yet, and almost no other major operator signs at all. An unsigned request is not evidence of anything — yet.

That second point is the single most important operational rule of the current moment: verify-and-allow, never punish absence. A valid signature is a strong positive signal. A missing signature is, for now, just Tuesday.

Identity is not permission

It’s tempting to read Web Bot Auth as “the good-bot problem is solved.” It’s better understood as the good-bot problem becoming decidable. A signature tells you exactly who a bot is. Whether you want it on your site is still your call — and that call is getting more interesting, not less, as AI agents proliferate.

Once identities are verifiable, per-bot policy becomes real: allowlist Googlebot everywhere, let a verified AI search crawler read your blog but not your pricing, rate-limit a verified scraper of unknown intent, block a verified agent whose operator ignored your robots.txt. None of that is possible when identity is a spoofable string.

And the inverse still holds: the bots that cause damage — scrapers, credential stuffers, inventory bots — will never sign their requests honestly. Web Bot Auth cleanly splits the population into provably legitimate and everything else, and “everything else” is exactly the population behavioral detection, JA4 fingerprinting, and honeypot decoys exist to sort out.

Where it fits in a detection stack

WebDecoy models actor identity in three tiers, each trading coverage against strength:

• Network tier — JA4/TLS fingerprints and header ordering. Works on every request, even from bots that never run JavaScript, but coarse: many clients share a fingerprint.
• Device tier — canvas, WebGL, audio, and hardware signals from the browser. Much higher uniqueness; needs JavaScript.
• Crypto tier — verifiable, cryptographically bound identity. The strongest signal there is, but only present when the client cooperates.

Web Bot Auth is the crypto tier growing real-world coverage for the first time. A valid signature is a better identity key than any fingerprint — it can’t collide, can’t be spoofed without the private key, and survives any IP change by construction. When it’s present, it should anchor the actor’s identity. When it’s absent, the other tiers carry on as before.

How WebDecoy uses it

Two places, one in detection and one in enforcement:

Detection: the strongest actor key. WebDecoy’s composite actor model keys each actor by the strongest identity tier present. A verified Web Bot Auth signature outranks every fingerprint, so signed operators resolve to clean, stable actors — and their traffic stops consuming the anomaly budget that should be spent on adversaries.

Enforcement: the verified-bot allowlist. WebDecoy’s closed-loop enforcement evaluates a verified-bot allowlist before any rule, challenge, or session clearance check applies. Cryptographic verification is the strongest possible membership test for that allowlist: a crawler that proves it’s Googlebot is exempted by proof, not by a user-agent pattern or a hand-maintained IP list. Legitimate automation never gets swept up in enforcement aimed at abusive automation — which is precisely the failure mode that makes site owners afraid to turn enforcement on.

There’s a third use coming, but it isn’t here yet: once an operator signs all of its traffic, “claims to be that operator but carries no valid signature” flips from noise into a high-confidence impersonation signal. That switch should stay off until adoption matures — throwing challenges at unsigned Googlebot traffic in 2026 would just hurt your crawl budget.

What to do today

1. Don’t block or penalize unsigned traffic. Adoption is one major operator deep. Absence of a signature means nothing yet.
2. Start logging Signature-Agent headers. Even before you verify, knowing which of your bot traffic claims a signable identity tells you how fast the ecosystem is moving on your own properties.
3. Keep IP/rDNS verification as the fallback for the operators you care about, exactly as Google recommends.
4. Decide policy by identity, not by string. As verification becomes available, move your allow/deny decisions from user-agent patterns to verified identities.
5. Keep ground-truth detection running. Signed bots are the easy case. The bots that matter to your bottom line remain unsigned, spoofed, and increasingly browser-shaped — the case deterministic tripwires and behavioral analysis exist for.

Web Bot Auth doesn’t detect a single bad bot. What it does is remove the good bots from the argument — cryptographically, permanently — so the rest of your stack can concentrate on the traffic that’s actually lying to you.