Web Bot Auth

Web Bot Auth is a cryptographic standard that lets bots and AI agents prove their identity by signing HTTP requests with published keys under RFC 9421.

Web Bot Auth is a cryptographic identity standard for automated clients — crawlers, fetchers, and AI agents — that replaces spoofable User-Agent strings with verifiable signatures.

How it works

  • The bot operator publishes public keys (Ed25519 or RSA-PSS) as a JSON Web Key Set at /.well-known/http-message-signatures-directory on a domain it controls.
  • Each bot request is signed using RFC 9421 HTTP Message Signatures with a tag="web-bot-auth" parameter, and carries a Signature-Agent header pointing at that key directory.
  • The receiving server fetches the directory, matches the signature’s keyid against the published key thumbprints, and verifies the signature. A valid signature proves which operator sent the request, regardless of source IP or claimed user agent.

Why it matters

Anyone can send User-Agent: Googlebot; only Google can produce a signature that validates against Google’s published keys. Web Bot Auth makes bot identity decidable — enabling per-bot policy based on verified identity (allowlist, rate-limit, or block a specific operator) rather than string matching. Identity and permission stay separate decisions: a valid signature proves who a bot is, not that it’s welcome.

Because adoption is early, verification should be verify-and-allow: treat a valid signature as a strong positive signal, and never treat the absence of one as suspicious.

Deep dives: Web Bot Auth: Google Now Cryptographically Signs Crawlers and Signed Agents and the Coming Web Identity Schism

Frequently Asked Questions

Who uses Web Bot Auth today? +

Google is the first major operator, signing crawler traffic since June 2026 with Signature-Agent headers pointing to its key directory. The mechanism is still labeled experimental, and other large crawl operators have yet to deploy it, so coverage remains limited.

Does Web Bot Auth stop bad bots? +

Not directly. Malicious bots simply won't sign their requests. Its value is separating provably legitimate automation from everything else, so detection and enforcement can focus on the unsigned, spoofed traffic that actually causes harm.

← All glossary terms

See these concepts in action

WebDecoy puts deterministic detection and rotation-proof enforcement behind a 5-minute setup.

Start Free Trial