Cryptographic Verification for Trusted Bots

Not all bots are malicious. Search engines, AI assistants, and monitoring services need a reliable way to prove their identity to your site. But how do you tell a real AI agent from a scraper that copied its User-Agent?

Cryptography answers that question better than any header can. This post explains Web Bot Auth, the emerging IETF approach built on RFC 9421 HTTP Message Signatures, and what WebDecoy does with it today.

The Problem: Bots That Claim to Be Legitimate

Traditional bot identification relies on claims and heuristics:

• Does the User-Agent match a known bot?
• Does the IP fall in the operator’s published ranges, and does forward-confirmed reverse DNS agree?
• Does the behavior look like that bot?

A User-Agent is one header and costs nothing to copy. A scraper that sets User-Agent: GPTBot inherits whatever you granted OpenAI, unless you check. IP ranges and reverse DNS are much better, and WebDecoy uses them for crawlers that publish ranges. But many agents publish nothing, and some run from shared cloud infrastructure where an IP says little about who is behind it.

That’s where cryptographic verification comes in.

How Web Bot Auth Works

With Web Bot Auth, the agent signs each request with a private key and publishes the matching public keys where anyone can fetch them.

RFC 9421: HTTP Message Signatures

Agent sends a request:
├── Signature-Agent: which origin publishes its keys
├── Signature-Input: which components are signed, plus keyid, created/expires, tag="web-bot-auth"
└── Signature: the signature over those components

Verifier receives it:
├── Resolves the key directory at
│   <Signature-Agent origin>/.well-known/http-message-signatures-directory
├── Finds the key named by keyid (a JWK thumbprint)
├── Rebuilds the signature base from the live request (RFC 9421)
├── Checks the signature and the created/expires window
└── Verified → this request was signed by the holder of that key

Example signed request (shape only):

GET /pricing HTTP/1.1
Host: example.com
Signature-Agent: "https://chatgpt.com"
Signature-Input: sig1=("@authority" "signature-agent");created=1735689600;expires=1735693200;keyid="<jwk-thumbprint>";alg="ed25519";tag="web-bot-auth"
Signature: sig1=:<base64 signature>:

Key benefit: the signature covers the request’s @authority (the host it was sent to), so it cannot be lifted off one request and replayed against another site, and nobody without the private key can produce a valid one.

The caveat: a valid signature proves which key signed the request. Whether that key belongs to an agent you want on your site is a separate decision.

What WebDecoy Does With Signatures

Verification runs in several places, and they deliberately differ in which keys they trust.

On WebDecoy decoys: verify whoever signs

When a request hits a WebDecoy-hosted decoy, the ingest service verifies any Web Bot Auth signature at request time. It has to happen then: the signature covers the live request’s host, so it cannot be re-checked later. For detection purposes, WebDecoy will resolve the key directory of whatever https origin the request names (behind an SSRF guard), because the goal is to verify whoever claims an identity, not only a short list. Ed25519 and RSA-PSS-SHA512 signatures are supported.

The result is one of three outcomes:

At your edge: a curated set of trusted signers

The clearance validator that runs at your edge does not fetch arbitrary directories on the request path. Instead, WebDecoy’s backend fetches a small, curated set of signed-agent directories on a schedule and delivers their Ed25519 keys to the edge in its configuration. The curated set today is OpenAI ChatGPT (chatgpt.com), Google Agent (agent.bot.goog, Google’s AI browsing agent, which is not Googlebot), and WebDecoyBot (bot.webdecoy.com). Each carries a category: AI crawlers or monitoring.

The edge distinguishes three signed cases:

• Verified against a curated key
• Invalid: the signature failed against a key the edge actually holds. That is forgery evidence.
• Unknown key: well-formed, but signed with a key outside the curated set. The edge cannot judge it either way, so it treats it as unproven, not forged. Most honest signing agents on the internet fall here today, and branding them impersonators would be wrong.

WebDecoy also records the Signature-Agent origins it sees in traffic that are not in the curated set, so additions can be prioritized by real volume. Adding a signer is a deliberate human decision; an unrecognized signer never becomes trusted automatically.

In your own app: the Node.js SDK

The @webdecoy/node SDK verifies signatures locally against its own curated directories, with no API key required:

import { WebDecoy, webBotAuth } from '@webdecoy/node';

const webdecoy = new WebDecoy({ rules: [webBotAuth()] });

const verdict = await webdecoy.detectBot(request);
// verdict.status: 'verified' | 'impersonation' | 'claimed' | 'none'

The webBotAuth() rule denies impersonation by default and allows unverifiable claimed signatures unless you set onClaimed: 'DENY'. When used through the framework middleware, nothing is blocked until you switch from the default monitor mode to mode: 'enforce'.

Verified, Unproven, Forged: What Changes

A verification result is not an automatic allow or an automatic block. Here is what each outcome actually does.

A forged identity raises the score

When a request claims a known agent’s identity and the proof fails (a signature that does not verify, or, for crawlers that publish IP ranges, an address outside those ranges), WebDecoy scores it as agent impersonation. That is a tripwire-grade signal: it floors the threat score, and the detection records the claimed agent name so you can see who is being impersonated against you.

A verified identity changes how the numbers read

A verified crawler is exactly as automated as it says it is. So verified agents are not escalated just for being automated or for operating from many IP addresses, which is how a legitimate crawler fleet works.

Allowances are yours to switch on

On the Enforcement page’s Policy tab, under Tokenless clients, you can let verified agents through the clearance gate without a token, by category (search engines, AI crawlers, monitoring). Because the allowance keys on verification rather than on the User-Agent string, turning on “allow AI crawlers” no longer means trusting a header. An agent whose signature fails is denied that bypass and handled like any other untrusted client.

These toggles only matter where the clearance gate is enforcing. In monitor mode, nothing is blocked either way.

Where you see it

• Detections carry the signature fields (signature_agent, signature_verified) alongside the rest of the detection. signature_verified is also included in events forwarded to Splunk and CrowdStrike, and CrowdStrike events carry signature_agent too.
• The AI Crawlers & Agents report divides every request that claimed a known agent into proved who they are, unproven claims, and forged identities, with a per-agent breakdown and links straight into the forged detections.

To check a single signed request by hand, use the free Web Bot Auth signature checker.

WebDecoy Signs Its Own Crawler

We ask agents to sign, so we sign too. Since August 19, 2026, WebDecoyBot, the crawler WebDecoy uses to verify sensor installs and run readiness scans on domains their owners have verified, signs its requests with Web Bot Auth. Its User-Agent is Mozilla/5.0 (compatible; WebDecoyBot/1.0; +https://bot.webdecoy.com) and its keys are published at https://bot.webdecoy.com/.well-known/http-message-signatures-directory. Any Web Bot Auth verifier, not only ours, can check that a request claiming to be WebDecoyBot really is.

Why Standards Matter

Web Bot Auth builds on RFC 9421: HTTP Message Signatures, and the profile itself is being developed in the IETF’s webbotauth working group. Using a standard means:

1. Interoperability: an agent that signs once can be verified by any site or vendor
2. Security review: the design is vetted in the open
3. No lock-in: keys live in the agent’s own public directory, not in a vendor’s database

The profile is still a draft and moving. WebDecoy tracks the drafts and keeps the verification code isolated so protocol changes touch one place.

Limitations

• Few agents sign yet. Most crawlers do not sign, so IP-range and forward-confirmed reverse DNS verification still carry most verified traffic.
• The edge trusts a short list. Signatures from signers outside the curated set are recorded as unproven at the edge, not verified.
• Verification is identity, not intent. A verified agent is who it says it is. Whether it should read your content is your policy decision.

Conclusion

Cryptographic verification turns “it says it’s GPTBot” into a question with a real answer. A valid signature proves which key signed the request; a failed one is strong evidence of impersonation; and no signature leaves you where you were, relying on IP ranges and behavior.

WebDecoy verifies signatures where it can see them, scores forged identities as impersonation, reports proved, unproven, and forged traffic separately, and lets you decide which verified categories to let through.

Want to see who is claiming to be whom on your site? Start free and open the AI Crawlers & Agents report, or read the verified agents documentation.