Crawler Directory

Every crawler below announces itself in a User-Agent header. That header is a claim, and claims cost nothing to make. Some operators publish the IP ranges their crawler runs from, which means the claim can actually be checked. So we checked it, request by request, across our production detection corpus.

Why four of them have no number

Google, Microsoft, OpenAI and Perplexity publish machine-readable lists of the addresses their crawlers operate from. Anyone can fetch the list and settle the question in a single lookup.

For Amazon, Meta, Anthropic and ByteDance we could not find an equivalent list. Those operators point site owners at reverse DNS instead, which works one address at a time but gives you nothing to audit traffic against in bulk. So we leave the verification column empty rather than invent a figure. That gap is not a hole in our data. It is a limit on what those operators make it possible for anyone to know.

What to do with this

Allowlisting a crawler by its User-Agent gives the same access to anyone willing to type that string. On the evidence here, that is not a hypothetical worry. It is roughly half the traffic for several of the biggest names. Check against published ranges where they exist, fall back to forward-confirmed reverse DNS where they do not, and treat whatever is left as what it really is: unattributed traffic in a convincing costume.

Related reading: detecting AI scrapers by operator, the RAG bots that fetch your pages in real time, and JA4 fingerprinting for the clients that never announce themselves at all.

Every request in WebDecoy's production detection corpus whose User-Agent declares one of these crawlers, checked IP-by-IP against the operator's own published IP range list. A request is counted as verified when its source IP falls inside a published prefix, and forged when it does not. Operators that publish no machine-readable range list cannot be checked this way and are reported without a forgery figure.