ClaudeBot

AI training crawler operated by Anthropic

None found

Published IP ranges

Reverse DNS

Verification method

ClaudeBot

robots.txt token

AI training crawler

Category

Measured across WebDecoy's production detection corpus, 2025-11-14 to 2026-08-19.

What ClaudeBot is

Collects public web content used to train Claude models.

It identifies itself with a User-Agent containing ClaudeBot/1.0; [email protected], and honours the robots.txt token ClaudeBot. Operator documentation: Anthropic.

How much of this traffic is actually ClaudeBot

There is no number here, and the reason is the interesting part. We could not find a published, machine-readable list of the IP ranges ClaudeBot crawls from. Anthropic points site owners at reverse DNS instead. With no range list there is nothing to check traffic against in bulk, so any forgery rate we published would be guesswork dressed up as data.

The absence is the finding. Google, Microsoft, OpenAI and Perplexity all publish ranges, which is what lets anyone confirm or reject their crawlers in a single lookup. Where no list exists, the User-Agent string is the only claim on offer, and copying it costs an attacker nothing.

Verifying it yourself

With no published range list, use a forward-confirmed reverse DNS lookup. Resolve the address, check the hostname belongs to Anthropic, then resolve that hostname back and confirm it returns the address you started with:

# 1. reverse lookup: hostname should belong to Anthropic
dig +short -x 203.0.113.42

# 2. forward lookup: must return the address you started with
dig +short that-hostname.example

The round trip is the point. A reverse record alone proves nothing, because whoever controls the address also controls what it claims to be called.

Blocking it

If you want the real crawler to stay away, robots.txt is the polite way to ask:

User-agent: ClaudeBot
Disallow: /

That only works on a crawler willing to obey it, which rules out every request that was already lying about who it is. Stopping those takes enforcement at the network or application layer, keyed on something the client cannot simply retype. That is the problem WebDecoy exists to solve.

For the wider picture, see how to detect AI scrapers across GPTBot, ClaudeBot and Perplexity, or the mechanics of cryptographically verifying a trusted bot rather than taking its word for it.

Frequently Asked Questions

Is ClaudeBot a real crawler or is it being faked? +

ClaudeBot is a genuine crawler operated by Anthropic. We cannot report a forgery rate for it, because Anthropic does not publish a machine-readable IP range list, so there is no authoritative source to check a request against. Anyone can send its User-Agent string, and without a published range there is no cheap way to disprove them.

How do I verify ClaudeBot myself? +

Anthropic publishes no machine-readable range list, so use a reverse DNS lookup: resolve the source IP to a hostname, confirm the hostname belongs to Anthropic, then resolve that hostname back to an IP and confirm it matches the original address. The forward-confirmed round trip is what makes it trustworthy; a reverse lookup on its own can be spoofed.

Should I block ClaudeBot? +

That depends on what it does for you. Collects public web content used to train Claude models. Blocking the verified traffic is a business decision, and only you can make it. Blocking the unverified traffic is not much of a decision at all, because requests that only claim to be ClaudeBot bring none of the upside.

Does ClaudeBot respect robots.txt? +

Anthropic says it does, and you can disallow it with the token ClaudeBot. We are not claiming to have checked. Proving compliance needs a disallowed path to watch, and the sites in this corpus disallow nothing, so no crawler had anything to violate. Treat published compliance as a vendor statement until someone measures it. What the data here answers is the question that comes first: whether a request came from Anthropic at all.

Methodology

Every request in WebDecoy's production detection corpus whose User-Agent declares one of these crawlers, checked IP-by-IP against the operator's own published IP range list. A request is counted as verified when its source IP falls inside a published prefix, and forged when it does not. Operators that publish no machine-readable range list cannot be checked this way and are reported without a forgery figure. Corpus: 81,034 detections from 17,187 distinct addresses, 2025-11-14 to 2026-08-19. Published ranges retrieved 2026-08-19. Operators revise their ranges over time, so a request logged months ago is checked against today's list; we re-ran the comparison over a recent 30-day window and the rates held.

Other crawlers

← All crawlers

See which ClaudeBot requests are real

WebDecoy verifies declared crawlers against published ranges and identifies the ones that fail, as persistent actors rather than disposable IPs.

Start Free Trial