Googlebot

What Googlebot is

Google Search indexing. The single most impersonated crawler on the web.

It identifies itself with a User-Agent containing Googlebot/2.1; +http://www.google.com/bot.html, and honours the robots.txt token Googlebot. Operator documentation: Google.

How much of this traffic is actually Googlebot

Google publishes the IP ranges its crawler operates from, so this has a real answer rather than an opinion. We took every request in our corpus declaring this User-Agent and checked its source address against that published list.

46.5% of traffic claiming to be Googlebot could not be traced back to Google. It is something else wearing the name. Most sites wave this User-Agent straight through without checking, which is exactly why the name is worth wearing.

Verifying it yourself

Fetch the published ranges and test the source address against them:

# 1. fetch the published ranges
curl -s -o ranges.json \
  https://developers.google.com/search/apis/ipranges/googlebot.json

# 2. check an address against them
python3 - 203.0.113.42 <<'PY'
import json, sys, ipaddress
ip = ipaddress.ip_address(sys.argv[1])
data = json.load(open('ranges.json'))
hit = False
for p in data['prefixes']:
    net = p.get('ipv4Prefix') or p['ipv6Prefix']
    if ip in ipaddress.ip_network(net):
        hit = True
print('verified' if hit else 'NOT VERIFIED')
PY

Swap in the address you are checking. Never trust the User-Agent on its own. It is a header the client picks for itself, and as the numbers above show, plenty of clients pick a lie.

Blocking it

If you want the real crawler to stay away, robots.txt is the polite way to ask:

User-agent: Googlebot
Disallow: /

That only works on a crawler willing to obey it, which rules out every request that was already lying about who it is. Stopping those takes enforcement at the network or application layer, keyed on something the client cannot simply retype. That is the problem WebDecoy exists to solve.

For the wider picture, see how to detect AI scrapers across GPTBot, ClaudeBot and Perplexity, or the mechanics of cryptographically verifying a trusted bot rather than taking its word for it.

Frequently Asked Questions

Methodology

Every request in WebDecoy's production detection corpus whose User-Agent declares one of these crawlers, checked IP-by-IP against the operator's own published IP range list. A request is counted as verified when its source IP falls inside a published prefix, and forged when it does not. Operators that publish no machine-readable range list cannot be checked this way and are reported without a forgery figure. Corpus: 81,034 detections from 17,187 distinct addresses, 2025-11-14 to 2026-08-19. Published ranges retrieved 2026-08-19. Operators revise their ranges over time, so a request logged months ago is checked against today's list; we re-ran the comparison over a recent 30-day window and the rates held.

Other crawlers

• ChatGPT-User 53.0% not from OpenAI
• Amazonbot unverifiable
• Meta-ExternalAgent unverifiable
• ClaudeBot unverifiable
• Bytespider unverifiable
• bingbot 10.8% not from Microsoft
• PerplexityBot 36.4% not from Perplexity AI
• GPTBot 54.9% not from OpenAI
• OAI-SearchBot 54.1% not from OpenAI

← All crawlers