Every crawler below announces itself in a User-Agent header. That header is a claim, and claims cost nothing to make. Some operators publish the IP ranges their crawler runs from, which means the claim can actually be checked. So we checked it, request by request, across the detections WebDecoy has recorded. Most of that traffic hit WebDecoy's own properties, so read the figures as one small sample, not a measure of the whole web.

45.8%

of verifiable crawler requests did not come from the operator that was named.

Measured across the 6 crawlers whose operators publish IP ranges. The worst is GPTBot at 54.9%; the best is bingbot at 10.8%.

CrawlerOperatorPublishes IP rangesNot from operator
ChatGPT-User User-triggered fetcherOpenAIYes53.0%
Amazonbot Search / assistant crawlerAmazonNone foundunverifiable
Meta-ExternalAgent AI training crawlerMetaYesunverifiable
ClaudeBot AI training crawlerAnthropicYesunverifiable
Googlebot Search crawlerGoogleYes46.5%
Bytespider AI training crawlerByteDanceNone foundunverifiable
bingbot Search crawlerMicrosoftYes10.8%
PerplexityBot AI search crawlerPerplexity AIYes36.4%
GPTBot AI training crawlerOpenAIYes54.9%
OAI-SearchBot AI search crawlerOpenAIYes54.1%

The 10 most active crawlers, observed 2025-11-14 to 2026-08-19. Ordered by traffic volume. Licensed under CC BY 4.0; reuse is permitted with attribution to WebDecoy.

Download the underlying data for every bot WebDecoy recognises. Measured fields are filled only for the crawlers above; fields without an authoritative source are left empty.

Download CSV View JSON

Why four of them have no number

Google, Microsoft, OpenAI and Perplexity publish machine-readable lists of the addresses their crawlers operate from. Anyone can fetch the list and settle the question in a single lookup.

When we measured, Amazon, Meta, Anthropic and ByteDance offered no equivalent list, so we left their verification column empty rather than invent a figure. Since then Anthropic has published ranges for its crawlers, and Meta documents verification by its own network, so both can now be checked. Their pages below say how.

What to do with this

Allowlisting a crawler by its User-Agent gives the same access to anyone willing to type that string. On the evidence here, that is not a hypothetical worry. It is roughly half the traffic for several of the biggest names. Check against published ranges where they exist, fall back to forward-confirmed reverse DNS where they do not, and treat whatever is left as what it really is: unattributed traffic in a convincing costume.

Related reading: detecting AI scrapers by operator, the RAG bots that fetch your pages in real time, and JA4 fingerprinting for the clients that never announce themselves at all.

Every request recorded by WebDecoy's sensors, mostly on WebDecoy's own properties, whose User-Agent declares one of these crawlers, checked IP-by-IP against the operator's own published IP range list. A request is counted as verified when its source IP falls inside a published prefix, and forged when it does not. Operators that publish no machine-readable range list cannot be checked this way and are reported without a forgery figure.

Every crawler WebDecoy recognises

209 bots with a named operator, grouped by what they do. Each page says how to verify the bot and how to block it.

AI training crawlers

BotOperatorHow to verify
AbliterantBotAbliterantSee operator's docs
AI2BotAllen Institute for AISee operator's docs
AmazonbotAmazonReverse DNS
AnthropicAnthropicPublished IP ranges
Applebot-ExtendedApplePublished IP ranges
ByteSpiderByteDanceSee operator's docs
CCBotCommon CrawlReverse DNS
ChatGLM-SpiderZhipu AISee operator's docs
Claude-WebAnthropicSee operator's docs
ClaudeBotAnthropicPublished IP ranges
CohereCohereSee operator's docs
DeepSeekDeepSeekSee operator's docs
ElevenlabsCrawlerElevenLabsSee operator's docs
FacebookBotMetaOperator network (ASN)
FriendlyCrawlerUnknownSee operator's docs
GeminiGoogleSee operator's docs
Google-ExtendedGooglePublished IP ranges
GPTBotOpenAIPublished IP ranges
HunyuanUnknownSee operator's docs
img2datasetLAIONSee operator's docs
ISSCyberRiskCrawlerISSSee operator's docs
KimiBotMoonshot AISee operator's docs
Meta-ExternalAgentMetaOperator network (ASN)
MistralBotMistral AISee operator's docs
MoonshotBotMoonshot AISee operator's docs
OmgiliWebz.ioSee operator's docs
QwenAlibabaSee operator's docs
ReflectionbotReflection AISee operator's docs
RiverBotRiverSee operator's docs
SentibotSentibotSee operator's docs
SperidLabsResearchCrawlerSperidlabsSee operator's docs
TimpibotTimpiSee operator's docs
TokenFoundryBotUnknownSee operator's docs
VelenPublicWebCrawlerVelenSee operator's docs
Webz.ioWebz.ioSee operator's docs
YouBotYou.comSee operator's docs

AI search crawlers

BotOperatorHow to verify
Aranet-SearchBotAranetSee operator's docs
AzureAI-SearchBotMicrosoftSee operator's docs
BraveSearchBraveSee operator's docs
Claude-SearchBotAnthropicPublished IP ranges
DuckAssistBotDuckDuckGoSee operator's docs
ExaExa AISee operator's docs
GrokBotxAISee operator's docs
IsonSearchBotIson SearchSee operator's docs
KagiKagiSee operator's docs
KeenableBotKeenableSee operator's docs
Kimi-SearchBotMoonshot AISee operator's docs
LinkupBotLinkupSee operator's docs
Meta-WebIndexerMetaSee operator's docs
MetaphorMetaphorSee operator's docs
NeevaBotNeevaSee operator's docs
OAI-SearchBotOpenAIPublished IP ranges
PerplexityBotPerplexity AIPublished IP ranges
PhindPhindSee operator's docs
SearchGPTOpenAISee operator's docs
ShapBotParallel Web SystemsSee operator's docs
SofyaBotSofyaSee operator's docs
TalarionSearchBotTalarionSee operator's docs
TavilyTavilySee operator's docs
UnfragileBotUnfragileSee operator's docs
xAI-SearchBotxAISee operator's docs

AI agents

BotOperatorHow to verify
AmazonBuyForMeAmazonSee operator's docs
Claude-UserAnthropicPublished IP ranges
Gemini-UserGoogleSee operator's docs
GoogleAgent-MarinerGoogleSee operator's docs
Manus-UserManusSee operator's docs
MultiOnMultiOnSee operator's docs
OpenCode-UserOpenCodeSee operator's docs
OperatorOpenAISee operator's docs
StagehandBrowserbaseSee operator's docs

AI assistants

BotOperatorHow to verify
ChatGPT-UserOpenAIPublished IP ranges
CortanaMicrosoftSee operator's docs
ElevenlabsBotElevenLabsSee operator's docs
ExaUserExa AISee operator's docs
Gemini-Deep-ResearchGoogleSee operator's docs
Keenable-UserKeenableSee operator's docs
Kimi-UserMoonshot AISee operator's docs
KlaviyoAIBotKlaviyoSee operator's docs
Meta-ExternalFetcherMetaSee operator's docs
Microsoft CopilotMicrosoftSee operator's docs
MistralAI-UserMistral AISee operator's docs
Perplexity-UserPerplexity AIPublished IP ranges
Shap-UserParallel Web SystemsSee operator's docs
SiriAppleSee operator's docs

Search engine crawlers

BotOperatorHow to verify
360SpiderQihoo 360See operator's docs
ApplebotApplePublished IP ranges
BaiduspiderBaiduReverse DNS
BingbotMicrosoftPublished IP ranges
BravebotBraveSee operator's docs
CGMSearchBotCGMIMMSee operator's docs
Coc CocCoc CocSee operator's docs
DuckDuckBotDuckDuckGoPublished IP ranges
EcosiaEcosiaSee operator's docs
Google-InspectionToolGoogleSee operator's docs
GooglebotGooglePublished IP ranges
IbouBotIbouSee operator's docs
MojeekBotMojeekSee operator's docs
MSNBotMicrosoftSee operator's docs
NaverNaverSee operator's docs
PetalBotHuaweiReverse DNS
QwantifyQwantSee operator's docs
SeznamBotSeznamSee operator's docs
Sogou SpiderSogouReverse DNS
Yahoo! SlurpYahooSee operator's docs
YandexBotYandexReverse DNS
YepBotAhrefsSee operator's docs
YisouSpiderAlibabaSee operator's docs

SEO and marketing crawlers

BotOperatorHow to verify
AhrefsBotAhrefsReverse DNS
AwarioBotAwarioSee operator's docs
BarkrowlerBabbarSee operator's docs
BotifyBotifySee operator's docs
ContentKingContentKingSee operator's docs
CrolBotcloroSee operator's docs
DataForSeoBotDataForSEOSee operator's docs
DeepCrawlLumarSee operator's docs
DotBotMozSee operator's docs
HubSpot CrawlerHubSpotSee operator's docs
meta-externaladsMetaSee operator's docs
MJ12botMajesticReverse DNS
OnCrawlOnCrawlSee operator's docs
RogerbotMozSee operator's docs
Screaming FrogScreaming FrogSee operator's docs
ScrunchbotScrunchSee operator's docs
SemrushBotSemrushReverse DNS
SEOJuice-SearchBotSEOJuiceSee operator's docs
SERankingBacklinksBotSE RankingSee operator's docs
SerpstatSerpstatSee operator's docs
SISTRIXSISTRIXSee operator's docs
SitebulbSitebulbSee operator's docs
SpyFuSpyFuSee operator's docs
trendictionbotTalkwalkerSee operator's docs
webXtrendBotwebXtrendSee operator's docs

Link and preview fetchers

BotOperatorHow to verify
AdsBot-GoogleGoogleSee operator's docs
DiscordbotDiscordSee operator's docs
EmbedlyMediumSee operator's docs
FacebookMetaOperator network (ASN)
FacebotMetaSee operator's docs
Google-Read-AloudGoogleSee operator's docs
Google-SafetyGoogleSee operator's docs
IframelyIframelySee operator's docs
KakaoBotKakaoSee operator's docs
LinkedInBotLinkedInReverse DNS
OAI-AdsBotOpenAISee operator's docs
PinterestPinterestReverse DNS
RedditRedditSee operator's docs
SlackbotSlackSee operator's docs
StripebotStripeSee operator's docs
TelegramBotTelegramSee operator's docs
TumblrAutomatticSee operator's docs
TwitterbotX CorpReverse DNS
WgetGNUSee operator's docs
WhatsAppMetaSee operator's docs

Commercial scrapers

BotOperatorHow to verify
AntiCloneBotAntiCloneSee operator's docs
ApifyApifySee operator's docs
Bright DataBright DataSee operator's docs
BrightbotBright DataSee operator's docs
Cloudflare Browser Rendering CrawlerCloudflareSee operator's docs
CrawlbaseCrawlbaseSee operator's docs
DiffbotDiffbotSee operator's docs
ev-crawlerHeadlineSee operator's docs
Import.ioImport.ioSee operator's docs
NetAPI BotNetAPISee operator's docs
OxylabsOxylabsSee operator's docs
ScrapingAntScrapingAntSee operator's docs
ScrapingBeeScrapingBeeSee operator's docs
SpiderLingLexical ComputingSee operator's docs
TernCrawlerUnknownSee operator's docs
WebScrapingAPIWebScrapingAPISee operator's docs
ZyteZyteSee operator's docs

Security scanners

BotOperatorHow to verify
AcunetixInvictiSee operator's docs
Burp SuitePortSwiggerSee operator's docs
CensysCensysSee operator's docs
Cloudflare Security CenterCloudflareSee operator's docs
DetectifyDetectifySee operator's docs
IntruderIntruderSee operator's docs
NessusTenableSee operator's docs
NetsparkerInvictiSee operator's docs
NucleiProjectDiscoverySee operator's docs
OWASP ZAPOWASPSee operator's docs
Palo Alto Networks Cortex XpansePalo Alto NetworksSee operator's docs
QualysQualysSee operator's docs
Recorded Future CrawlerRecorded FutureSee operator's docs
ShodanShodanSee operator's docs
VirusTotalGoogleSee operator's docs

Uptime monitors

BotOperatorHow to verify
Better UptimeBetter StackSee operator's docs
Datadog SyntheticsDatadogPublished IP ranges
FreshpingFreshworksSee operator's docs
GTmetrixGTmetrixSee operator's docs
HetrixToolsHetrixToolsSee operator's docs
LighthouseGoogleSee operator's docs
New RelicNew RelicSee operator's docs
NodePingNodePingSee operator's docs
PingdomSolarWindsPublished IP ranges
Site24x7ZohoSee operator's docs
StatusCakeStatusCakeSee operator's docs
UptimeRobotUptimeRobotPublished IP ranges
WebDecoyBotWebDecoySee operator's docs

Feed readers

BotOperatorHow to verify
FeedbinFeedbinSee operator's docs
Feedfetcher-GoogleGoogleSee operator's docs
FeedlyFeedlySee operator's docs
InoreaderInoreaderSee operator's docs
LivelapBotLivelapSee operator's docs
NewsBlurNewsBlurSee operator's docs
The Old ReaderThe Old ReaderSee operator's docs

Web archivers

BotOperatorHow to verify
Archive.orgInternet ArchiveSee operator's docs
BrozzlerInternet ArchiveSee operator's docs
HeritrixInternet ArchiveSee operator's docs
Internet ArchiveInternet ArchiveSee operator's docs
Wayback MachineInternet ArchiveSee operator's docs

Tools, libraries and headless browsers

42 HTTP libraries, headless browsers and open-source tools. Anyone can run these, so their User-Agent says which software sent a request, not who sent it. Each page says what the tool is and how to handle it.

Find out which crawlers are lying to your site

WebDecoy checks declared crawlers against published ranges and tracks the ones that fail as persistent actors, not disposable IP addresses.

Start Free Trial