Crawler Directory
Who runs each bot, how to verify it, and how much of the traffic using its name really is it
Every crawler below announces itself in a User-Agent header. That header is a claim, and claims cost nothing to make. Some operators publish the IP ranges their crawler runs from, which means the claim can actually be checked. So we checked it, request by request, across the detections WebDecoy has recorded. Most of that traffic hit WebDecoy's own properties, so read the figures as one small sample, not a measure of the whole web.
45.8%
of verifiable crawler requests did not come from the operator that was named.
Measured across the 6 crawlers whose operators publish IP ranges. The worst is GPTBot at 54.9%; the best is bingbot at 10.8%.
| Crawler | Operator | Publishes IP ranges | Not from operator |
|---|---|---|---|
| ChatGPT-User User-triggered fetcher | OpenAI | Yes | 53.0% |
| Amazonbot Search / assistant crawler | Amazon | None found | unverifiable |
| Meta-ExternalAgent AI training crawler | Meta | Yes | unverifiable |
| ClaudeBot AI training crawler | Anthropic | Yes | unverifiable |
| Googlebot Search crawler | Yes | 46.5% | |
| Bytespider AI training crawler | ByteDance | None found | unverifiable |
| bingbot Search crawler | Microsoft | Yes | 10.8% |
| PerplexityBot AI search crawler | Perplexity AI | Yes | 36.4% |
| GPTBot AI training crawler | OpenAI | Yes | 54.9% |
| OAI-SearchBot AI search crawler | OpenAI | Yes | 54.1% |
The 10 most active crawlers, observed 2025-11-14 to 2026-08-19. Ordered by traffic volume. Licensed under CC BY 4.0; reuse is permitted with attribution to WebDecoy.
Download the underlying data for every bot WebDecoy recognises. Measured fields are filled only for the crawlers above; fields without an authoritative source are left empty.
Download CSV View JSONWhy four of them have no number
Google, Microsoft, OpenAI and Perplexity publish machine-readable lists of the addresses their crawlers operate from. Anyone can fetch the list and settle the question in a single lookup.
When we measured, Amazon, Meta, Anthropic and ByteDance offered no equivalent list, so we left their verification column empty rather than invent a figure. Since then Anthropic has published ranges for its crawlers, and Meta documents verification by its own network, so both can now be checked. Their pages below say how.
What to do with this
Allowlisting a crawler by its User-Agent gives the same access to anyone willing to type that string. On the evidence here, that is not a hypothetical worry. It is roughly half the traffic for several of the biggest names. Check against published ranges where they exist, fall back to forward-confirmed reverse DNS where they do not, and treat whatever is left as what it really is: unattributed traffic in a convincing costume.
Related reading: detecting AI scrapers by operator, the RAG bots that fetch your pages in real time, and JA4 fingerprinting for the clients that never announce themselves at all.
Every request recorded by WebDecoy's sensors, mostly on WebDecoy's own properties, whose User-Agent declares one of these crawlers, checked IP-by-IP against the operator's own published IP range list. A request is counted as verified when its source IP falls inside a published prefix, and forged when it does not. Operators that publish no machine-readable range list cannot be checked this way and are reported without a forgery figure.
Every crawler WebDecoy recognises
209 bots with a named operator, grouped by what they do. Each page says how to verify the bot and how to block it.
AI training crawlers
| Bot | Operator | How to verify |
|---|---|---|
| AbliterantBot | Abliterant | See operator's docs |
| AI2Bot | Allen Institute for AI | See operator's docs |
| Amazonbot | Amazon | Reverse DNS |
| Anthropic | Anthropic | Published IP ranges |
| Applebot-Extended | Apple | Published IP ranges |
| ByteSpider | ByteDance | See operator's docs |
| CCBot | Common Crawl | Reverse DNS |
| ChatGLM-Spider | Zhipu AI | See operator's docs |
| Claude-Web | Anthropic | See operator's docs |
| ClaudeBot | Anthropic | Published IP ranges |
| Cohere | Cohere | See operator's docs |
| DeepSeek | DeepSeek | See operator's docs |
| ElevenlabsCrawler | ElevenLabs | See operator's docs |
| FacebookBot | Meta | Operator network (ASN) |
| FriendlyCrawler | Unknown | See operator's docs |
| Gemini | See operator's docs | |
| Google-Extended | Published IP ranges | |
| GPTBot | OpenAI | Published IP ranges |
| Hunyuan | Unknown | See operator's docs |
| img2dataset | LAION | See operator's docs |
| ISSCyberRiskCrawler | ISS | See operator's docs |
| KimiBot | Moonshot AI | See operator's docs |
| Meta-ExternalAgent | Meta | Operator network (ASN) |
| MistralBot | Mistral AI | See operator's docs |
| MoonshotBot | Moonshot AI | See operator's docs |
| Omgili | Webz.io | See operator's docs |
| Qwen | Alibaba | See operator's docs |
| Reflectionbot | Reflection AI | See operator's docs |
| RiverBot | River | See operator's docs |
| Sentibot | Sentibot | See operator's docs |
| SperidLabsResearchCrawler | Speridlabs | See operator's docs |
| Timpibot | Timpi | See operator's docs |
| TokenFoundryBot | Unknown | See operator's docs |
| VelenPublicWebCrawler | Velen | See operator's docs |
| Webz.io | Webz.io | See operator's docs |
| YouBot | You.com | See operator's docs |
AI search crawlers
| Bot | Operator | How to verify |
|---|---|---|
| Aranet-SearchBot | Aranet | See operator's docs |
| AzureAI-SearchBot | Microsoft | See operator's docs |
| BraveSearch | Brave | See operator's docs |
| Claude-SearchBot | Anthropic | Published IP ranges |
| DuckAssistBot | DuckDuckGo | See operator's docs |
| Exa | Exa AI | See operator's docs |
| GrokBot | xAI | See operator's docs |
| IsonSearchBot | Ison Search | See operator's docs |
| Kagi | Kagi | See operator's docs |
| KeenableBot | Keenable | See operator's docs |
| Kimi-SearchBot | Moonshot AI | See operator's docs |
| LinkupBot | Linkup | See operator's docs |
| Meta-WebIndexer | Meta | See operator's docs |
| Metaphor | Metaphor | See operator's docs |
| NeevaBot | Neeva | See operator's docs |
| OAI-SearchBot | OpenAI | Published IP ranges |
| PerplexityBot | Perplexity AI | Published IP ranges |
| Phind | Phind | See operator's docs |
| SearchGPT | OpenAI | See operator's docs |
| ShapBot | Parallel Web Systems | See operator's docs |
| SofyaBot | Sofya | See operator's docs |
| TalarionSearchBot | Talarion | See operator's docs |
| Tavily | Tavily | See operator's docs |
| UnfragileBot | Unfragile | See operator's docs |
| xAI-SearchBot | xAI | See operator's docs |
AI agents
| Bot | Operator | How to verify |
|---|---|---|
| AmazonBuyForMe | Amazon | See operator's docs |
| Claude-User | Anthropic | Published IP ranges |
| Gemini-User | See operator's docs | |
| GoogleAgent-Mariner | See operator's docs | |
| Manus-User | Manus | See operator's docs |
| MultiOn | MultiOn | See operator's docs |
| OpenCode-User | OpenCode | See operator's docs |
| Operator | OpenAI | See operator's docs |
| Stagehand | Browserbase | See operator's docs |
AI assistants
| Bot | Operator | How to verify |
|---|---|---|
| ChatGPT-User | OpenAI | Published IP ranges |
| Cortana | Microsoft | See operator's docs |
| ElevenlabsBot | ElevenLabs | See operator's docs |
| ExaUser | Exa AI | See operator's docs |
| Gemini-Deep-Research | See operator's docs | |
| Keenable-User | Keenable | See operator's docs |
| Kimi-User | Moonshot AI | See operator's docs |
| KlaviyoAIBot | Klaviyo | See operator's docs |
| Meta-ExternalFetcher | Meta | See operator's docs |
| Microsoft Copilot | Microsoft | See operator's docs |
| MistralAI-User | Mistral AI | See operator's docs |
| Perplexity-User | Perplexity AI | Published IP ranges |
| Shap-User | Parallel Web Systems | See operator's docs |
| Siri | Apple | See operator's docs |
Search engine crawlers
| Bot | Operator | How to verify |
|---|---|---|
| 360Spider | Qihoo 360 | See operator's docs |
| Applebot | Apple | Published IP ranges |
| Baiduspider | Baidu | Reverse DNS |
| Bingbot | Microsoft | Published IP ranges |
| Bravebot | Brave | See operator's docs |
| CGMSearchBot | CGMIMM | See operator's docs |
| Coc Coc | Coc Coc | See operator's docs |
| DuckDuckBot | DuckDuckGo | Published IP ranges |
| Ecosia | Ecosia | See operator's docs |
| Google-InspectionTool | See operator's docs | |
| Googlebot | Published IP ranges | |
| IbouBot | Ibou | See operator's docs |
| MojeekBot | Mojeek | See operator's docs |
| MSNBot | Microsoft | See operator's docs |
| Naver | Naver | See operator's docs |
| PetalBot | Huawei | Reverse DNS |
| Qwantify | Qwant | See operator's docs |
| SeznamBot | Seznam | See operator's docs |
| Sogou Spider | Sogou | Reverse DNS |
| Yahoo! Slurp | Yahoo | See operator's docs |
| YandexBot | Yandex | Reverse DNS |
| YepBot | Ahrefs | See operator's docs |
| YisouSpider | Alibaba | See operator's docs |
SEO and marketing crawlers
| Bot | Operator | How to verify |
|---|---|---|
| AhrefsBot | Ahrefs | Reverse DNS |
| AwarioBot | Awario | See operator's docs |
| Barkrowler | Babbar | See operator's docs |
| Botify | Botify | See operator's docs |
| ContentKing | ContentKing | See operator's docs |
| CrolBot | cloro | See operator's docs |
| DataForSeoBot | DataForSEO | See operator's docs |
| DeepCrawl | Lumar | See operator's docs |
| DotBot | Moz | See operator's docs |
| HubSpot Crawler | HubSpot | See operator's docs |
| meta-externalads | Meta | See operator's docs |
| MJ12bot | Majestic | Reverse DNS |
| OnCrawl | OnCrawl | See operator's docs |
| Rogerbot | Moz | See operator's docs |
| Screaming Frog | Screaming Frog | See operator's docs |
| Scrunchbot | Scrunch | See operator's docs |
| SemrushBot | Semrush | Reverse DNS |
| SEOJuice-SearchBot | SEOJuice | See operator's docs |
| SERankingBacklinksBot | SE Ranking | See operator's docs |
| Serpstat | Serpstat | See operator's docs |
| SISTRIX | SISTRIX | See operator's docs |
| Sitebulb | Sitebulb | See operator's docs |
| SpyFu | SpyFu | See operator's docs |
| trendictionbot | Talkwalker | See operator's docs |
| webXtrendBot | webXtrend | See operator's docs |
Link and preview fetchers
| Bot | Operator | How to verify |
|---|---|---|
| AdsBot-Google | See operator's docs | |
| Discordbot | Discord | See operator's docs |
| Embedly | Medium | See operator's docs |
| Meta | Operator network (ASN) | |
| Facebot | Meta | See operator's docs |
| Google-Read-Aloud | See operator's docs | |
| Google-Safety | See operator's docs | |
| Iframely | Iframely | See operator's docs |
| KakaoBot | Kakao | See operator's docs |
| LinkedInBot | Reverse DNS | |
| OAI-AdsBot | OpenAI | See operator's docs |
| Reverse DNS | ||
| See operator's docs | ||
| Slackbot | Slack | See operator's docs |
| Stripebot | Stripe | See operator's docs |
| TelegramBot | Telegram | See operator's docs |
| Tumblr | Automattic | See operator's docs |
| Twitterbot | X Corp | Reverse DNS |
| Wget | GNU | See operator's docs |
| Meta | See operator's docs |
Commercial scrapers
| Bot | Operator | How to verify |
|---|---|---|
| AntiCloneBot | AntiClone | See operator's docs |
| Apify | Apify | See operator's docs |
| Bright Data | Bright Data | See operator's docs |
| Brightbot | Bright Data | See operator's docs |
| Cloudflare Browser Rendering Crawler | Cloudflare | See operator's docs |
| Crawlbase | Crawlbase | See operator's docs |
| Diffbot | Diffbot | See operator's docs |
| ev-crawler | Headline | See operator's docs |
| Import.io | Import.io | See operator's docs |
| NetAPI Bot | NetAPI | See operator's docs |
| Oxylabs | Oxylabs | See operator's docs |
| ScrapingAnt | ScrapingAnt | See operator's docs |
| ScrapingBee | ScrapingBee | See operator's docs |
| SpiderLing | Lexical Computing | See operator's docs |
| TernCrawler | Unknown | See operator's docs |
| WebScrapingAPI | WebScrapingAPI | See operator's docs |
| Zyte | Zyte | See operator's docs |
Security scanners
| Bot | Operator | How to verify |
|---|---|---|
| Acunetix | Invicti | See operator's docs |
| Burp Suite | PortSwigger | See operator's docs |
| Censys | Censys | See operator's docs |
| Cloudflare Security Center | Cloudflare | See operator's docs |
| Detectify | Detectify | See operator's docs |
| Intruder | Intruder | See operator's docs |
| Nessus | Tenable | See operator's docs |
| Netsparker | Invicti | See operator's docs |
| Nuclei | ProjectDiscovery | See operator's docs |
| OWASP ZAP | OWASP | See operator's docs |
| Palo Alto Networks Cortex Xpanse | Palo Alto Networks | See operator's docs |
| Qualys | Qualys | See operator's docs |
| Recorded Future Crawler | Recorded Future | See operator's docs |
| Shodan | Shodan | See operator's docs |
| VirusTotal | See operator's docs |
Uptime monitors
| Bot | Operator | How to verify |
|---|---|---|
| Better Uptime | Better Stack | See operator's docs |
| Datadog Synthetics | Datadog | Published IP ranges |
| Freshping | Freshworks | See operator's docs |
| GTmetrix | GTmetrix | See operator's docs |
| HetrixTools | HetrixTools | See operator's docs |
| Lighthouse | See operator's docs | |
| New Relic | New Relic | See operator's docs |
| NodePing | NodePing | See operator's docs |
| Pingdom | SolarWinds | Published IP ranges |
| Site24x7 | Zoho | See operator's docs |
| StatusCake | StatusCake | See operator's docs |
| UptimeRobot | UptimeRobot | Published IP ranges |
| WebDecoyBot | WebDecoy | See operator's docs |
Feed readers
| Bot | Operator | How to verify |
|---|---|---|
| Feedbin | Feedbin | See operator's docs |
| Feedfetcher-Google | See operator's docs | |
| Feedly | Feedly | See operator's docs |
| Inoreader | Inoreader | See operator's docs |
| LivelapBot | Livelap | See operator's docs |
| NewsBlur | NewsBlur | See operator's docs |
| The Old Reader | The Old Reader | See operator's docs |
Web archivers
| Bot | Operator | How to verify |
|---|---|---|
| Archive.org | Internet Archive | See operator's docs |
| Brozzler | Internet Archive | See operator's docs |
| Heritrix | Internet Archive | See operator's docs |
| Internet Archive | Internet Archive | See operator's docs |
| Wayback Machine | Internet Archive | See operator's docs |
Tools, libraries and headless browsers
42 HTTP libraries, headless browsers and open-source tools. Anyone can run these, so their User-Agent says which software sent a request, not who sent it. Each page says what the tool is and how to handle it.
- aiohttp
- Apache-HttpClient
- ARTEX
- Axios
- BeautifulSoup
- Browser-Use
- Browserbase
- Browserless
- Colly
- CommaFeed
- cURL
- Faraday
- FreshRSS
- Go-HTTP-Client
- Got
- GuzzleHttp
- HeadlessChrome
- HTTPX
- HTTrack
- Java
- Jsoup
- libwww-perl
- LWP
- Mechanize
- Miniflux
- Nikto
- Nmap
- node-fetch
- Norma WebFetch
- OkHttp
- PhantomJS
- PHP
- Playwright
- Puppeteer
- Python
- Python-Requests
- Python-urllib
- Ruby
- Scrapy
- Selenium
- Splash
- Undici
Find out which crawlers are lying to your site
WebDecoy checks declared crawlers against published ranges and tracks the ones that fail as persistent actors, not disposable IP addresses.