WebDecoy names crawlers from a registry of 245 known bots, the same list behind our crawler directory. Every month some traffic arrives under names that no list has yet. These are the ones we saw between mid-September and mid-November 2026 that were worth a closer look.

About the data. These are detections recorded by WebDecoy, mostly on our own sites, from the edge sensor, our server SDKs and Vercel log drains. It is a small sample, so read the counts as minimums, not as each crawler’s size. A User-Agent and a contact address are what the crawler says about itself. Anyone can send them.

The list

CrawlerFirst seenLast seenRequestsAddressesNetwork (sampled)
TalarionSearchBotOct 18Nov 84739Google Cloud
CrolBotNov 12Nov 1646111WorldStream (NL)
SperidLabsResearchCrawlerNov 1Nov 136595Amazon Web Services
AbliterantBotOct 10Nov 43131US
Unnamed “crawler”Nov 1Nov 1228158cleardocks LLC, 15 countries
TernCrawlerOct 27Nov 1318814US and Europe
UnfragileBotOct 30Nov 15576US
Aranet-SearchBotSep 20Nov 153217US
TokenFoundryBotOct 20Nov 42524Alibaba Cloud (HK, SG)

TokenFoundryBot: an undocumented crawler with an Alibaba address

TokenFoundryBot/1.0 ([email protected])

The User-Agent gives no website, only a contact address at service.alibaba.com. Every request came from Alibaba Cloud ranges: 23 addresses in 8.217.0.0/16 (Alibaba Cloud Hong Kong) and one in 8.218.0.0/16 (Alibaba Cloud Singapore). We saw 25 requests from 24 addresses over six days, almost one address per request. We found no public documentation for it.

The name suggests collecting text, since tokens are the units language models train on, but that is a guess from a name. The addresses are consistent with an Alibaba crawler, and Alibaba Cloud also rents machines to anyone, so the network alone does not prove who runs it. With no documentation, there is also no published robots.txt token or IP list to check it against.

An unnamed crawler on 158 addresses in one day

Mozilla/5.0 (compatible; crawler)

On 1 November, 228 requests arrived with this User-Agent from 158 addresses in 15 countries, then stopped. The address we looked up belongs to cleardocks LLC. Spreading a crawl over that many addresses in that many countries in one day looks like a proxy network, not one company’s crawler. Without a name there is nothing to verify and nobody to ask.

SperidLabsResearchCrawler: 95 addresses in seven hours

SperidLabsResearchCrawler/1.0 (+https://speridlabs.com/crawler; research use; contact [email protected])

It names itself clearly, with a URL and a contact, which is how a crawler should behave. It also crawled from 95 Amazon Web Services addresses within seven hours on the same day as the unnamed crawler above. We have no evidence the two are related.

CrolBot: new this week

Mozilla/5.0 (compatible; CrolBot/0.1; +https://cloro.dev)

First seen on 12 November and active since, from 11 addresses at WorldStream, a hosting provider in the Netherlands. Its URL points to cloro.dev, which describes itself as “the search API for the AI era”: structured results from Google and AI search engines, for tracking visibility.

The rest

  • TalarionSearchBot (research; contact: [email protected]): 473 requests from 9 Google Cloud addresses over three weeks. It calls itself research and gives an email address but no web page.
  • AbliterantBot (+https://abliterant.ai): 313 requests from a single address between 10 October and 4 November.
  • TernCrawler: 188 requests from 14 addresses in the US and Europe. Its User-Agent has no URL or contact at all.
  • UnfragileBot (AI infrastructure search index): 57 requests. Two of them were for decoy URLs that no person is ever shown, which means it followed links a visitor cannot see.
  • Aranet-SearchBot (+https://aranet.ai/bot): 32 requests from 17 addresses since September, one of them for a decoy URL.

What to do about them

  • Each has a distinct name, and a robots.txt Disallow rule for that name is the polite request. A crawler with no documentation may not read robots.txt at all.
  • Blocking by name stops only the crawlers that keep their name. The unnamed one above shows how little a User-Agent can tell you.
  • Judging by behaviour works whatever a crawler calls itself: how fast it arrives, from how many addresses, and whether it follows links no person can see. That is how WebDecoy flagged the decoy requests above.

The eight named crawlers above are now in the registry, so WebDecoy and the crawler directory name them. The unnamed one gives nothing to name. If you run one of them and want its description corrected, write to [email protected].

Frequently Asked Questions

What is TokenFoundryBot? +

A crawler identifying itself as TokenFoundryBot/1.0 with the contact address [email protected]. We saw it from 24 addresses in Alibaba Cloud ranges in Hong Kong and Singapore between 20 October and 4 November 2026. We found no public documentation for it, and a contact address in a User-Agent is a claim anyone can make.

How do I block a crawler that is not on any list yet? +

If it names a robots.txt token, a Disallow rule asks it to stay away, but a new crawler may not read robots.txt. Blocking by User-Agent stops only crawlers that keep the name. Judging requests by behaviour, such as rate, paths and whether they follow links no person can see, works whatever the name.

Why does a crawler with no name matter? +

A User-Agent of just 'crawler' tells a site owner nothing about who is collecting their pages or why. The one we saw came from 158 addresses in 15 countries on a single day, which looks like a proxy network rather than one company's crawler.

Continue your investigation

  1. Browse the crawler directory

    Who runs each of 245 known crawlers and how to verify it.

  2. Block AI training crawlers on WordPress

    Keep training crawlers out by category with an automatic robots.txt.

Want to see WebDecoy in action?

Get a personalized demo from our team.

Request Demo