New AI Crawlers Seen in November 2026
Nine crawlers no bot list named yet, from an undocumented Alibaba bot to an anonymous crawler on 158 addresses in one day. What each sent and how to handle it.
WebDecoy names crawlers from a registry of 245 known bots, the same list behind our crawler directory. Every month some traffic arrives under names that no list has yet. These are the ones we saw between mid-September and mid-November 2026 that were worth a closer look.
About the data. These are detections recorded by WebDecoy, mostly on our own sites, from the edge sensor, our server SDKs and Vercel log drains. It is a small sample, so read the counts as minimums, not as each crawler’s size. A User-Agent and a contact address are what the crawler says about itself. Anyone can send them.
The list
| Crawler | First seen | Last seen | Requests | Addresses | Network (sampled) |
|---|---|---|---|---|---|
| TalarionSearchBot | Oct 18 | Nov 8 | 473 | 9 | Google Cloud |
| CrolBot | Nov 12 | Nov 16 | 461 | 11 | WorldStream (NL) |
| SperidLabsResearchCrawler | Nov 1 | Nov 1 | 365 | 95 | Amazon Web Services |
| AbliterantBot | Oct 10 | Nov 4 | 313 | 1 | US |
| Unnamed “crawler” | Nov 1 | Nov 1 | 228 | 158 | cleardocks LLC, 15 countries |
| TernCrawler | Oct 27 | Nov 13 | 188 | 14 | US and Europe |
| UnfragileBot | Oct 30 | Nov 15 | 57 | 6 | US |
| Aranet-SearchBot | Sep 20 | Nov 15 | 32 | 17 | US |
| TokenFoundryBot | Oct 20 | Nov 4 | 25 | 24 | Alibaba Cloud (HK, SG) |
TokenFoundryBot: an undocumented crawler with an Alibaba address
TokenFoundryBot/1.0 ([email protected])The User-Agent gives no website, only a contact address at service.alibaba.com. Every request came from Alibaba Cloud ranges: 23 addresses in 8.217.0.0/16 (Alibaba Cloud Hong Kong) and one in 8.218.0.0/16 (Alibaba Cloud Singapore). We saw 25 requests from 24 addresses over six days, almost one address per request. We found no public documentation for it.
The name suggests collecting text, since tokens are the units language models train on, but that is a guess from a name. The addresses are consistent with an Alibaba crawler, and Alibaba Cloud also rents machines to anyone, so the network alone does not prove who runs it. With no documentation, there is also no published robots.txt token or IP list to check it against.
An unnamed crawler on 158 addresses in one day
Mozilla/5.0 (compatible; crawler)On 1 November, 228 requests arrived with this User-Agent from 158 addresses in 15 countries, then stopped. The address we looked up belongs to cleardocks LLC. Spreading a crawl over that many addresses in that many countries in one day looks like a proxy network, not one company’s crawler. Without a name there is nothing to verify and nobody to ask.
SperidLabsResearchCrawler: 95 addresses in seven hours
SperidLabsResearchCrawler/1.0 (+https://speridlabs.com/crawler; research use; contact [email protected])It names itself clearly, with a URL and a contact, which is how a crawler should behave. It also crawled from 95 Amazon Web Services addresses within seven hours on the same day as the unnamed crawler above. We have no evidence the two are related.
CrolBot: new this week
Mozilla/5.0 (compatible; CrolBot/0.1; +https://cloro.dev)First seen on 12 November and active since, from 11 addresses at WorldStream, a hosting provider in the Netherlands. Its URL points to cloro.dev, which describes itself as “the search API for the AI era”: structured results from Google and AI search engines, for tracking visibility.
The rest
- TalarionSearchBot (
research; contact: [email protected]): 473 requests from 9 Google Cloud addresses over three weeks. It calls itself research and gives an email address but no web page. - AbliterantBot (
+https://abliterant.ai): 313 requests from a single address between 10 October and 4 November. - TernCrawler: 188 requests from 14 addresses in the US and Europe. Its User-Agent has no URL or contact at all.
- UnfragileBot (
AI infrastructure search index): 57 requests. Two of them were for decoy URLs that no person is ever shown, which means it followed links a visitor cannot see. - Aranet-SearchBot (
+https://aranet.ai/bot): 32 requests from 17 addresses since September, one of them for a decoy URL.
What to do about them
- Each has a distinct name, and a robots.txt
Disallowrule for that name is the polite request. A crawler with no documentation may not read robots.txt at all. - Blocking by name stops only the crawlers that keep their name. The unnamed one above shows how little a User-Agent can tell you.
- Judging by behaviour works whatever a crawler calls itself: how fast it arrives, from how many addresses, and whether it follows links no person can see. That is how WebDecoy flagged the decoy requests above.
The eight named crawlers above are now in the registry, so WebDecoy and the crawler directory name them. The unnamed one gives nothing to name. If you run one of them and want its description corrected, write to [email protected].
Frequently Asked Questions
What is TokenFoundryBot? +
A crawler identifying itself as TokenFoundryBot/1.0 with the contact address [email protected]. We saw it from 24 addresses in Alibaba Cloud ranges in Hong Kong and Singapore between 20 October and 4 November 2026. We found no public documentation for it, and a contact address in a User-Agent is a claim anyone can make.
How do I block a crawler that is not on any list yet? +
If it names a robots.txt token, a Disallow rule asks it to stay away, but a new crawler may not read robots.txt. Blocking by User-Agent stops only crawlers that keep the name. Judging requests by behaviour, such as rate, paths and whether they follow links no person can see, works whatever the name.
Why does a crawler with no name matter? +
A User-Agent of just 'crawler' tells a site owner nothing about who is collecting their pages or why. The one we saw came from 158 addresses in 15 countries on a single day, which looks like a proxy network rather than one company's crawler.
Continue your investigation
- Browse the crawler directory
Who runs each of 245 known crawlers and how to verify it.
- Block AI training crawlers on WordPress
Keep training crawlers out by category with an automatic robots.txt.
Share this post
Like this post? Share it with your friends!
Want to see WebDecoy in action?
Get a personalized demo from our team.