State of Bots 2026

Who's crawling and who's lying

168 agents. 115 organizations. 31,518 bot detections analysed. One in four AI training crawlers ignores robots.txt and says so in its own documentation. Some of what arrives claiming to be Googlebot is not Googlebot at all.

0% of search crawlers ignore robots.txt 0 of 13
25% of AI training crawlers ignore robots.txt 7 of 28
168 agents tracked and classified across 115 organizations

Who ignores robots.txt, by what they are

Share of agents in each category that declare they do not honour robots.txt. Hover a bar for the agents behind it.

The landscape, by category

Every agent WebDecoy classifies, sized by how many are in each category and coloured by whether they respect robots.txt. Hover any cell for the agent.

The AI agents that say they won't honour robots.txt

Named rather than aggregated. A claim about "some AI crawlers" cannot be checked; a claim naming an operator can be, against their own published documentation. Every entry below is classified from the operator's stated behaviour, not inferred from traffic.

Sentibot

Sentibot

Sentiment analysis training crawler

VelenPublicWebCrawler

Velen

Velen public web data crawler

ISSCyberRiskCrawler

ISS

ISS cyber risk assessment crawler

Part two: how they behave

What the traffic actually looks like

31,518 bot detections from 11,868 addresses, resolved into 2,063 distinct actors and analysed since November 2025. Observed on webdecoy.com, which runs the complete sensor stack: honeypot decoys, the behavioural scanner, and a Cloudflare edge sensor that catches the crawlers which never execute JavaScript. What follows are ratios and distributions rather than raw totals, because how an operator behaves is the part that transfers to your site.

1,520 addresses used by a single actor, for 1,523 requests. Roughly one request per IP, across 9 countries.
2,186 addresses sharing one TLS fingerprint. The IP changed 2,186 times; the handshake never did.
63% of checked requests came from VPN or hosting infrastructure, not consumer connections. Tor was 39.

Those first two numbers are the same story told from both ends, and together they are the case against blocking by address. An actor that spends one request per IP is unreachable by IP, and was identifiable the whole time by the one thing it could not rotate.

Speed is the other half of it. One actor moved through 529 addresses in 14 hours, across 28 countries, at roughly 38 new addresses an hour. A blocklist updated daily is behind before it is written.

Claiming to be a crawler, and not being one

Share of requests claiming each identity that failed forward-confirmed reverse DNS, the method Google itself recommends for verifying Googlebot. Only agents our verification demonstrably works for are shown; see the method note below.

Requests per IP address, by network

The top networks by volume. Position on the diagonal is the tell: hosting providers reuse a handful of addresses heavily, consumer ISPs spread thin. Hover for the numbers.

Part three: where it comes from

Most of it was already known to be trouble

Every source address is checked against AbuseIPDB, whose scores are contributed by thousands of operators. This is the one measurement here that does not depend on us: a detection landing on a maxed-out address is traffic the rest of the internet had already reported.

52% arrived from addresses already reported for abuse by someone else, before they reached us.
16% came from addresses scored 100 out of 100, the maximum abuse confidence AbuseIPDB assigns.
10,760 abuse reports filed against a single address in our data. Tor accounted for 86 detections in total.

64% came from data centre address space, which is the easy half. The rest arrived on consumer broadband, where residential proxy networks resell access by the gigabyte, and where blocking the address means blocking whoever is assigned it next.

How much of this traffic was already known

AbuseIPDB confidence score for the source address at the time of detection. The score is community-reported, so the right-hand bar is traffic that thousands of other operators had already flagged.

Where it came from

Top source countries by detection count.

What kind of connection

Network type of the source address. Consumer broadband is where residential proxy networks resell access, which is why it is not a safe signal on its own.

Part four: how they give themselves away

The tells, and who is willing to identify themselves

24,915 scored requests, and the signals that fired on them. A single request usually trips several at once, which is the whole design: any one of these alone would be a guess.

1,713 detections of AI browser extensions reading pages inside a real person's session. ChatGPT leads, then Copilot, Perplexity, Gemini and Claude.
0.9% carried a Web Bot Auth signature, the emerging standard for bots that want to prove who they are. Adoption is real but early.
881 separate operators had independently reported the same address before it reached us.

Why bother claiming to be Googlebot? Because it still works. A great many WAF rules, CDN configurations and SEO plugins carry a hardcoded allowlist that matches on the User-Agent string alone, written years ago so that nobody would accidentally deindex their own site. A scraper that sets one header inherits that exemption. Verifying the claim takes a reverse DNS lookup that resolves back to the source address, which is the check Google itself publishes and most stacks never run.

The agents that do sign are Ahrefs, DuckDuckBot, YouBot. Search and SEO crawlers, not the AI labs. The companies with the most to gain from being trusted are the ones not yet proving who they are.

What is actually on the other end

Only 21% of automated traffic identifies itself. The rest arrives as a headless browser, a bare script, or a real browser with an assistant attached.

37% Browser-shaped, unidentified 9,116 detections
23% Headless browser 5,717 detections
21% Declared crawler 5,240 detections
17% Script, no browser 4,266 detections
3% Real browser, AI extension 694 detections

What gives them away

The signals that fired most often across scored traffic. A single request usually trips several, which is the point: any one of them alone would be a guess.

AI assistants reading the page over your visitor's shoulder

Browser extensions that pull page content into an assistant. These arrive inside a real person's session, on a real browser, from a residential address. Nothing about the connection looks automated, because it isn't.

What to do about it

Four things worth doing, none of which need us

Every finding above has a fix you can apply this afternoon with tools you already have. None of them require WebDecoy.

1. Verify Googlebot instead of trusting the header

A User-Agent is a claim, not an identity. Confirm it the way Google documents: reverse the address, then forward-resolve the result and check it returns the same IP. A PTR record alone is not enough.

# 1. reverse the address
$ host 66.249.66.1
  -> crawl-66-249-66-1.googlebot.com

# 2. resolve that name back
$ host crawl-66-249-66-1.googlebot.com
  -> 66.249.66.1   (must match)

Match on dot boundaries. A hostname like fake.googlebot.com.example.net ends with neither .googlebot.com nor .google.com, and a naive substring check passes it.

2. Rate limit by network, not by address

Rotation defeats per-IP limits by design: 529 addresses in 14 hours means no single address ever reaches a threshold. Limit by ASN or by /24 instead, so the budget follows the network rather than the address. Cloudflare, Fastly and most WAFs expose the ASN as a rule field.

3. Put your robots.txt policy in writing anyway

A quarter of AI training crawlers ignore it, which means three quarters do not. Name the ones you want to exclude explicitly, by the token their operator publishes, rather than relying on a wildcard. Compliance you can get for free is worth taking.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Bytespider
Disallow: /

4. Log the TLS fingerprint, not just the IP

The address changed 2,186 times while the handshake stayed identical. If your CDN exposes a JA3 or JA4 value, log it beside the IP. You do not need to act on it today, but you cannot investigate an incident with data you never recorded.

Method, and what this is not

Part one counts the registry WebDecoy runs in production. Every agent is one the product recognises by name, with a category, an operator, and a position on robots.txt taken from that operator's own published documentation. Each figure can be checked against the table that classifies live traffic.

Part two is observed behaviour, recorded on webdecoy.com: 31,518 bot detections from 11,868 addresses, resolved into 2,063 actors. The site carries every sensor the product ships, which is why one property yields a corpus this size. Reported as ratios such as requests per address and share of claims that failed verification, because those describe how an operator behaves rather than how busy one week was.

"Ignores robots.txt" means declared, not observed. It records what the operator publishes about its own crawler, not that we caught it in the act. Several entries, notably HTTP client libraries and security scanners, are marked non-compliant because robots.txt is not a concept that applies to them. That is why the charts break the figure out by category instead of quoting one number for all 168.

Impersonation rates only include agents our verification demonstrably works for. Not every operator publishes the reverse DNS records this method depends on, so an agent is only reported once verification has demonstrably succeeded on its traffic. Agents below that bar are left out rather than reported as failures, because an unverifiable claim is not the same as a false one.

Signals are what fired, not what was decisive. A request usually trips several, and no single one classifies anything on its own. The counts are of signals raised across scored traffic, so they sum to more than the number of requests.

IP reputation is third-party. Abuse scores come from AbuseIPDB, recorded at the moment of detection. Scores move as operators report and as addresses change hands, so these are a snapshot rather than a standing verdict on any address.

Corrections welcome. If your crawler is listed wrongly, or its policy has changed, tell us and we will change the table the product runs on, not just this page.