Secret Scanners Are Wearing AI Crawler Names
In 90 days, two sites we run logged 5,259 requests for .env files and other secrets under names like ClaudeBot and Googlebot. One name is fake by definition.
Site owners have spent the past two years deciding what to do about AI crawlers. Some block them, some welcome them, and many have written a rule that treats a request differently when its user agent says ClaudeBot, GPTBot or Googlebot.
The people who scan the web for leaked secrets have noticed. Over the last 90 days, two sites we operate logged 5,259 requests for secret files that introduced themselves as a well-known crawler. None of the ones we could check passed an identity check.
What we counted
We looked at every request to two production sites we run between August 15 and November 12, 2026, and kept the ones that met both conditions:
- The user agent named one of 20 well-known search or AI crawlers.
- The path asked for something no crawler has a reason to fetch: environment files, Git internals, cloud credentials, SSH keys and similar.
We then removed requests from crawlers that passed identity verification (more on those below). What was left:
| Requests | 5,259 |
| Days with at least one | 74 of 90 |
| Distinct secret paths requested | 315 |
| Crawler names used | 18 |
Which names they borrow
| Name in the user agent | Requests | Share |
|---|---|---|
| ClaudeBot | 1,193 | 23% |
| Googlebot | 506 | 10% |
| PerplexityBot | 454 | 9% |
| ChatGPT-User | 386 | 7% |
| Amazonbot | 322 | 6% |
| OAI-SearchBot | 308 | 6% |
| Meta-ExternalAgent | 307 | 6% |
| GPTBot | 291 | 6% |
| Baiduspider | 284 | 5% |
| CCBot | 263 | 5% |
| 8 other names | 945 | 18% |
Two things stand out. AI crawler names account for 77% of these requests, and classic search crawlers for 23%. And ClaudeBot alone was used more than twice as often as Googlebot, which for most of the web’s history was the default costume for anything that wanted to look harmless.
This is a count of disguises, not a statement about the real crawlers. It says nothing about how Anthropic, Google or anyone else behaves.
What they ask for
| Target | Requests | Distinct paths |
|---|---|---|
Environment files (/.env, /.env.production, /.env.bak, /api/.env …) | 3,817 | 226 |
Vite dev-server file reads (/@fs/...) | 736 | 31 |
Git internals (/.git/config, /.git/HEAD, /.git-credentials) | 256 | 9 |
AWS credentials (/.aws/credentials, /.aws/config) | 160 | 26 |
rclone configuration (/rclone.conf) | 75 | 1 |
Other (/.ssh/id_rsa, /.htpasswd, /.npmrc, /phpinfo.php) | 215 | 22 |
The 226 environment-file variants are the tell that this is tooling and not curiosity. Nobody types /.env.save, /.env.old and /admin/.env by hand. The /@fs/ requests target development servers that were left reachable from the internet.
One scanner, 87 seconds
The clearest single example arrived on August 19. One virtual machine at a large cloud provider sent 391 requests in 87 seconds, starting at 22:23:38 UTC. Almost every request used a different path, and the user agent rotated between eight crawler names: ClaudeBot, GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, Amazonbot, Amzn-SearchBot and Google-Extended.
A few of the requests, in arrival order:
Google-Extended /proc/1/environ
Amazonbot /@fs/home/ec2-user/.aws/credentials?raw??
ChatGPT-User /@fs/root/.aws/credentials?raw??
Google-Extended /.env
ClaudeBot /media../etc/passwd
GPTBot /static../etc/passwd
Amazonbot /fetch?url=http://169.254.169.254/latest/meta-data/iam/security-credentials/
ClaudeBot /media../.envEach line is a known technique. The ?raw?? suffix matches a file-read bypass in the Vite development server (CVE-2025-30208). The /static../ and /media../ shapes probe for a common web-server alias misconfiguration. The 169.254.169.254 request tries to make the site fetch its own cloud credentials. The crawler names are decoration on top of an ordinary vulnerability scanner.
The address belonged to a cloud provider’s general customer range, not to Anthropic, OpenAI, Perplexity, Amazon’s crawler or Google’s crawler.
The name that is always fake
One of those eight names deserves its own section, because it gives you a rule with no false positives.
Google-Extended is not a crawler. It is a robots.txt token that lets a site control whether content Google has already crawled may be used for its AI models. Google’s crawler documentation says it “doesn’t have a separate HTTP request user agent string” and that crawling is done with Google’s existing user agents.
So no genuine request will ever carry it. Across the two sites we still saw 535 requests in 90 days with Google-Extended in the user agent, in forms like:
Mozilla/5.0 (compatible; Google-Extended; +http://www.google.com/bot.html)
Mozilla/5.0 (compatible; Google-Extended/1.0; +http://www.google.com/bot.html)The likely explanation is that someone built a list of crawler names from robots.txt examples and turned every entry into a user agent without checking which ones exist on the wire. Whatever the cause, the rule is simple: a user agent containing Google-Extended was not sent by Google.
What we could verify, and what we could not
Checking a crawler’s identity means comparing where the request came from against where the operator says its crawler lives: reverse DNS for Googlebot and Bingbot, published IP ranges for OpenAI, Anthropic and Perplexity, or a cryptographic signature where one is offered.
That check is only as good as the address you compare. The two sites differ here, and the difference matters:
- Site one records the true visitor address. It accounts for 1,049 of the requests, from 144 addresses on 60 different days. None of them passed verification as the crawler named in the user agent. Of those 144 addresses, 86 used two or more crawler names and one used eleven.
- Site two accounts for the other 4,210. While preparing this post we found that its install was recording a proxy hop in front of the site as the visitor address. Network verification is meaningless under that condition, because every request appears to come from the proxy. We therefore report that site’s requests by name and path only, and make no claim about how many senders were involved.
We almost published a different story. With the proxy addresses taken at face value, site two looked like a single operator with about 60 servers in nine countries, cycling through crawler names every day for three months. It was a tidy narrative and it was wrong: the “servers” were the proxy’s points of presence. The misconfiguration also had the opposite effect on genuine crawlers, which all failed verification on that site for the same reason.
The one real crawler in the data
Before filtering, 32 requests in the set did pass verification. All 32 were Meta’s crawler asking for paths like /industries/publishers.git/info/refs, which is a page URL with a Git suffix appended. That looks like a fetcher testing whether a URL is a Git repository, not a hunt for secrets, and we removed those requests from every number above. We mention them because a method that never finds an exception is usually not looking.
What to do with this
- Never grant access on a crawler name alone. If you have a rule that allows, skips or rate-limits differently when the user agent says ClaudeBot or Googlebot, anyone can borrow it by typing that name.
- Verify, then allow. Use reverse DNS for Googlebot and Bingbot, the operator’s published IP ranges for OpenAI, Anthropic and Perplexity, and signatures where the operator supports them. Our free crawler verification tool runs these checks for a single address.
- Treat
Google-Extendedin a user agent as fake. It costs one line and cannot hit a genuine crawler. - Refuse secret paths for everyone. No crawler, verified or not, needs
/.env,/.git/configor/.aws/credentials. Deny those paths before identity is even considered. - Confirm you are verifying the right address. If your site sits behind a CDN, load balancer or platform proxy, make sure your application reads the visitor address from the correct hop. Get it wrong in one direction and real crawlers look forged. Get it wrong in the other and a visitor can supply an address of their choosing.
- Keep development servers off the internet. The
/@fs/requests only work against a dev server that is publicly reachable.
Limits
This is two sites that we operate, not a sample of the web, and the counts are small in absolute terms. It does not give a forgery rate for any crawler, and the ranking of names could look different on a site with a different audience. What it does show is concrete: secret-scanning tools now ship with AI crawler user agents built in, and a name in a header is not an identity.
Check the traffic reaching your own site
Start with one suspicious request: take its real visitor IP and User-Agent from your logs and check the crawler claim. If your logs contain a proxy address, fix that before interpreting the result. The tool uses current verification data; it cannot reconstruct historical IP ownership or DNS.
For ongoing visibility in WordPress, install the free WebDecoy plugin. It shows local counts of recognized bots and supported identity checks in wp-admin, without a WebDecoy account or API key. Keep monitor mode on while reviewing your normal site flows. Requests served before WordPress PHP, including cached pages, are outside that local view.
Using another stack? Choose a WebDecoy integration and verify that its sensor reports before interpreting the traffic.
Frequently Asked Questions
Is a request from ClaudeBot or Googlebot asking for /.env really from that crawler? +
Almost certainly not. On the site where we could check network origin, 1,049 requests for secret files arrived under well-known crawler names over 90 days, and none passed identity verification. Search and AI crawlers fetch pages and public files. They do not ask for environment files, cloud credentials or Git internals.
Is Google-Extended a real crawler user agent? +
No. Google-Extended is a robots.txt token that controls how Google may use content it has already crawled. Google's documentation states that it has no separate HTTP user agent string. Any request whose user agent contains Google-Extended was written by someone else.
Should I allow AI crawlers by user agent? +
Not by user agent alone. The name is a text field the sender chooses. Verify the crawler by reverse DNS, by the operator's published IP ranges, or by a cryptographic signature where the operator offers one, and apply the allowance only to requests that pass.
Why do attackers use AI crawler names? +
Our data cannot show intent. A reasonable reading is that many sites now carry rules that treat AI and search crawlers differently from other traffic, and a scanner that borrows the name inherits whatever that rule allows.
Continue your investigation
- Check whether a crawler is genuine
Enter an IP address and a claimed crawler name to test it against the operator's published network.
- Install WebDecoy for WordPress
See local bot counts and supported identity checks in wp-admin. No WebDecoy account or API key needed.
- Add WebDecoy to another stack
Choose the integration for your site and check which requests it can observe.
Share this post
Like this post? Share it with your friends!
Want to see WebDecoy in action?
Get a personalized demo from our team.