Meta-ExternalAgent

AI training crawler operated by Meta

None found

Published IP ranges

Reverse DNS

Verification method

meta-externalagent

robots.txt token

AI training crawler

Category

Measured across WebDecoy's production detection corpus, 2025-11-14 to 2026-08-19.

What Meta-ExternalAgent is

Collects public web data used to train Meta AI models.

It identifies itself with a User-Agent containing meta-externalagent/1.1, and honours the robots.txt token meta-externalagent. Operator documentation: Meta.

How much of this traffic is actually Meta-ExternalAgent

There is no number here, and the reason is the interesting part. We could not find a published, machine-readable list of the IP ranges Meta-ExternalAgent crawls from. Meta points site owners at reverse DNS instead. With no range list there is nothing to check traffic against in bulk, so any forgery rate we published would be guesswork dressed up as data.

The absence is the finding. Google, Microsoft, OpenAI and Perplexity all publish ranges, which is what lets anyone confirm or reject their crawlers in a single lookup. Where no list exists, the User-Agent string is the only claim on offer, and copying it costs an attacker nothing.

Verifying it yourself

With no published range list, use a forward-confirmed reverse DNS lookup. Resolve the address, check the hostname belongs to Meta, then resolve that hostname back and confirm it returns the address you started with:

# 1. reverse lookup: hostname should belong to Meta
dig +short -x 203.0.113.42

# 2. forward lookup: must return the address you started with
dig +short that-hostname.example

The round trip is the point. A reverse record alone proves nothing, because whoever controls the address also controls what it claims to be called.

Blocking it

If you want the real crawler to stay away, robots.txt is the polite way to ask:

User-agent: meta-externalagent
Disallow: /

That only works on a crawler willing to obey it, which rules out every request that was already lying about who it is. Stopping those takes enforcement at the network or application layer, keyed on something the client cannot simply retype. That is the problem WebDecoy exists to solve.

For the wider picture, see how to detect AI scrapers across GPTBot, ClaudeBot and Perplexity, or the mechanics of cryptographically verifying a trusted bot rather than taking its word for it.

Frequently Asked Questions

Is Meta-ExternalAgent a real crawler or is it being faked? +

Meta-ExternalAgent is a genuine crawler operated by Meta. We cannot report a forgery rate for it, because Meta does not publish a machine-readable IP range list, so there is no authoritative source to check a request against. Anyone can send its User-Agent string, and without a published range there is no cheap way to disprove them.

How do I verify Meta-ExternalAgent myself? +

Meta publishes no machine-readable range list, so use a reverse DNS lookup: resolve the source IP to a hostname, confirm the hostname belongs to Meta, then resolve that hostname back to an IP and confirm it matches the original address. The forward-confirmed round trip is what makes it trustworthy; a reverse lookup on its own can be spoofed.

Should I block Meta-ExternalAgent? +

That depends on what it does for you. Collects public web data used to train Meta AI models. Blocking the verified traffic is a business decision, and only you can make it. Blocking the unverified traffic is not much of a decision at all, because requests that only claim to be Meta-ExternalAgent bring none of the upside.

Does Meta-ExternalAgent respect robots.txt? +

Meta says it does, and you can disallow it with the token meta-externalagent. We are not claiming to have checked. Proving compliance needs a disallowed path to watch, and the sites in this corpus disallow nothing, so no crawler had anything to violate. Treat published compliance as a vendor statement until someone measures it. What the data here answers is the question that comes first: whether a request came from Meta at all.

Methodology

Every request in WebDecoy's production detection corpus whose User-Agent declares one of these crawlers, checked IP-by-IP against the operator's own published IP range list. A request is counted as verified when its source IP falls inside a published prefix, and forged when it does not. Operators that publish no machine-readable range list cannot be checked this way and are reported without a forgery figure. Corpus: 81,034 detections from 17,187 distinct addresses, 2025-11-14 to 2026-08-19. Published ranges retrieved 2026-08-19. Operators revise their ranges over time, so a request logged months ago is checked against today's list; we re-ran the comparison over a recent 30-day window and the rates held.

Other crawlers

← All crawlers

See which Meta-ExternalAgent requests are real

WebDecoy verifies declared crawlers against published ranges and identifies the ones that fail, as persistent actors rather than disposable IPs.

Start Free Trial