Meta

Operator

Operator network (ASN)

Verification

Meta-ExternalAgent

robots.txt token

Yes

Honours robots.txt

Measured across detections recorded by WebDecoy, mostly on WebDecoy's own properties, 2025-11-14 to 2026-08-19. A small sample, not the whole web.

What Meta-ExternalAgent is

Collects public web data used to train Meta AI models.

It identifies itself with a User-Agent containing meta-externalagent/1.1. Its robots.txt token is Meta-ExternalAgent , and it is listed as honouring robots.txt. Operator documentation: Meta .

Verifying it yourself

Meta documents verification by network rather than a list: genuine requests come from addresses its own autonomous system announces. Look up the announced prefixes and check the source address against them:

curl -s 'https://stat.ripe.net/data/announced-prefixes/data.json?resource=AS32934'

Swap in the address you are checking. Never trust the User-Agent on its own: it is a header the client picks for itself.

Blocking it

If you want the real crawler to stay away, robots.txt is the polite way to ask:

User-agent: Meta-ExternalAgent
Disallow: /

robots.txt only works on a client willing to obey it, which rules out every request that was already lying about who it is. Stopping those takes enforcement at the network or application layer, keyed on something the client cannot simply retype. That is the problem WebDecoy exists to solve.

Frequently Asked Questions

What is Meta-ExternalAgent? +

Meta-ExternalAgent is an ai training crawler operated by Meta. Collects public web data used to train Meta AI models.

How do I verify Meta-ExternalAgent? +

Meta documents verification by network: genuine requests come from addresses announced by its own autonomous system. Check the source address against the prefixes that network announces (WebDecoy reads them from https://stat.ripe.net/data/announced-prefixes/data.json?resource=AS32934).

How do I block Meta-ExternalAgent? +

Add a robots.txt rule for the token Meta-ExternalAgent. That stops the real crawler, but not clients that only borrow its name, so pair it with verification and enforcement at your edge.

Methodology

Every request recorded by WebDecoy's sensors, mostly on WebDecoy's own properties, whose User-Agent declares one of these crawlers, checked IP-by-IP against the operator's own published IP range list. A request is counted as verified when its source IP falls inside a published prefix, and forged when it does not. Operators that publish no machine-readable range list cannot be checked this way and are reported without a forgery figure. Corpus: 81,034 detections from 17,187 distinct addresses, 2025-11-14 to 2026-08-19. Published ranges retrieved 2026-08-19. Operators revise their ranges over time, so a request logged months ago is checked against today's list; we re-ran the comparison over a recent 30-day window and the rates held.

This crawler dataset is licensed under CC BY 4.0; reuse is permitted with attribution to WebDecoy.

See which Meta-ExternalAgent requests are real

WebDecoy verifies declared crawlers against what their operators publish and identifies the ones that fail, as persistent actors rather than disposable IPs.

Start Free Trial