Scraping library

Type

Open source

Made by

No

Verifiable

None

robots.txt token

What Norma WebFetch is

HTTP WebFetch client used by agent runtimes, including ARTEX.

It has no robots.txt token. Documentation: project site .

Why it cannot be verified

Crawlers run by a company can be checked: the company publishes the addresses or DNS names its crawler uses, and a request from anywhere else is not really it. Norma WebFetch is software anyone can run, so there is no such list. The User-Agent tells you which software sent a request, or which software the sender chose to name. Changing it is one line of code, so the default User-Agent mostly identifies people who did not bother.

Handling it

Before blocking it, check whether you rely on it: your own scripts, uptime checks, webhooks and partner integrations often use tools like this. Then judge requests by what they do rather than the name they give:

  • How fast they arrive, and from how many addresses
  • Which paths they ask for, such as admin pages, config files or every product page in order
  • Whether they touch links or fields no person could see

robots.txt does not help here: it only works on a client that chooses to read and obey it.

That is the approach WebDecoy takes: hidden decoy links and traps that only automated clients follow, so a request is judged by its behaviour, whatever User-Agent it sends.

Frequently Asked Questions

What is Norma WebFetch? +

Norma WebFetch is a scraping library, open-source software. HTTP WebFetch client used by agent runtimes, including ARTEX.

Can I verify a request from Norma WebFetch? +

No. Norma WebFetch is software anyone can run, so there is no operator whose IP ranges or DNS names a request could be checked against. Its User-Agent says which software sent the request, or which software the sender chose to name, not who sent it.

Should I block Norma WebFetch? +

It depends on your site. Your own scripts, monitoring and integrations may use it. Blocking on the User-Agent alone only stops clients that leave the default in place, and anyone can change it. Judge requests by behaviour instead: request rate, the paths they ask for, and whether they touch pages no person would.

Related tools

← All crawlers

Catch automated clients, whatever they call themselves

WebDecoy sets hidden decoys only automated clients follow, and tracks them as persistent actors rather than disposable IPs.

Start Free Trial