libwww-perl
Link and preview fetcher · Open source
Link and preview fetcher
Type
Open source
Made by
No
Verifiable
None
robots.txt token
What libwww-perl is
Perl LWP library.
It has no robots.txt token. Documentation: project site .
Why it cannot be verified
Crawlers run by a company can be checked: the company publishes the addresses or DNS names its crawler uses, and a request from anywhere else is not really it. libwww-perl is software anyone can run, so there is no such list. The User-Agent tells you which software sent a request, or which software the sender chose to name. Changing it is one line of code, so the default User-Agent mostly identifies people who did not bother.
Handling it
Before blocking it, check whether you rely on it: your own scripts, uptime checks, webhooks and partner integrations often use tools like this. Then judge requests by what they do rather than the name they give:
- How fast they arrive, and from how many addresses
- Which paths they ask for, such as admin pages, config files or every product page in order
- Whether they touch links or fields no person could see
robots.txt does not help here: it only works on a client that chooses to read and obey it.
That is the approach WebDecoy takes: hidden decoy links and traps that only automated clients follow, so a request is judged by its behaviour, whatever User-Agent it sends.
Frequently Asked Questions
What is libwww-perl? +
libwww-perl is a link and preview fetcher, open-source software. Perl LWP library.
Can I verify a request from libwww-perl? +
No. libwww-perl is software anyone can run, so there is no operator whose IP ranges or DNS names a request could be checked against. Its User-Agent says which software sent the request, or which software the sender chose to name, not who sent it.
Should I block libwww-perl? +
It depends on your site. Your own scripts, monitoring and integrations may use it. Blocking on the User-Agent alone only stops clients that leave the default in place, and anyone can change it. Judge requests by behaviour instead: request rate, the paths they ask for, and whether they touch pages no person would.
Catch automated clients, whatever they call themselves
WebDecoy sets hidden decoys only automated clients follow, and tracks them as persistent actors rather than disposable IPs.