Browser-Use
AI agent · Open source
AI agent
Type
Open source
Made by
No
Verifiable
Browser-Use
robots.txt token
What Browser-Use is
Browser-Use AI browser automation framework.
Its robots.txt token is Browser-Use , but it is not listed as honouring robots.txt. Documentation: project site .
Why it cannot be verified
Crawlers run by a company can be checked: the company publishes the addresses or DNS names its crawler uses, and a request from anywhere else is not really it. Browser-Use is software anyone can run, so there is no such list. The User-Agent tells you which software sent a request, or which software the sender chose to name. Changing it is one line of code, so the default User-Agent mostly identifies people who did not bother.
Handling it
Before blocking it, check whether you rely on it: your own scripts, uptime checks, webhooks and partner integrations often use tools like this. Then judge requests by what they do rather than the name they give:
- How fast they arrive, and from how many addresses
- Which paths they ask for, such as admin pages, config files or every product page in order
- Whether they touch links or fields no person could see
robots.txt does not help here: it only works on a client that chooses to read and obey it.
That is the approach WebDecoy takes: hidden decoy links and traps that only automated clients follow, so a request is judged by its behaviour, whatever User-Agent it sends.
Frequently Asked Questions
What is Browser-Use? +
Browser-Use is an ai agent, open-source software. Browser-Use AI browser automation framework.
Can I verify a request from Browser-Use? +
No. Browser-Use is software anyone can run, so there is no operator whose IP ranges or DNS names a request could be checked against. Its User-Agent says which software sent the request, or which software the sender chose to name, not who sent it.
Should I block Browser-Use? +
It depends on your site. Your own scripts, monitoring and integrations may use it. Blocking on the User-Agent alone only stops clients that leave the default in place, and anyone can change it. Judge requests by behaviour instead: request rate, the paths they ask for, and whether they touch pages no person would.
Catch automated clients, whatever they call themselves
WebDecoy sets hidden decoys only automated clients follow, and tracks them as persistent actors rather than disposable IPs.