Selenium
Headless browser · Open source
Headless browser
Type
Open source
Made by
No
Verifiable
None
robots.txt token
What Selenium is
Selenium WebDriver automation.
It has no robots.txt token. Documentation: project site .
Why it cannot be verified
Crawlers run by a company can be checked: the company publishes the addresses or DNS names its crawler uses, and a request from anywhere else is not really it. Selenium is software anyone can run, so there is no such list. The User-Agent tells you which software sent a request, or which software the sender chose to name. Changing it is one line of code, so the default User-Agent mostly identifies people who did not bother.
Handling it
Before blocking it, check whether you rely on it: your own scripts, uptime checks, webhooks and partner integrations often use tools like this. Then judge requests by what they do rather than the name they give:
- How fast they arrive, and from how many addresses
- Which paths they ask for, such as admin pages, config files or every product page in order
- Whether they touch links or fields no person could see
robots.txt does not help here: it only works on a client that chooses to read and obey it.
That is the approach WebDecoy takes: hidden decoy links and traps that only automated clients follow, so a request is judged by its behaviour, whatever User-Agent it sends.
Frequently Asked Questions
What is Selenium? +
Selenium is a headless browser, open-source software. Selenium WebDriver automation.
Can I verify a request from Selenium? +
No. Selenium is software anyone can run, so there is no operator whose IP ranges or DNS names a request could be checked against. Its User-Agent says which software sent the request, or which software the sender chose to name, not who sent it.
Should I block Selenium? +
It depends on your site. Your own scripts, monitoring and integrations may use it. Blocking on the User-Agent alone only stops clients that leave the default in place, and anyone can change it. Judge requests by behaviour instead: request rate, the paths they ask for, and whether they touch pages no person would.
Related tools
- aiohttp Open Source
- ARTEX Open Source
- Axios Open Source
- BeautifulSoup Open Source
- Browser-Use Open Source
- Colly Open Source
- CommaFeed Open Source
- cURL Open Source
- Faraday Open Source
- FreshRSS Open Source
- Got Open Source
- GuzzleHttp Open Source
Catch automated clients, whatever they call themselves
WebDecoy sets hidden decoys only automated clients follow, and tracks them as persistent actors rather than disposable IPs.