Scrapy
Scraping library · Open source
Scraping library
Type
Open source
Made by
No
Verifiable
Scrapy
robots.txt token
What Scrapy is
Scrapy web scraping framework.
Its robots.txt token is Scrapy , and it is listed as honouring robots.txt. Documentation: project site .
Why it cannot be verified
Crawlers run by a company can be checked: the company publishes the addresses or DNS names its crawler uses, and a request from anywhere else is not really it. Scrapy is software anyone can run, so there is no such list. The User-Agent tells you which software sent a request, or which software the sender chose to name. Changing it is one line of code, so the default User-Agent mostly identifies people who did not bother.
Handling it
Before blocking it, check whether you rely on it: your own scripts, uptime checks, webhooks and partner integrations often use tools like this. Then judge requests by what they do rather than the name they give:
- How fast they arrive, and from how many addresses
- Which paths they ask for, such as admin pages, config files or every product page in order
- Whether they touch links or fields no person could see
It is listed as honouring robots.txt by default, so a rule asks well-configured copies to stay away. Anyone running it can turn that off.
User-agent: Scrapy
Disallow: /That is the approach WebDecoy takes: hidden decoy links and traps that only automated clients follow, so a request is judged by its behaviour, whatever User-Agent it sends.
Frequently Asked Questions
What is Scrapy? +
Scrapy is a scraping library, open-source software. Scrapy web scraping framework.
Can I verify a request from Scrapy? +
No. Scrapy is software anyone can run, so there is no operator whose IP ranges or DNS names a request could be checked against. Its User-Agent says which software sent the request, or which software the sender chose to name, not who sent it.
Should I block Scrapy? +
It depends on your site. Your own scripts, monitoring and integrations may use it. Blocking on the User-Agent alone only stops clients that leave the default in place, and anyone can change it. Judge requests by behaviour instead: request rate, the paths they ask for, and whether they touch pages no person would.
Related tools
- aiohttp Open Source
- Apache-HttpClient Apache
- Axios Open Source
- BeautifulSoup Open Source
- Colly Open Source
- Faraday Open Source
- Go-HTTP-Client Go
- Got Open Source
- GuzzleHttp Open Source
- HTTPX Open Source
- Java Oracle
- Jsoup Open Source
Catch automated clients, whatever they call themselves
WebDecoy sets hidden decoys only automated clients follow, and tracks them as persistent actors rather than disposable IPs.