Feed reader

Type

Open source

Made by

No

Verifiable

FreshRSS

robots.txt token

What FreshRSS is

FreshRSS self-hosted reader.

Its robots.txt token is FreshRSS , and it is listed as honouring robots.txt. Documentation: project site .

Why it cannot be verified

Crawlers run by a company can be checked: the company publishes the addresses or DNS names its crawler uses, and a request from anywhere else is not really it. FreshRSS is software anyone can run, so there is no such list. The User-Agent tells you which software sent a request, or which software the sender chose to name. Changing it is one line of code, so the default User-Agent mostly identifies people who did not bother.

Handling it

Before blocking it, check whether you rely on it: your own scripts, uptime checks, webhooks and partner integrations often use tools like this. Then judge requests by what they do rather than the name they give:

  • How fast they arrive, and from how many addresses
  • Which paths they ask for, such as admin pages, config files or every product page in order
  • Whether they touch links or fields no person could see

It is listed as honouring robots.txt by default, so a rule asks well-configured copies to stay away. Anyone running it can turn that off.

User-agent: FreshRSS
Disallow: /

That is the approach WebDecoy takes: hidden decoy links and traps that only automated clients follow, so a request is judged by its behaviour, whatever User-Agent it sends.

Frequently Asked Questions

What is FreshRSS? +

FreshRSS is a feed reader, open-source software. FreshRSS self-hosted reader.

Can I verify a request from FreshRSS? +

No. FreshRSS is software anyone can run, so there is no operator whose IP ranges or DNS names a request could be checked against. Its User-Agent says which software sent the request, or which software the sender chose to name, not who sent it.

Should I block FreshRSS? +

It depends on your site. Your own scripts, monitoring and integrations may use it. Blocking on the User-Agent alone only stops clients that leave the default in place, and anyone can change it. Judge requests by behaviour instead: request rate, the paths they ask for, and whether they touch pages no person would.

Related tools

← All crawlers

Catch automated clients, whatever they call themselves

WebDecoy sets hidden decoys only automated clients follow, and tracks them as persistent actors rather than disposable IPs.

Start Free Trial