Lemmy newb here, not sure if this is right for this /c.
An article I found from someone who hosts their own website and micro-social network, and their experience with web-scraping robots who refuse to respect robots.txt, and how they deal with them.
F04118F@feddit.nl 4 days ago
Interesting approach but looks like this ultimately ends up:
Anubis seems like a much better option, for those wanting to block bots without relying on Cloudflare:
anubis.techaro.lol
irotsoma@lemmy.blahaj.zone 4 days ago
Are there any guides to using it with reverse proxies like traefik? I’ve been wanting to try it out but haven’t had time to do the research yet.
HelloRoot@lemy.lol 4 days ago
github.com/TecharoHQ/anubis/issues/92