Comment on Anubis is awesome! Stopping (AI)crawlbots
dan@upvote.au 5 days ago
The Anubis site thinks my phone is a bot :/
tbh I would have just configured a reasonable rate limit in Nginx and left it at that.
Won’t the bots just hammer the API instead now?
Flipper@feddit.org 5 days ago
No. The rate limit doesn’t work as they use huge IP Spaces to crawl. Each IP alone is not bad they just use several thousand of them.
Using the API would assume some basic changes. We don’t do that here. If they wanted that, they could run their own instance and would even get notified about changes. No crawling required at all.