Comment on Anubis is awesome and I want to talk aout it
JustTesting@lemmy.hogru.ch 12 hours agoThis is the way. I also have rules for hits to url, without a referer, that should never be hit without a referer, with some threshold to account for a user hitting F5. Plus a whitelist of real users (ones that got a 200 on a login endpoint).
then there’s ratelimiting and banning ip’s that hit the ratelimit regularly.
Dowloading abuse ip lists nightly and banning those, that’s around 60k abusive ip’s gone. At that point you probably need to use nftables though, for the sets, as having 60k rules would be a bad idea.
there’s lists of all datacenter ip ranges out there, so you could block as well, though that’s a pretty nuclear option, so better make sure traffic you want is whitelisted. E.g. for lemmy, you can get a list of the ips of all other instances nightly, so you don’t accidentally block them. Lemmy traffic is very spammy…
there’s so much that can be done with f2b and a bit of scripting/writing filters
iopq@lemmy.world 6 hours ago
Can’t you just bookmark the page?
JustTesting@lemmy.hogru.ch 6 hours ago
You mean for the referer part? Of course you don’t want it for all urls and there’s some legitimate cases. I have that on specific urls where it’s highly unlikely, not every url. E.g. a direct link to a single comment in lemmy, plus whitelisting logged-in users. Plus a limit, like >3 times an hour before a ban.
It’s a pretty consistent bot pattern, they will go to some subsubpage with no referer with no prior traffic from that it, and then no other traffic from that ip after that for a bit (since they cycle though ip’s on each request) but you will get a ton of these requests across all ips they use. It was one of the most common patterns i saw when i followed the logs for a while.