Comment on Aggressive AI scrapers are making it kinda suck to run wikis
thatsnothowyoudoit@lemmy.ca 3 weeks ago
We use NGINX’s 444 response A LOT.
In coordination with careful rate-limiting, it’s been a dramatic improvement.
The worst of the bots don’t advertise their User Agent (or worse, attempt to present they’re a normal user making 100s of requests a second) but there’s lots of low hanging fruit.
grrgyle@slrpnk.net 3 weeks ago
Hmm interesting. I wasn’t aware of this one