Rate limiting in itself requires resources that are not always available. For one thing you can only rate limit individuals you can identify so you need to keep data about past requests in memory and attach counters to them and even then that won’t help if the requests come from IPs that are easily changed.
Comment on AI crawlers cause Wikimedia Commons bandwidth demands to surge 50%.
catloaf@lemm.ee 1 year agoAn HTTP request is a request. Servers are free to rate limit or deny access
taladar@sh.itjust.works 1 year ago
grysbok@lemmy.sdf.org 1 year ago
Bots lie about who they are, ignore robots.txt, and come from a gazillion different IPs.
catloaf@lemm.ee 1 year ago
That’s what ddos protection is for.