Bots lie about who they are, ignore robots.txt, and come from a gazillion different IPs.
Comment on AI crawlers cause Wikimedia Commons bandwidth demands to surge 50%.
catloaf@lemm.ee 5 days agoAn HTTP request is a request. Servers are free to rate limit or deny access
grysbok@lemmy.sdf.org 4 days ago
catloaf@lemm.ee 4 days ago
That’s what ddos protection is for.
taladar@sh.itjust.works 4 days ago
Rate limiting in itself requires resources that are not always available. For one thing you can only rate limit individuals you can identify so you need to keep data about past requests in memory and attach counters to them and even then that won’t help if the requests come from IPs that are easily changed.