Bots lie about who they are, ignore robots.txt, and come from a gazillion different IPs.
Comment on AI crawlers cause Wikimedia Commons bandwidth demands to surge 50%.
catloaf@lemm.ee 1 month agoAn HTTP request is a request. Servers are free to rate limit or deny access
grysbok@lemmy.sdf.org 1 month ago
catloaf@lemm.ee 1 month ago
That’s what ddos protection is for.
taladar@sh.itjust.works 1 month ago
Rate limiting in itself requires resources that are not always available. For one thing you can only rate limit individuals you can identify so you need to keep data about past requests in memory and attach counters to them and even then that won’t help if the requests come from IPs that are easily changed.