The amount of stupid AI scraping behavior I see even on my small websites is ridiculous, they’ll endlessly pound identical pages as fast as possible over an entire week, apparently not even checking if the contents changed. Probably some vibe coded shit that barely functions.
Comment on AI crawlers cause Wikimedia Commons bandwidth demands to surge 50%.
thisbenzingring@lemmy.sdf.org 1 year ago
what assholes … just fucking download the full package and quit hitting the URL
Glitchvid@lemmy.world 1 year ago
gravitas_deficiency@sh.itjust.works 1 year ago
If I was running infra for them, I’d just start blacklisting abusive IPs without warning
cm0002@lemmy.world 1 year ago
Right‽ This is ridiculously stupid when you can download the entirety of Wikipedia in a single package and parse it to your hearts desire
ChaoticCookie@sh.itjust.works 1 year ago
Yay interrobang :D
TheTechnician27@lemmy.world 1 year ago
Not only that, but we make it goddamn trivial. Doing this is just stealing without attribution like the CC BY-SA 4.0 license demands and then on top of that kicking down the ladder for people who actually want to use Wikipedia and not the hallucinatory slop you’re trying to supplant it with. LLM companies have caused incalculable damage to critical thinking, the open web, and the climate.
XTL@sopuli.xyz 1 year ago
Scraper bots don’t read instructions, they just follow links