The amount of stupid AI scraping behavior I see even on my small websites is ridiculous, they’ll endlessly pound identical pages as fast as possible over an entire week, apparently not even checking if the contents changed. Probably some vibe coded shit that barely functions.
Comment on AI crawlers cause Wikimedia Commons bandwidth demands to surge 50%.
thisbenzingring@lemmy.sdf.org 10 months ago
what assholes … just fucking download the full package and quit hitting the URL
Glitchvid@lemmy.world 10 months ago
gravitas_deficiency@sh.itjust.works 10 months ago
If I was running infra for them, I’d just start blacklisting abusive IPs without warning
XTL@sopuli.xyz 10 months ago
Scraper bots don’t read instructions, they just follow links
cm0002@lemmy.world 10 months ago
Right‽ This is ridiculously stupid when you can download the entirety of Wikipedia in a single package and parse it to your hearts desire
TheTechnician27@lemmy.world 10 months ago
Not only that, but we make it goddamn trivial. Doing this is just stealing without attribution like the CC BY-SA 4.0 license demands and then on top of that kicking down the ladder for people who actually want to use Wikipedia and not the hallucinatory slop you’re trying to supplant it with. LLM companies have caused incalculable damage to critical thinking, the open web, and the climate.
ChaoticCookie@sh.itjust.works 10 months ago
Yay interrobang :D