Apparently the dump doesn’t include media, though there’s ongoing discussion within wikimedia about changing that. It also seems likely to me that AI scrapers don’t care about externalizing costs onto others if it might mean a competitive advantage.
Comment on AI crawlers cause Wikimedia Commons bandwidth demands to surge 50%.
Mubelotix@jlai.lu 1 month ago
Doesn’t make any sense. Why would you crawl wikipedia when you can just download a dump as a torrent ?
mke@programming.dev 1 month ago
cabillaud@lemmy.world 1 month ago
To have the most recent data?
umbraroze@slrpnk.net 1 month ago
To just have the most recent data within reasonable time frame is one thing. AI companies are like “I must have every single article within 5 minutes they get updated, or I’ll throw my pacifier out of the pram”. No regard for the considerations of the source sites.
Kolrami@lemmy.world 1 month ago
There’s a chance this isn’t being done by someone who only wants Wikipedia’s data. As the amount of websites you scrape increases, your desire to use the easy tools loses out to creating the most general tool that can look at most webpages.
ChairmanMeow@programming.dev 1 month ago
AI bros aren’t that smart.