Comment on AI crawlers cause Wikimedia Commons bandwidth demands to surge 50%.

<- View Parent
Kolrami@lemmy.world ⁨2⁩ ⁨weeks⁩ ago

There’s a chance this isn’t being done by someone who only wants Wikipedia’s data. As the amount of websites you scrape increases, your desire to use the easy tools loses out to creating the most general tool that can look at most webpages.

source
Sort:hotnewtop