Comment

Comment on Cloudflare CEO warns AI and zero-click internet are killing the web's business model

But with the rise of AI, the dynamic is changing: We are observing a significant increase in request volume, with most of this traffic being driven by scraping bots collecting training data for large language models (LLMs) and other use cases. Automated requests for our content have grown exponentially, alongside the broader technology economy, via mechanisms including scraping, APIs, and bulk downloads. This expansion happened largely without sufficient attribution, which is key to drive new users to participate in the movement, and is causing a significant load on the underlying infrastructure that keeps our sites available for everyone.

- diff.wikimedia.org/…/how-crawlers-impact-the-oper…

source

Sort:hotnew top

pinkapple@lemmy.ml ⁨5⁩ ⁨weeks⁩ ago

via mechanisms including scraping, APIs, and bulk downloads.

Omg exactly! Thanks. Yet nothing about having to use logins to stop bots because that kinda isn’t a thing when you already provide data dumps and an API to wikimedia commons.

While undergoing a migration of our systems, we noticed that only a fraction of the expensive traffic hitting our core datacenters was behaving how web browsers would usually do, interpreting javascript code. When we took a closer look, we found out that at least 65% of this resource-consuming traffic we get for the website is coming from bots, a disproportionate amount given the overall pageviews from bots are about 35% of the total.

Source for traffic being scraping data for training models: they’re blocking javascript therefore bots therefore crawlers, just trust me bro.

source