Comment on Cloudflare blocking AI crawlers
daniskarma@lemmy.dbzer0.com 1 week ago
How does it differentiate an “AI crawler”, from any other crawler? Search engine crawler? Someone monitoring data to offer statistics? Archiving?
This is not good. They are most likely doing the crawling themselves and them selling the data to the best bidder. That bidder could obviously be openAI for all we know.
They just know that introducing the sentence “this is anti AI” a lot of people is not going to question anything.
_cryptagion@lemmy.dbzer0.com 1 week ago
Well, they have access to logs showing who connects to 24 million websites, how they use those websites, and for how long. So if there’s anyone who knows what traffic is crawlers, and which crawlers are AI, it’s Cloudflare. There’s no way they wouldn’t know, they have all the data they would ever need to figure it out. In fact, there’s nobody on the internet who is better positioned to be able to identify AI crawlers than Cloudflare.
x00z@lemmy.world 1 week ago
This.
They also have a form to submit AI crawlers.
CloudFlare can also easily maintain an anti AI crawler service completely by itself if it takes a fee on top of their pay per crawl functionality. However, considering CloudFlare already has all the tools and infrastructure to do this cheaply, providing a good service wouldn’t be too hard.