Comment on Cloudflare blocking AI crawlers

<- View Parent
_cryptagion@lemmy.dbzer0.com ⁨2⁩ ⁨days⁩ ago

Well, they have access to logs showing who connects to 24 million websites, how they use those websites, and for how long. So if there’s anyone who knows what traffic is crawlers, and which crawlers are AI, it’s Cloudflare. There’s no way they wouldn’t know, they have all the data they would ever need to figure it out. In fact, there’s nobody on the internet who is better positioned to be able to identify AI crawlers than Cloudflare.

source
Sort:hotnewtop