Comment on Cloudflare blocking AI crawlers
3dcadmin@lemmy.relayeasy.com 5 weeks ago
Seen plenty of people who think this is a bad thing, do they just want everything to be crawled. I mean I don’t think this is the saviour but it has got to be better than wholesale theft
GenderNeutralBro@lemmy.sdf.org 5 weeks ago
Yes. Web crawling has been a normal and vital part of the web from day 1. We’d have no search engines without crawlers.
The web is user-centric by design. I’m sick of tech companies trying to flip the script and hoard information, most of which is not theirs to begin with (e.g. Google, Reddit, Twitter, Facebook, etc.).
Prontomomo@lemmy.world 5 weeks ago
I don’t think this blocks crawlers. About 1/5 websites uses cloudflare, the significant thing here’s is that AI scraping is now blocked by default on most of those sites, NOT crawling