Comment on Cloudflare CEO warns AI and zero-click internet are killing the web's business model
TheOneCurly@lemm.ee 6 days agoLmao you linked to the same page I did where this text appears:
GPTBot is used to make our generative AI foundation models more useful and safe. It is used to crawl content that may be used in training our generative AI foundation models.
Also you’re so capitalism brained you assume anyone running a website must be doing so for profit. My hobby projects (personal homepage and personal git forge) were getting slammed by bots while I just paid the bills. I could have locked them both behind an auth portal but then I might as well just take them off the internet and run everything on my LAN.
pinkapple@lemmy.ml 6 days ago
So far none of your ramblings disproves what I said. Yeah there are crawlers for niche collecting probably, nobody crawls the entire internet when they can use the weekly updated common crawl. Unless you or anyone else has access to unknown internal openAI policies on why they intentionally reinvent the wheel, your fake anecdotes (lol bots literally telling you they’re going to use scraping for training in the user agent) don’t cut it. You’re probably seeing search bots.
If you didn’t care for ad money and search engine exposure bozo you’d block everything in robots.txt and be done instead of whining about specific bots you don’t like.
You didn’t link to this but go on take their IPs json files and block them.