It’s almost necessary at this point. At least some form of AI scraper prevention.
I had to take my public repos down a couple days ago, individuals and belligerents using botnets make blocking scrapers via normal means (user-agent/CIDR block) ineffective. So things like CloudFlare or Anubis are becoming necessary.
Aatube@lemmy.dbzer0.com 2 days ago
well it’s anti-LLM crawlers