Comment on LLM crawlers continue to DDoS SourceHut.

thatsnothowyoudoit@lemmy.ca ⁨1⁩ ⁨year⁩ ago

• mesamunefire@lemmy.world ⁨1⁩ ⁨year⁩ ago

  I haved the same issue. OpenAI was just slamming my tiny little server, ignoring the robots.txt. I had to install a LLM black hole and put a very basic password protection around my git server, since it kept getting slammed by the crawler.

  As much as I dont like google, I did see them come in, look at the robot.txt and no other calls for a week. Thats how it should work.

  source