Submitted 3 weeks ago by Tea@programming.dev to technology@lemmy.world
https://status.sr.ht/issues/2025-03-17-git.sr.ht-llms/
SourceHut continues to face disruptions due to aggressive LLM crawlers. We are continuously working to deploy mitigations. We have deployed a number of mitigations which are keeping the problem contained for now. However, some of our mitigations may impact end-users.
I wonder how much of the load problems I observe with lemmy.world are due to AI crawlers.
The companies that run these residential proxy networks are sketchy as shit and in a better world would be criminally prosecuted. They’re tricking random low-information users into installing VPNs and other software with backdoors that turn them into a veritable botnet.
thatsnothowyoudoit@lemmy.ca 3 weeks ago
We 444 every LLM crawler we see.
mesamunefire@lemmy.world 3 weeks ago
I haved the same issue. OpenAI was just slamming my tiny little server, ignoring the robots.txt. I had to install a LLM black hole and put a very basic password protection around my git server, since it kept getting slammed by the crawler.
As much as I dont like google, I did see them come in, look at the robot.txt and no other calls for a week. Thats how it should work.