Search engines been going relatively fine for decades now. But the crawlers from AI companies basically DDOS hosts in comparison, sending so many requests in such a short interval. Crawling dynamic links as well that are expensive to render compared to a static page, ignoring the robots.txt entirely, or even using it discover unlinked pages.
Servers have finite resources, especially self hosted sites, while AI companies have disproportinately more at their disposal, easily grinding other systems to a halt by overwhelming them with requests.
pr06lefs@lemmy.ml 7 months ago
If a neighborhood is beset by roving bands of thieves, sooner or later strangers will be greeted by a shotgun rather than an invitation to tea, regardless of their intentions. Them’s the breaks. Bots are going to take a hit now and their operators are just going to have to deal with it. Sucks when people don’t play nice, but this is what you get.
FauxLiving@lemmy.world 7 months ago
I’m sure people that are attempting to drive to their house in a new vehicle wouldn’t appreciate being riddled with bullets because the neighborhood watch makes no attempt to distinguish between thieves and homeowners.
pr06lefs@lemmy.ml 7 months ago
So sad for them. Try not living in a war zone?
FauxLiving@lemmy.world 7 months ago
It isn’t a war zone, it’s a gated community where the guards have suddenly decided that any vehicle made after 2020 is full of thieves without bothering to consult the residents or giving them the ability to opt out of having their dinner guests murdered for driving a vehicle the security guards don’t like.