@Ninjazzon@infosec.pub please add the required [Opinion] prefix.
AI crawlers destroying websites in hunger for content
Submitted 7 months ago by Ninjazzon@infosec.pub to technology@lemmy.zip
https://www.theregister.com/2025/08/29/ai_web_crawlers_are_destroying/
Comments
BrikoX@lemmy.zip 7 months ago
fuckwit_mcbumcrumble@lemmy.dbzer0.com 7 months ago
Moreover, AI crawlers are much more aggressive than standard crawlers. As the InMotionhosting web hosting company notes, they also tend to disregard crawl delays or bandwidth-saving guidelines and extract full page text, and sometimes attempt to follow dynamic links or scripts.
So they’re just lazily programmed crawlers. Ironically trying to block them can cause web traffic to go up not down when people use more advanced methods to get around blocking. When you switch from a simple wget command ripping the bare page to a full blown chrome browser loading all the pictures, JS, and other junk that shit adds up.
Catoblepas@piefed.blahaj.zone 7 months ago
The result? If you're using a shared server for your website, as many small businesses do, even if your site isn't being shaken down for content, other sites on the same hardware with the same Internet pipe may be getting hit. This means your site's performance drops through the floor even if an AI crawler isn't raiding your website.
Wow, what innovation! Give these companies a trillion dollars!!
belated_frog_pants@beehaw.org 7 months ago
That a fuckin ai image on an AI negative piece? God i hate AI and how everyone makes excuses to use it
MrTolkinghoen@lemmy.zip 7 months ago
Sounds like a feature not a bug. Steal the content and take away the source of the content? Win win. Then the content can only be obtained via the AI.