Comment on AI companies are violating a basic social contract of the web and and ignoring robots.txt
lvxferre@mander.xyz 7 months agoYup, it’s the same approach as email spam traps. Except the naughty list, but… holy fuck that’s a great idea, if people can share their bot IP banlists with each other the damage to those web crawling businesses would be even bigger.
Nighed@sffa.community 7 months ago
but with all of the cloud resources now, you can switch through IP addresses without any trouble. hell, you could just browse by IP6 and not even worry with how cheap those are!
lvxferre@mander.xyz 7 months ago
Yeah, that throws a monkey wrench into the idea. That’s a shame, because “either respect robots.txt or you’re denied access to a lot of websites!” is appealing.
Nighed@sffa.community 7 months ago
That’s when Google’s browser DRM thing starts sounding like a good idea 😭