If this were true (which is nearly impossible since you said “all"), stuff like Anubis wouldn’t exist since you could just toss up a crowd-sourced robots.txt and be done with it.
Comment on A Project to Poison LLM Crawlers
Lembot_0006@programming.dev 3 weeks agoAll crawlers ignore robots text since the very start. Anyway, if THAT is the problem then IT is a problem, not the LLMs as a whole.
arcterus@piefed.blahaj.zone 2 weeks ago
Lembot_0006@programming.dev 2 weeks ago
I don’t think the word “ignore” means what you think it means :)
FauxLiving@lemmy.world 3 weeks ago
You can tell when you’re talking with someone who has been given the position of ‘AI Bad’, but doesn’t actually understand the moral positions or technological details that form the foundation of that argument by how confidently they repeat some detail that is clearly nonsense to anybody with knowledge of the subject.