Comment on How to combat large amounts of Ai scrapers
Sheldan@lemmy.world 4 days agoSome of them are at least honest and have it as a user agent.
Comment on How to combat large amounts of Ai scrapers
Sheldan@lemmy.world 4 days agoSome of them are at least honest and have it as a user agent.
krakenfury@lemmy.sdf.org 4 days ago
Is ignoring robots.txt considered “honest”?
Sheldan@lemmy.world 4 days ago
That’s not what I was talking about