Comment on How to combat large amounts of Ai scrapers
Sheldan@lemmy.world 4 weeks agoSome of them are at least honest and have it as a user agent.
Comment on How to combat large amounts of Ai scrapers
Sheldan@lemmy.world 4 weeks agoSome of them are at least honest and have it as a user agent.
krakenfury@lemmy.sdf.org 4 weeks ago
Is ignoring robots.txt considered “honest”?
Sheldan@lemmy.world 4 weeks ago
That’s not what I was talking about