Some of them are at least honest and have it as a user agent.
Comment on How to combat large amounts of Ai scrapers
daniskarma@lemmy.dbzer0.com 10 months ago
How do you know it’s “AI” scrappers?
I’ve have my server up before AI was a thing.
It’s totally normal to get thousands of bot hits and to get scraped.
Sheldan@lemmy.world 10 months ago
krakenfury@lemmy.sdf.org 10 months ago
Is ignoring robots.txt considered “honest”?
Sheldan@lemmy.world 10 months ago
That’s not what I was talking about
DrunkAnRoot@sh.itjust.works 10 months ago
bot hits i dont care my issue is when i see the same ip querying every file on 3 resource intensive sites millions of times
daniskarma@lemmy.dbzer0.com 10 months ago
Do you have a proper robots.txt file?
Do they do weird things like invalid url, invalid post tries? Weird user agents?
Millions of times by the same ip sound much more like vulnerability proving than crawler.
DrunkAnRoot@sh.itjust.works 10 months ago
since its the frontends i run getting scraped its the robots.txt included there