Comment on How to combat large amounts of Ai scrapers
DrunkAnRoot@sh.itjust.works 5 weeks agobot hits i dont care my issue is when i see the same ip querying every file on 3 resource intensive sites millions of times
Comment on How to combat large amounts of Ai scrapers
DrunkAnRoot@sh.itjust.works 5 weeks agobot hits i dont care my issue is when i see the same ip querying every file on 3 resource intensive sites millions of times
daniskarma@lemmy.dbzer0.com 5 weeks ago
Do you have a proper robots.txt file?
Do they do weird things like invalid url, invalid post tries? Weird user agents?
Millions of times by the same ip sound much more like vulnerability proving than crawler.
DrunkAnRoot@sh.itjust.works 4 weeks ago
since its the frontends i run getting scraped its the robots.txt included there