Comment on How to combat large amounts of Ai scrapers

<- View Parent
daniskarma@lemmy.dbzer0.com ⁨4⁩ ⁨days⁩ ago

Do you have a proper robots.txt file?

Do they do weird things like invalid url, invalid post tries? Weird user agents?

Millions of times by the same ip sound much more like vulnerability proving than crawler.

source
Sort:hotnewtop