Comment on Anubis is awesome! Stopping (AI)crawlbots

<- View Parent
Flipper@feddit.org ⁨4⁩ ⁨days⁩ ago

No. The rate limit doesn’t work as they use huge IP Spaces to crawl. Each IP alone is not bad they just use several thousand of them.

Using the API would assume some basic changes. We don’t do that here. If they wanted that, they could run their own instance and would even get notified about changes. No crawling required at all.

source
Sort:hotnewtop