Comment

Comment on Anubis is awesome! Stopping (AI)crawlbots

This all appears to be based on the user agent, so wouldn’t that mean that bad-faith scrapers could just declare themselves to be typical search engine user agent?

source

Sort:hotnew top

SorteKanin@feddit.dk ⁨3⁩ ⁨days⁩ ago
Most search engine bots publish a list of verified IP addresses where they crawl from, so you could check the IP of a search bot against that to know.

source
SheeEttin@lemmy.zip ⁨3⁩ ⁨days⁩ ago
Yes. There’s no real way to differentiate.

source
- SorteKanin@feddit.dk ⁨3⁩ ⁨days⁩ ago
  Actually I think most search engine bots publish a list of verified IP addresses where they crawl from, so you could check the IP of a search bot against that to know.
  
  source