Comment on Anubis is awesome! Stopping (AI)crawlbots
SheeEttin@lemmy.zip 4 weeks agoYes. There’s no real way to differentiate.
Comment on Anubis is awesome! Stopping (AI)crawlbots
SheeEttin@lemmy.zip 4 weeks agoYes. There’s no real way to differentiate.
SorteKanin@feddit.dk 4 weeks ago
Actually I think most search engine bots publish a list of verified IP addresses where they crawl from, so you could check the IP of a search bot against that to know.