Comment on Anubis is awesome! Stopping (AI)crawlbots
SheeEttin@lemmy.zip 2 months agoYes. There’s no real way to differentiate.
Comment on Anubis is awesome! Stopping (AI)crawlbots
SheeEttin@lemmy.zip 2 months agoYes. There’s no real way to differentiate.
SorteKanin@feddit.dk 2 months ago
Actually I think most search engine bots publish a list of verified IP addresses where they crawl from, so you could check the IP of a search bot against that to know.