Comment on Anubis is awesome! Stopping (AI)crawlbots
SorteKanin@feddit.dk 4 days agoMost search engine bots publish a list of verified IP addresses where they crawl from, so you could check the IP of a search bot against that to know.
Comment on Anubis is awesome! Stopping (AI)crawlbots
SorteKanin@feddit.dk 4 days agoMost search engine bots publish a list of verified IP addresses where they crawl from, so you could check the IP of a search bot against that to know.