iirc there’s instructions on completing the anubis challenge manually
Comment on Anubis is awesome! Stopping (AI)crawlbots
e0qdk@reddthat.com 5 days ago
I don’t like Anubis because it requires me to enable JS – making me less secure. reddthat started using go-away recently as an alternative that doesn’t require JS when we were getting hammered by scrapers.
Jumuta@sh.itjust.works 4 days ago
BakedCatboy@lemmy.ml 5 days ago
Fwiw Anubis is adding a nojs meta refresh challenge that if it doesn’t have issues will soon be the new default challenge
dan@upvote.au 5 days ago
Won’t the bots just switch to using that instead of the heavier JS challenge?
Sekoia@lemmy.blahaj.zone 5 days ago
They can, but it’s not trivial. The challenge uses a bunch of modern browser features that these scrapers don’t use, regarding metadata and compression and a few other things. Things that are annoying to implement and not worth the effort. Check the recent discussion on lobste.rs if you’re interested in the exact details.
baod_rate@programming.dev 3 days ago
For those coming from the future: lobste.rs/…/anubis_now_supports_non_js_challenges
yetAnotherUser@discuss.tchncs.de 3 days ago
Plus even if they were to implement those features, the challenges would still get increasingly harder the more bot-like a scraper behaves.
You can’t prevent scraping entirely but you can certainly prevent scraping that behaves like a DOS attack.