Plus even if they were to implement those features, the challenges would still get increasingly harder the more bot-like a scraper behaves.
You can’t prevent scraping entirely but you can certainly prevent scraping that behaves like a DOS attack.
Comment on Anubis is awesome! Stopping (AI)crawlbots
Sekoia@lemmy.blahaj.zone 5 days agoThey can, but it’s not trivial. The challenge uses a bunch of modern browser features that these scrapers don’t use, regarding metadata and compression and a few other things. Things that are annoying to implement and not worth the effort. Check the recent discussion on lobste.rs if you’re interested in the exact details.
yetAnotherUser@discuss.tchncs.de 4 days ago
baod_rate@programming.dev 3 days ago
For those coming from the future: lobste.rs/…/anubis_now_supports_non_js_challenges