Comment on Anubis is awesome! Stopping (AI)crawlbots

• Sekoia@lemmy.blahaj.zone ⁨1⁩ ⁨day⁩ ago

  They can, but it’s not trivial. The challenge uses a bunch of modern browser features that these scrapers don’t use, regarding metadata and compression and a few other things. Things that are annoying to implement and not worth the effort. Check the recent discussion on lobste.rs if you’re interested in the exact details.

  source

  • baod_rate@programming.dev ⁨3⁩ ⁨hours⁩ ago

    Check the recent discussion on lobste.rs if you’re interested in the exact details.

    For those coming from the future: lobste.rs/…/anubis_now_supports_non_js_challenges

    source
  • yetAnotherUser@discuss.tchncs.de ⁨7⁩ ⁨hours⁩ ago

    Plus even if they were to implement those features, the challenges would still get increasingly harder the more bot-like a scraper behaves.

    You can’t prevent scraping entirely but you can certainly prevent scraping that behaves like a DOS attack.

    source