Comment on Anubis is awesome! Stopping (AI)crawlbots

• dan@upvote.au ⁨4⁩ ⁨days⁩ ago

  Won’t the bots just switch to using that instead of the heavier JS challenge?

  source

  • Sekoia@lemmy.blahaj.zone ⁨4⁩ ⁨days⁩ ago

    They can, but it’s not trivial. The challenge uses a bunch of modern browser features that these scrapers don’t use, regarding metadata and compression and a few other things. Things that are annoying to implement and not worth the effort. Check the recent discussion on lobste.rs if you’re interested in the exact details.

    source

    • baod_rate@programming.dev ⁨3⁩ ⁨days⁩ ago

      Check the recent discussion on lobste.rs if you’re interested in the exact details.

      For those coming from the future: lobste.rs/…/anubis_now_supports_non_js_challenges

      source
    • yetAnotherUser@discuss.tchncs.de ⁨3⁩ ⁨days⁩ ago

      Plus even if they were to implement those features, the challenges would still get increasingly harder the more bot-like a scraper behaves.

      You can’t prevent scraping entirely but you can certainly prevent scraping that behaves like a DOS attack.

      source