Comment on Codeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.

zifk@sh.itjust.works ⁨7⁩ ⁨months⁩ ago

• randomblock1@lemmy.world ⁨7⁩ ⁨months⁩ ago

  No, it’s expensive to comply (at a massive scale), but easy to avoid. Just change the user agent. There’s even a dedicated extension for bypassing Anubis.

  Even then AI servers have plenty of compute, it realistically doesn’t cost much. Maybe like a thousandth of a cent per solve? They’re spending billions on GPU power, they don’t care.

  I’ve been saying this since day 1 of Anubis but nobody wants to hear it.

  source

  • T156@lemmy.world ⁨7⁩ ⁨months⁩ ago

    The website would also have to display to users at the end of the day. It’s a similar problem as trying to solve media piracy. Worst comes to it, the crawlers could read the page like a person would.

    source

    • acockworkorange@mander.xyz ⁨7⁩ ⁨months⁩ ago

      You could have a server for open code access with very limited bandwidth and another for authenticated users with higher bandwidth.

      source
• sudo@programming.dev ⁨7⁩ ⁨months⁩ ago

  This is what I’ve kept saying about POW being a shit bot management tactic. Its a flat tax across all users, real or fake. The fake users are getting making money to access your site and will just eat the added expense. You can raise the tax to cost more than what your data is worth to them, but that also affects your real users. Nothing about Anubis even attempts to differentiate between bots and real users.

  If the bots take the time, they can set up a pipeline to solve Anubis tokens outside of the browser more efficiently than real users.

  source

  • OpenPassageways@lemmy.zip ⁨7⁩ ⁨months⁩ ago

    What the alternative?

    source

    • sudo@programming.dev ⁨7⁩ ⁨months⁩ ago

      Not much for open source solutions. A simple captcha however would cost scrapers more to crack than Anubis.

      But when it comes to “real” bot management solutions: The least invasive solutions will try to match User-Agent and other headers against the TLS fingerprint and block if they don’t match. More invasive solutions will fingerprint your browser and even your GPU, then either block you or issue you a tracking cookie which is often pinned to your IP and user-agent. Both of those solutions require a large base of data to know what real and fake traffic actually looks like. Only large hosting providers like CloudFlare and Akamai have that data and can provide those sorts of solutions.

      source
  • black_flag@lemmy.dbzer0.com ⁨7⁩ ⁨months⁩ ago

    Yeah but ai companies are losing money so in the long run Anubis seems like it should eventually return to working.

    source

    • sudo@programming.dev ⁨7⁩ ⁨months⁩ ago

      Costs of solving PoW for Anubis is absolutely not a factor in any AI companies budget. Just the costs of answering one question is millions of times more expensive than running sha256sum for Anubis.

      Just in case you’re being glib and mean the businesses will go under regardless of Anubis: most of these are coming from China. China is absolutely will keep running these companies at a loss for the sake of strategic development.

      source

      • black_flag@sh.itjust.works ⁨7⁩ ⁨months⁩ ago

        Thanks for the info 👍 would not have thought Anubis would be so irrelevant

        source