Anubis is awesome! Stopping (AI)crawlbots

Comments

• ikidd@lemmy.world ⁨1⁩ ⁨day⁩ ago

  Something that’s less annoying than Anubis is fail2ban tarpitting the scrapers by putting in a hidden honeypot page link that they follow, and adding the followers to fail2ban.

  petermolnar.net/…/anti-ai-nepenthes-fail2ban/

  source

  • N0x0n@lemmy.ml ⁨14⁩ ⁨hours⁩ ago

    Wow, what a combo ! I guess this would reduce the tarpit’s overall power consumption?

    I haven’t looked at your link yet and maybe it already contains my answer, but I wish to customize for how long they are traped into the tarpit before fail2ban kicks in so I can still poison teir AI while saving alot of ressources !!

    source
• Mora@pawb.social ⁨1⁩ ⁨day⁩ ago

  Besides that point: why tf do they even crawl lemmy. They could just as well create a “read only” instance with an accoubt that subsribes to all communities … and the other instances would send their data. Oh, right, AI has to be as unethical as possible for most companies for some reason.

  source

  • wizardbeard@lemmy.dbzer0.com ⁨1⁩ ⁨day⁩ ago

    They crawl wikipedia too, and are adding significant extra load on their servers, even though Wikipedia has a regularly updated torrent to download all its content.

    source
  • ZombiFrancis@sh.itjust.works ⁨1⁩ ⁨day⁩ ago

    See your brain went immediately to a solution based on knowing how something works. That’s not in the AI wheelhouse.

    source
  • dan@upvote.au ⁨22⁩ ⁨hours⁩ ago

    They’re likely not intentionally crawling Lemmy. They’re probably just crawling all sites they can find.

    source
  • AmbitiousProcess@piefed.social ⁨23⁩ ⁨hours⁩ ago

    Because the easiest solution for them is a simple web scraper. If they don't give a shit about ethics, then something that just crawls every page it can find is loads easier for them to set up than a custom implementation to get torrent downloads for wikipedia, making lemmy/mastodon/pixelfed instances for the fediverse, using rss feeds and checking if they have full or only partial articles, implementing proper checks to prevent double (or more) downloading of the same content, etc.

    source
• reddeadhead@awful.systems ⁨16⁩ ⁨hours⁩ ago

  Anubis just released the no-JS challenge in a update anubis.techaro.lol/blog/release/v1.20.0/

  source
• SorteKanin@feddit.dk ⁨13⁩ ⁨hours⁩ ago

  I’ve, once again, noticed Amazon and Anthropic absolutely hammering my Lemmy instance to the point of the lemmy-ui container crashing.

  I’m just curious, how did you notice this in the first place? What are you monitoring to know and how do you present that information?

  source
• e0qdk@reddthat.com ⁨1⁩ ⁨day⁩ ago

  I don’t like Anubis because it requires me to enable JS – making me less secure. reddthat started using go-away recently as an alternative that doesn’t require JS when we were getting hammered by scrapers.

  source

  • BakedCatboy@lemmy.ml ⁨1⁩ ⁨day⁩ ago

    Fwiw Anubis is adding a nojs meta refresh challenge that if it doesn’t have issues will soon be the new default challenge

    source

    • dan@upvote.au ⁨22⁩ ⁨hours⁩ ago

      Won’t the bots just switch to using that instead of the heavier JS challenge?

      source

      • -> View More Comments
  • Jumuta@sh.itjust.works ⁨19⁩ ⁨hours⁩ ago

    iirc there’s instructions on completing the anubis challenge manually

    source
• Charlxmagne@lemmy.world ⁨8⁩ ⁨hours⁩ ago

  Been seeing this on people’s invidious instances

  source
• rtxn@lemmy.world ⁨23⁩ ⁨hours⁩ ago

  But don’t you know that Anubis is MALWARE?

  …according to some of the clowns at the FSF, which is definitely one of the opinions to have. www.fsf.org/…/our-small-team-vs-millions-of-bots

  source

  • dan@upvote.au ⁨22⁩ ⁨hours⁩ ago

    tbh I kinda understand their viewpoint. Not saying I agree with it.

    The Anubis JavaScript program’s calculations are the same kind of calculations done by crypto-currency mining programs. A program which does calculations that a user does not want done is a form of malware.

    source

    • Natanox@discuss.tchncs.de ⁨22⁩ ⁨hours⁩ ago

      That’s guilt by association. Their viewpoint is awful.

      I also wished there was no security at the gate of concerts, but I happily accept it if that means actual security (if done reasonably of course). And quite frankly, cute anime girl doing some math is so, so much better than those god damn freaking captchas. Or the service literally dying due to AI DDoS.

      source

      • -> View More Comments
    • interdimensionalmeme@lemmy.ml ⁨18⁩ ⁨hours⁩ ago

      Requiring client to runs client side code, if tolerated, will lead to the extinction of pure http clients. That in turn will enable to drm the whole web. I rather see it all burn.

      source
  • chihuamaranian@lemmy.ca ⁨22⁩ ⁨hours⁩ ago

    The FSF explanation of why they dislike Anubis could just as easily apply to the process of decrypting TLS/HTTPS. You know, something uncontroversial that every computer is expected to do when they want to communicate securely.

    I don’t fundamentally see the difference between “The computer does math to ensure end-to-end privacy” and “The computer does math to mitigate DDoS attempts on the server”. Either way, without such protections the client/server relationship is lacking crucial fundamentals that many interactions depend on.

    source

    • SheeEttin@lemmy.zip ⁨8⁩ ⁨hours⁩ ago

      Right. One of the facets of encryption is rounds: if you apply the same algorithm 10,000 times instead of just one, it might make it slightly slower each time you need to run it, but it makes it vastly slower for someone trying to brute-force your password.

      source
    • rtxn@lemmy.world ⁨21⁩ ⁨hours⁩ ago

      I’ve made that exact comparison before. TLS uses encryption; ransomware also uses encryption; by their logic, serving web content through HTTPS with no way to bypass it is a form of malware. The same goes for injecting their donation banner using an iframe.

      source
• fossilesque@mander.xyz ⁨9⁩ ⁨hours⁩ ago

  I’ve been planning on seeing this up for ages. Love the creators vibe. Thanks for this.

  source
• possiblylinux127@lemmy.zip ⁨1⁩ ⁨day⁩ ago

  It doesn’t stop bots

  All it does is make clients do as much or more work than the server which makes it less temping to hammer the web.

  source

  • sailorzoop@lemmy.librebun.com ⁨1⁩ ⁨day⁩ ago

    Yeah, from what I understand it’s nothing crazy for any regular client, but really messes with the bots.
    I don’t know, I’m just so glad and happy it works, it doesn’t mess with federation and it’s barely visible when accessing the sites.

    source

    • possiblylinux127@lemmy.zip ⁨1⁩ ⁨day⁩ ago

      Personally my only real complaint is the lack of wasm. Outside if that it works fairly well.

      source
• blob42@lemmy.ml ⁨21⁩ ⁨hours⁩ ago

  I am planning to use it. For caddy users I came up some time ago with a solution that works after being bombarded by AI crawlers for weeks.

  It is a custom caddy CEL expression filter coupled with caddy-ratelimit and caddy-defender.

  Now here’s the fun part, the defender plugin can produce garbage as response so when a matching AI crawler fits it will poison their training dataset.

  Originally I only relied on the rate limited and noticed the AI bots kept trying whenever the rate limit was reset. Once I introduced data poisoning they all stopped :)

  `git.blob42.xyz { @bot <<CEL header({‘Accept-Language’: ‘zh-CN’}) || header_regexp(‘User-Agent’, ‘(?i:(.bot.|.crawler.|.meta.|.google.|.microsoft.|.spider.))’) CEL

  abort @bot
  
  
  defender garbage {
  
      ranges aws azurepubliccloud deepseek gcloud githubcopilot openai 47.0.0.0/8
    
  }
  
  rate_limit {
      zone dynamic_botstop {
          match {
              method GET
               # to use with defender
               #header X-RateLimit-Apply true
               #not header LetMeThrough 1
          }
          key {remote_ip}
          events 1500
          window 30s
          #events 10
          #window 1m
      }
  }
  
  reverse_proxy upstream.server:4242
  
  handle_errors 429 {
      respond "429: Rate limit exceeded."
  }
  

  }`

  source

  • azertyfun@sh.itjust.works ⁨7⁩ ⁨hours⁩ ago

    If I am not mistaken the 47.0.0.0/8 ip block is for Alibaba cloud

    That’s an ARIN block according to Wikipedia so North America, under Northen Telecom until 2010. It does look like Alibaba operate many networks under that /8, but I very much doubt it’s the whole /8 which would be worth a lot; a /16 is apparently worth around $3-4M, so a /8 can be extrapolated to be worth upwards of a billion dollars! I doubt they put all their eggs into that particular basket. So you’re probably matching a lot of innocent North American IPs with this.

    source

    • blob42@lemmy.ml ⁨6⁩ ⁨hours⁩ ago

      Right I must have just blanket banned the whole /8 to be sure alibaba cloud is included. Did some time ago so I forgot

      source

      • -> View More Comments
• NotSteve_@piefed.ca ⁨23⁩ ⁨hours⁩ ago

  I love Annubis just because the dev is from my city that's never talked about (Ottawa)

  source

  • SheeEttin@lemmy.zip ⁨22⁩ ⁨hours⁩ ago

    Well not never, you’ve got the Senators.

    Which will never not be funny to me since it’s Latin for “old men”.

    source

    • NotSteve_@piefed.ca ⁨21⁩ ⁨hours⁩ ago

      Hahaha I didn't know that but that is funny. Admittedly I'm not too big into hockey so I've got no gauge on how popular the Sens are

      source
• SorteKanin@feddit.dk ⁨13⁩ ⁨hours⁩ ago

  Also your avatar and the image posted here (not the thumbnail) seem broken - I wonder if that’s due to Anubis?

  source

  • sailorzoop@lemmy.librebun.com ⁨4⁩ ⁨hours⁩ ago

    Just updated the post again, yeah. But I think that was due to me changing nameservers for my domain at the time. Cheers.

    source
• TomAwezome@lemmy.world ⁨1⁩ ⁨day⁩ ago

  Thanks for the “incoherent rant”, I’m setting some stuff up with Anubis and Caddy so hearing your story was very welcome :)

  source
• dan@upvote.au ⁨22⁩ ⁨hours⁩ ago

  The Anubis site thinks my phone is a bot :/

  tbh I would have just configured a reasonable rate limit in Nginx and left it at that.

  Won’t the bots just hammer the API instead now?

  source

  • Flipper@feddit.org ⁨20⁩ ⁨hours⁩ ago

    No. The rate limit doesn’t work as they use huge IP Spaces to crawl. Each IP alone is not bad they just use several thousand of them.

    Using the API would assume some basic changes. We don’t do that here. If they wanted that, they could run their own instance and would even get notified about changes. No crawling required at all.

    source
• sic_semper_tyrannis@lemmy.today ⁨1⁩ ⁨day⁩ ago

  Futo gave them a micro-grant this month

  source
• danielquinn@lemmy.ca ⁨1⁩ ⁨day⁩ ago

  I’ve been thinking about setting up Anubis to protect my blog from AI scrapers, but I’m not clear on whether this would also block search engines. It would, wouldn’t it?

  source

  • RedBauble@sh.itjust.works ⁨1⁩ ⁨day⁩ ago

    You can setup the policies to allow search engines through, the default policy linked in the docs does that

    source

    • danielquinn@lemmy.ca ⁨23⁩ ⁨hours⁩ ago

      This all appears to be based on the user agent, so wouldn’t that mean that bad-faith scrapers could just declare themselves to be typical search engine user agent?

      source

      • -> View More Comments
  • sailorzoop@lemmy.librebun.com ⁨1⁩ ⁨day⁩ ago

    I’m not entirely sure, but if you look here github.com/TecharoHQ/anubis/tree/main/data/bots
    They have separate configs for each bot. github.com/TecharoHQ/anubis/…/botPolicies.json

    source
• lambalicious@lemmy.sdf.org ⁨1⁩ ⁨day⁩ ago

  Positives: nice uwu art.

  Negatives: requires javascript, intrinsically ableist.

  source

  • phase@lemmy.8th.world ⁨1⁩ ⁨day⁩ ago

    There’s another challenge available, without javascript.

    source
  • AmbitiousProcess@piefed.social ⁨22⁩ ⁨hours⁩ ago

    Could you elaborate on how it's ableist?

    As far as I'm aware, not only are they making a version that doesn't even require JS, but the JS is only needed for the challenge itself, and the browser can then view the page(s) afterwards entirely without JS being necessary to parse the content in any way. Things like screen readers should still do perfectly fine at parsing content after the browser solves the challenge.

    source
  • ohshit604@sh.itjust.works ⁨22⁩ ⁨hours⁩ ago

    How is the art a positive?

    source

    • lambalicious@lemmy.sdf.org ⁨7⁩ ⁨hours⁩ ago

      What do you mean, how?

      Cute anime catgirl, a staple of the internet, without having to be showy or anything. And there are hooks to change it.

      (Was actually half-surprised they didn’t go with “anime!stereotypical egyptian priestess” given the context of the software, but I feel that would have ended up too thematically overloaded in the end)

      source
• TheHobbyist@lemmy.zip ⁨23⁩ ⁨hours⁩ ago

  @demigodrick@leppy.zip

  Perhaps of interest? I don’t know how many bots you’re facing.

  source