Anubis is awesome and I want to talk aout it

Submitted ⁨⁨3⁩ ⁨weeks⁩ ago⁩ by ⁨SmokeyDope@piefed.social⁩ to ⁨selfhosted@lemmy.world⁩

https://media.piefed.social/posts/kT/Nc/kTNcXnTEexk4sCW.webp

I got into the self-hosting send this year via my own website run on old recycled thinkpad on my own home connection. I a lot of time was spent learning about ufw, reverse proxies, header security hardening, fail2ban.

Despite all that I still had a problem with bots knocking on my ports spamming my logs. I tried some hackery getting fail2ban to read caddy logs but that didnt work for me. I nearly considered giving up and going with cloufdlare like half the internet does. But my stubbornness for open source self hosting and the recent cloud flare outages this year have encouraged trying alternatives.

Image

Coinciding with that has been an increase in exposure to seeing this thing in the places I frequent like codeberg. This is Anubis, a proxy type firewall that forces the browser client to do a proof-of-work security check and some other nice clever things to stop bots from knocking. I got interested and started thinking about beefing up security.

I’m here to tell you to try it if you have a public facing site and want to break away from cloudflare It was VERY easy to install and configure with caddyfile on a debian distro with systemctl. In an hour its filtered multiple bots and so far it seems the knocks have slowed down.

source

Comments

Sort:hotnew top

non_burglar@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
Anubis is an elegant solution to the ai bot scraper issue, I just wish the solution to everything wasn’t just spending compute everywhere. In a world where we need to rethink our energy consumption and generation, even on clients, this is a stupid use of computing power.

source
- Dojan@pawb.social ⁨3⁩ ⁨weeks⁩ ago
  It also doesn’t function without JavaScript. If you’re security or privacy conscious chances are not zero that you have JS disabled, in which case this presents a roadblock.
  
  On the flip side of things, if you are a creator and you’d prefer to not make use of JS (there’s dozens of us) then forcing people to go through a JS “security check” feels kind of shit. The alternative is to just take the hammering, and that feels just as bad.
  
  source
  - SmokeyDope@piefed.social ⁨2⁩ ⁨weeks⁩ ago
    Theres a compute option that doesnt require javascript. Its on site owners to configure IMO, though you can make the argument its not default I guess.
    
    https://anubis.techaro.lol/docs/admin/configuration/challenges/metarefresh
    
    From docs on Meta Refresh Method
    
    Meta Refresh (No JavaScript)
    
    The metarefresh challenge sends a browser a much simpler challenge that makes it refresh the page after a set period of time. This enables clients to pass challenges without executing JavaScript.
    
    To use it in your Anubis configuration:
    
    # Generic catchall rule - name: generic-browser user_agent_regex: >- Mozilla|Opera action: CHALLENGE challenge: difficulty: 1 # Number of seconds to wait before refreshing the page algorithm: metarefresh # Specify a non-JS challenge method
    
    This is not enabled by default while this method is tested and its false positive rate is ascertained. Many modern scrapers use headless Google Chrome, so this will have a much higher false positive rate.
    
    source
    -> View More Comments
  - natecox@programming.dev ⁨3⁩ ⁨weeks⁩ ago
    I feel comfortable hating on Anubis for this. The compute cost per validation is vanishingly small to someone with the existing budget to run a cloud scraping farm, it’s just another cost of doing business.
    
    The cost to actual users though, particularly to lower income segments who may not have compute power to spare, is annoyingly large. There are plenty of complaints out there about Anubis being painfully slow on old or underpowered devices.
    
    Some of us do actually prefer to use the internet minus JS, too.
    
    Plus the minor irritation of having anime catgirls suddenly be a part of my daily browsing.
    
    source
    -> View More Comments
  - cecilkorik@piefed.ca ⁨2⁩ ⁨weeks⁩ ago
    
    if you are a creator and you’d prefer to not make use of JS (there’s dozens of us) then forcing people to go through a JS “security check” feels kind of shit. The alternative is to just take the hammering, and that feels just as bad.
    
    I’m with you here. I come from an older time on the Internet. I’m not much of a creator, but I do have websites, and unlike many self-hosters I think, in the spirit of the internet, they should be open to the public as a matter of principle, not cowering away for my own private use behind some encrypted VPN. I want it to be shared. Sometimes that means taking a hammering. It’s fine. It’s nothing that’s going to end the world if it goes down or goes away, and I try not to make a habit of being so irritating that anyone would have much legitimate reason to target me.
    
    I don’t like any of these sort of protections that put the burden onto legitimate users. I get that’s the reality we live in, but I reject that reality, and substitute my own. I understand that some people need to be able to block that sort of traffic to be able to limit and justify the very real costs of providing services for free on the Internet and Anubis does its job for that. But I’m not one of those people. It has yet to cost me a cent above what I have already decided to pay, and until it does, I have the freedom to adhere to my principles on this.
    
    To paraphrase another great movie: Why should any legitimate user be inconvenienced when the bots are the ones who suck. I refuse to punish the wrong party.
    
    source
  - quick_snail@feddit.nl ⁨2⁩ ⁨weeks⁩ ago
    This is why we need these sites to have .onions. Tor Browser has a PoW that doesn’t require js
    
    source
- cadekat@pawb.social ⁨2⁩ ⁨weeks⁩ ago
  Scarcity is what powers this type of challenge: you have to prove you spent a certain amount of electricity in exchange for access to the site, and because electricity isn’t free, this imposes a dollar cost on bots.
  
  You could skip the detour through hashes/electricity and do something with a proof-of-stake cryptocurrency, and just pay for access. The site owner actually gets compensated instead of burning dead dinosaurs.
  
  Obviously there are practical roadblocks to this today that a JavaScript proof-of-work challenge doesn’t face, but longer term…
  
  source
  - natecox@programming.dev ⁨2⁩ ⁨weeks⁩ ago
    The cost here only really impacts regular users, too. The type of users you actually want to block have budgets which easily allow for the compute needed anyways.
    
    source
    -> View More Comments
  - artyom@piefed.social ⁨2⁩ ⁨weeks⁩ ago
    
    You could skip the detour through hashes/electricity and do something with a proof-of-stake cryptocurrency, and just pay for access. The site owner actually gets compensated instead of burning dead dinosaurs.
    
    Maybe if the act of transferring crypto didn’t use a comparable or greater amount of energy…
    
    source
    -> View More Comments
  - daniskarma@lemmy.dbzer0.com ⁨2⁩ ⁨weeks⁩ ago
    I think the issue is that many sites are too aggressive with it. Anubis can be configured to only ask for challenges if the site is under unusual load, for instance when a botnet it’s actually ddosing the site. That’s when it shines.
    
    Making it constantly ask for challenges when the service is not under attack is just a massive waste of energy. And many sites just enable it constantly because they can defer bot pings from their logs that way. That’s for instance what op is doing. It’s just a big misunderstanding of the tool.
    
    source
- quick_snail@feddit.nl ⁨2⁩ ⁨weeks⁩ ago
  We have memory hard cryptographic functions
  
  source
sudo@programming.dev ⁨2⁩ ⁨weeks⁩ ago
I’ve repeatedly stated this before: Proof of Work bot-management is only Proof of Javascript bot-management. It is nothing to a headless browser to by-pass. Proof of JavaScript does work and will stop the vast majority of bot traffic. That’s how Anubis actually works. You don’t need to punish actual users by abusing their CPU. POW is a far higher cost on your actual users than the bots.

Last I checked Anubis has an JavaScript-less strategy called “Meta Refresh”. It first serves you a blank HTML page with a <meta> tag instructing the browser to refresh and load the real page. I highly advise using the Meta Refresh strategy. It should be the default.

I’m glad someone is finally making an open source and self hostable bot management solution. And I don’t give a shit about the cat-girls, nor should you. But Techaro admitted they had little idea what they were doing when they started and went for the “nuclear option”. Fuck Proof of Work. It was a Dead On Arrival idea decades ago. Techaro should strip it from Anubis.

I haven’t caught up with what’s new with Anubis, but if they want to get stricter bot-management, they should check for actual graphics acceleration.

source
- SmokeyDope@piefed.social ⁨2⁩ ⁨weeks⁩ ago
  Something that hasn’t been mentioned much in discussions about Anubis is that it has a graded tier system of how sketchy a client is and changing the kind of challenge based on a a weighted priority system. The default not policies it comes with has it so regular clients are passed through, only slightly weighted clients/IPs get the metarefresh, its when you get to moderate-suspicion level that JavaScript Proof of Work kicks. The bot policy and weight triggers for these levels, challenge action, and duration of clients validity are all configurable.
  
  It seems to me that the sites who heavy hand the proof of work for every client with validity that only last every 5 minutes are the ones who are giving Anubis a bad wrap. The default policy settings dont trigger PoW on the Firefox android clients ive tried including Firefox meanwhile other sites show the finger wag every connection. Its understandable why some choose strict policies but I’m glad theres config options to mitigate impact normal user experience.
  
  source
  - sudo@programming.dev ⁨2⁩ ⁨weeks⁩ ago
    
    Anubis is that it has a graded tier system of how sketchy a client is and changing the kind of challenge based on a a weighted priority system.
    
    Last I checked that was just User-Agent regexes and IP lists. But that’s where Anubis should continue development, and hopefully they’ve improved since. Discerning real users from bots is how you do proper bot management. Not imposing a flat tax on all connections.
    
    source
- rtxn@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
  
  POW is a far higher cost on your actual users than the bots.
  
  That sentence tells me that you don’t understand the purpose of Anubis. It’s not to punish the scrapers. It is to reduce the load on the web server when it is flooded by scraper requests. Bots running headless Chrome can easily solve the challenge, but every second a client is working on the challenge is a second that the web server doesn’t have to waste CPU cycles on serving clankers.
  
  POW is an inconvenience to users. The flood of scrapers is an existential threat to independent websites. And there is a simple fact that you conveniently ignored: it fucking works.
  
  source
  - sudo@programming.dev ⁨2⁩ ⁨weeks⁩ ago
    Its like you didn’t understand anything I said. Anubis does work. I said it works. But it works because most AI crawlers don’t have a headless browser to solve the PoW. To operate efficiently at the high volume required, they use raw http requests. The vast majority are probably using basic python requests module.
    
    You don’t need PoW to throttle general access to your site and that’s not the fundamental assumption of PoW. PoW assumes (incorrectly) that bots won’t pay the extra flops to scrape the website. But bots are paid to scape the website users aren’t. They’ll just scale horizontally and open more parallel connections. They have the money.
    
    source
    -> View More Comments
- ___qwertz___@feddit.org ⁨2⁩ ⁨weeks⁩ ago
  Funnily enough, PoW was a hot topic in academia around the late 90s / early 2000, and it’s somewhat clear that the autor of Anubis has not read much about the discussion back then.
  
  There was a paper called “Proof of work does not work” (or similar, can’t be bothered to look it up) that argued that PoW can not work for spam protection, because you have to support both low-powered consumer devices while blocking spammers with heavy hardware. And that is very valid concern. Then there was a paper arguing that PoW can still work, as long as you scale the difficulty in such a way that a legit user (e.g. only sending one email) has a low difficulty, while a spammer (sending thousands of emails) has a high difficulty.
  
  The idea of blocking known bad actors actually is used in email quite a lot in forms of DNS block lists (DNSBLs) such as spamhaus (this has nothing to do with PoW, but such a distributed list could be used to determine PoW difficulty).
  
  Anubis on the other hand does nothing like that and a bot developed to pass Anubis would do so trivially.
  
  Sorry for long text.
  
  source
  - Flipper@feddit.org ⁨2⁩ ⁨weeks⁩ ago
    At least in the beginning the scrapers just used curl with a different user agent. Forcing them to use a headless client is already a 100x increase in resources for them. That in itself is already a small victory and so far it is working beautifully.
    
    source
    -> View More Comments
  - sudo@programming.dev ⁨2⁩ ⁨weeks⁩ ago
    
    Then there was a paper arguing that PoW can still work, as long as you scale the difficulty in such a way that a legit user
    
    Telling a legit user from a fake user is the entire game. If you can do that you just block the fake user. Professional bot blockers like Cloudflare or Akamai have machine learning systems to analyze trends in network traffic and serve JS challenges to suspicious clients. Last I checked, all Anubis uses is User-Agent filters, which is extremely behind the curve. Bots are able to get down to faking TLS fingerprints and matching them with User-Agents.
    
    source
- quick_snail@feddit.nl ⁨2⁩ ⁨weeks⁩ ago
  Hashcash works great, what are you going on about?
  
  source
  - sudo@programming.dev ⁨2⁩ ⁨weeks⁩ ago
    LOL
    
    source
daniskarma@lemmy.dbzer0.com ⁨2⁩ ⁨weeks⁩ ago
I don’t think you have a usecase for Anubis.

Anubis is mainly aimed against bad AI scrappers and some ddos mitigation if you have a heavy service.

You are getting hit exactly the same, anubis doesn’t put up a block list or anything. And most bots are not AI scrappers they are just proving. So the hit on your server is the same.

What you want is to properly set up fail2ban or, even better, crowdsec. That would actually block and ban bots that try to prove your server.

source
natecox@programming.dev ⁨3⁩ ⁨weeks⁩ ago
Counterpoint: Anubis is not awesome: lock.cmpxchg8b.com/anubis.html

source
- Cyberflunk@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
  thank you! this needed said.
  
  This post is a bit critical of a small well-intentioned project, so I felt obliged to email the maintainer to discuss it before posting it online. I didn’t hear back.
  
  i used to watch the dev on mastodon, they seemed pretty radicalized on killing AI, and anyone who uses it (kidding!!) i’m not even surprised you didn’t hear back
  
  great take on the software, and as far as i can tell, playwright still works/completes the unit of work. at scale anubis still seems to work if you have popular content, but does hasnt stopped me using claude code + virtual browsers
  
  im not actively testing it though. im probably very wrong about a few things, but i know anubis isn’t hindering my personal scraping, it does fuck up perplexity and chatgpt bots, which is fun to see.
  
  good luck Blue team!
  
  source
  - kilgore_trout@feddit.it ⁨2⁩ ⁨weeks⁩ ago
    
    the dev […] seemed pretty radicalized on killing Ai
    
    As one should, to lead a similar project.
    
    source
  - natecox@programming.dev ⁨2⁩ ⁨weeks⁩ ago
    For clarity: I didn’t write the article, it’s just a good reference.
    
    source
  - SmokeyDope@piefed.social ⁨2⁩ ⁨weeks⁩ ago
    What use cases does perplexity do that Claude doesn’t for you?
    
    source
henfredemars@infosec.pub ⁨3⁩ ⁨weeks⁩ ago
I appreciate a simple piece of software that does exactly what it’s supposed to do.

source
- merc@sh.itjust.works ⁨2⁩ ⁨weeks⁩ ago
  The front page of the web site is excellent. It describes what it does, and it does its feature set in quick, simple terms.
  
  I can’t tell you how many times I’ve gone to a website for some open-source software and had no idea what it was or how it was trying to do it. They often dive deep into the 300 different ways of installing it, tell you what the current version is and what features it has over the last version, but often they just assume you know the basics.
  
  source
0_o7@lemmy.dbzer0.com ⁨2⁩ ⁨weeks⁩ ago
I don’t mind Anubis but the challenge page shouldn’t really load an image. It’s wasting extra bandwidth for nothing.

Just parse the challenge and move on.

source
- Allero@lemmy.today ⁨2⁩ ⁨weeks⁩ ago
  Afaik, you can set it up not to have any image, or have any other one.
  
  source
  - Voroxpete@sh.itjust.works ⁨2⁩ ⁨weeks⁩ ago
    It’s actually a brilliant monetization model. If you want to use it as is, it’s free, even for large corporate clients.
    
    If you want to get rid of the puppygirls though, that’s when you have to pay.
    
    (The absolute Chads at the UN left the puppygirls in, and I have to respect that
    
    source
    -> View More Comments
- kilgore_trout@feddit.it ⁨2⁩ ⁨weeks⁩ ago
  It’s a palette of 10 colours. I would guess it uses an indexed colorspace, reducing the size to a minimum.
  
  source
  - CameronDev@programming.dev ⁨2⁩ ⁨weeks⁩ ago
    A HTTP get request is a few hundred bytes. The response is 28KB. Thats 280x. If a large botnet wanted to denial of service an Anubis protected site, requesting that image could be enough.
    
    Ideally, Anubis should serve as little data as possible until the POW is completed. Caching the POW algorithm (and the image) to a CDN would also mitigate the issue.
    
    source
    -> View More Comments
quick_snail@feddit.nl ⁨2⁩ ⁨weeks⁩ ago
Kinda sucks how it makes websites inaccessible to folks who have to disable JavaScript for security.

source
- poVoq@slrpnk.net ⁨2⁩ ⁨weeks⁩ ago
  I kinda sucks how AI scrapers make websites inaccessible to everyone 🙄
  
  source
  - quick_snail@feddit.nl ⁨2⁩ ⁨weeks⁩ ago
    Not if the admin has a cache. It’s not a difficult problem for most websites
    
    source
    -> View More Comments
  - Mwa@thelemmy.club ⁨2⁩ ⁨weeks⁩ ago
    and they dont respect robots.txt
    
    source
  - elbarto777@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    You are both right.
    
    source
- WhyJiffie@sh.itjust.works ⁨2⁩ ⁨weeks⁩ ago
  there’s a fork that has non-js checks. I don’t remember the name but maybe that’s what should be made more known
  
  source
  - quick_snail@feddit.nl ⁨2⁩ ⁨weeks⁩ ago
    Please share if you know.
    
    The only way I know how to do this is running a Tor Onion Service, since the tor protocol has built-in pow support (without js)
    
    source
    -> View More Comments
url@feddit.fr ⁨2⁩ ⁨weeks⁩ ago
Honestly im not a big fan of anubis . it fucks users with slow devices

https://lock.cmpxchg8b.com/anubis.html

source
- url@feddit.fr ⁨2⁩ ⁨weeks⁩ ago
  Did i forgot to mention it doesnt work without js that i keep disabled
  
  source
smh@slrpnk.net ⁨2⁩ ⁨weeks⁩ ago
The creator is active on a professional slack I’m on and they’re lovely and receptive to user feedback. Their tool is very popular in the online archives/cultural heritage scene (we combine small budgets and juicy, juicy data).

My site has enabled js-free screening when the site load is low, under the theory that if the site load is too high then no one’s getting in anyway.

source
A_norny_mousse@feddit.org ⁨2⁩ ⁨weeks⁩ ago
At the time of commenting, this post is 8h old. I read all the top comments, many of them critical of Anubis.

I run a small website and don’t have problems with bots. Of course I know what a DDOS is - maybe that’s the only use case where something like Anubis would help, instead of the strictly server-side solution I deploy?

As a strictly server-side (and low computing) solution I use CrowdSec (it seems to work with caddy btw). It took a little setting up, but it does the job.

Am I missing something here? Why wouldn’t that be enough? Why do I need to heckle my visitors?

Despite all that I still had a problem with bots knocking on my ports

By the time Anubis gets to work, the knocking already happened so I don’t really understand this argument.

spamming my logs.

If spamming the logs is the only concern here, I rest my case.
Otherwise, if the system is set up to reject a certain type of requests, these are microsecond transactions of no (DDOS exception) harm.

source
- poVoq@slrpnk.net ⁨2⁩ ⁨weeks⁩ ago
  AI scraping is a massive issue for specific types of websites, such has git forges, wikis and to a lesser extend Lemmy etc, that rely on complex database operations that can not be easily cached. Unless you massively overprovision your infrastructure these web-applications come to a grinding halt by constantly maxing out the available CPU power.
  
  The vast majority of the critical commenters here seem to talk from a point of total ignorance about this, or assume operators of such web applications have time for hyperviligance to constantly monitor and manually block AI scrapers (that do their best to circumvent more basic blocks). The realistic options for such operators are right now: Anubis (or similar), Cloudflare or shutting down their servers. Of these Anubis is clearly the least bad option.
  
  source
  - chunes@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    Sounds like maybe webapps are a bad idea then.
    
    If they need dynamism, how about releasing a desktop application?
    
    source
- SmokeyDope@piefed.social ⁨2⁩ ⁨weeks⁩ ago
  Something to understand is that computer nerds who frequent places like this or hacker fourms have predisposition to be loud mouth pessimist with an ego types. Any time theres a working but imperfect solution to a complex problem but its not mathematically perfect, you’ll then have someone loudly declare that that solution makes no sense, write a two paragraph argument on halfassed assumptions that are often misrepresenting the problem out of ignorance or intention, link to a blogspam article written by a hypernerd who they got their talking points from, then laud about wondering how their obviously better solution isnt clearer to the stupid ignorant cretins. Thats just how it is.
  
  If crowdsec works for you thats great but also its a corporate product whos premium sub tier starts at 900$/month not exactly a pure self hosted solution.
  
  I’m not a hypernerd, still figuring all this out among the myriad of possible solutions with different complexity and setup times. All the self hosters in my internet circle started adopting anubis so I wanted to try it. Anubis was relatively plug and play with prebuilt packages and great install guide documentation.
  
  Allow me to expand on the problem I was having. It wasnt just that I was getting a knock or two, its that I was getting 40 knocks every few seconds scraping every page and searching for a bunch that didnt exist that would allow exploit points in unsecured production vps systems.
  
  On a computational level the constant network activity of bytes from webpage, zip files and images downloaded from scrapers pollutes traffic. Anubis stops this by trapping them in a landing page that transmits very little information from the server side. By traping the bot in an Anubis page which spams that 40 times on a single open connection before it gives up, it reduces overall network activity/ data transfered which is often billed as a metered thing as well as the logs.
  
  And this isnt all or nothing. You don’t have to pester all your visitors, only those with sketchy clients. Anubis uses a weighted priority which grades how legit a browser client is. Most regular connections get through without triggering, weird connections get various grades of checks by how sketchy they are. Some checks dont require proof of work or JavaScript.
  
  On a psychological level it gives me a bit of relief knowing that the bots are getting properly sinkholed and I’m punishing/wasting the compute of some asshole trying to find exploits my system to expand their botnet. And a bit of pride knowing I did this myself on my own hardware without having to cop out to a corporate product.
  
  Its nice that people of different skill levels and philosophies have options to work with. One tool can often complement another too. Anubis worked for what I wanted, filtering out bots from wasting network bandwith and giving me peace of mind where before I had no protection. All while not being noticeable for most people because I have the ability to configure it to not heckle every client every 5 minutes like some sites want to do.
  
  source
  - A_norny_mousse@feddit.org ⁨2⁩ ⁨weeks⁩ ago
    
    If crowdsec works for you thats great but also its a corporate product
    
    It’s also fully FLOSS with dozens of contributors (not to speak of the community-driven blocklists). If they make money with it (presumably for large firms), great.
    
    not exactly a pure self hosted solution.
    
    Why? I host it, I run it. It’s even in Debian repos, but I choose their own more up-to-date ones.
    
    All the self hosters in my internet circle started adopting anubis so I wanted to try it. Anubis was relatively plug and play with prebuilt packages
    
    Yeah…
    
    Allow me to expand on the problem I was having. It wasnt just that I was getting a knock or two, its that I was getting 40 knocks every few seconds scraping every page and searching for a bunch that didnt exist that would allow exploit points in unsecured production vps systems.
    
    Again, a properly set up WAF will deal with this pronto
    
    You should not have exploit points in unsecured production systems, full stop.
    
    On a computational level the constant network activity of bytes from webpage, zip files and images downloaded from scrapers pollutes traffic. Anubis stops this by trapping them in a landing page that transmits very little information from the server side.
    
    And instead you leave the computations to your clients. Which becomes a problem on slow hardware.
    
    Again, with a properly set up WAF there’s no “traffic pollution” or “downloading of zip files”.
    
    Anubis uses a weighted priority which grades how legit a browser client is.
    
    And apart from the user agent and a few other responses, all of which are easily spoofed, this means “do some javascript stuff on the local client” (there’s a link to an article here somewhere that explains this well) which is much less trivial than you make it sound.
    
    source
    -> View More Comments
- daniskarma@lemmy.dbzer0.com ⁨2⁩ ⁨weeks⁩ ago
  You are right. For most self-hosting usecases anubis is not only irrelevant, but it actually works against you. False sense of security and making your devices do extra work for nothing.
  
  Anubis is though for public facing services that may get ddos or AI scrapped by some not targeted bot (for a target bot it’s trivial to get over Anubis in order to scrap).
  
  source
- Pastime0293@discuss.tchncs.de ⁨2⁩ ⁨weeks⁩ ago
  I also used CrowdSec for almost a year, but as AI scrapers became more aggressive, CrowdSec alone wasn’t enough. The scrapers used distributed IP ranges and spoofed user agents, making them hard to detect and costing my Forgejo instance a lot in expensive routes. I tried custom CrowdSec rules but hit its limits.
  
  Then I discovered Anubis. It’s been an excellent complement to CrowdSec — I now run both. In my experience they work very well together, so the question isn’t “A or B?” but rather “How can I combine them, if needed?”
  
  source
- quick_snail@feddit.nl ⁨2⁩ ⁨weeks⁩ ago
  With varnish and wazuh, I’ve never had a need for Anubis.
  
  My first recommendation for anyone struggling with bots is to fix their cache.
  
  source
  - kalleboo@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    Anubis was originally created to protect git web interfaces since they have a lot of heavy-to-compute URLs that aren’t feasible to cache (revision diffs, zip downloads etc).
    
    After that I think it got adopted by a lot of people who didn’t actually need it, they just don’t like seeing AI scrapers in their logs.
    
    source
    -> View More Comments
sudoer777@lemmy.ml ⁨2⁩ ⁨weeks⁩ ago
I host my main server on my own hardware, and a VPN on Hetzner because my shitty ISP doesn’t let me port forward. For the past year, bots were hitting my Forgejo instance hard. I forgot to disable registration and they generated hundreds of accounts with hundreds of repos with sketchy links, generating terrabytes of traffic from my VPS, costing me money in traffic. I disabled registration and deleted the spam, and bots still kept hitting my server for several months, which would cause memory leaks over time and crash it, and still costed me money with terrabytes of traffic per month. A few weeks ago, I put Anubis on the VPS. Now, zero bots hit my Forgejo instance and I don’t pay for their traffic anymore. Problem solved.

source
- Jason2357@lemmy.ca ⁨2⁩ ⁨weeks⁩ ago
  Its always code forges and wikis that are effected by this because the scrapers spider down into every commit or edit in your entire history, then come back the next day and check every “page” again to see if any changed. Consider just blocking pages that are commit history at your reverse proxy.
  
  source
- LOLseas@lemmy.zip ⁨2⁩ ⁨weeks⁩ ago
  This is the first time I’ve ever seen it misspelled like that. It’s ‘terabyte/terabytes’. 1,024 GBs worth of data.
  
  source
  - sudoer777@lemmy.ml ⁨2⁩ ⁨weeks⁩ ago
    Oops, although terabyte is 1000 GB, 1024 GiB is tebibyte
    
    source
    -> View More Comments
- WorldsDumbestMan@lemmy.today ⁨2⁩ ⁨weeks⁩ ago
  Nice ads people! Good job!
  
  source
  - helix@feddit.org ⁨2⁩ ⁨weeks⁩ ago
    So you think techaro paid them?
    
    source
    -> View More Comments
panda_abyss@lemmy.ca ⁨2⁩ ⁨weeks⁩ ago
I like the quirky SPH character

source
TerHu@lemmy.dbzer0.com ⁨2⁩ ⁨weeks⁩ ago
yes, please be mindful when using cloudflare. with them you’re possibly inviting in a much much bigger problem

www.devever.net/~hl/cloudflare

source
- quick_snail@feddit.nl ⁨2⁩ ⁨weeks⁩ ago
  Great article, but I disagree about WAFs.
  
  Try to secure a nonprofits web infrastructure with as 1 IT guy and no budget for devs or security.
  
  It would be nice if we could update servers constantly and patch unmaintained code, but sometimes you just need to front it with something that plugs those holes until you have the capacity to do updates.
  
  But 100% the WAF should be run locally, not a MiTM from evil US corp in bed with DHS.
  
  source
drkt_@lemmy.dbzer0.com ⁨2⁩ ⁨weeks⁩ ago
Stop playing wack-a-mole with these fucking people and build TARPITS!

Make it HURT to crawl your site illegitimately.

source
orbituary@lemmy.dbzer0.com ⁨3⁩ ⁨weeks⁩ ago
It’s a great service. I hate the character.

source
- SmokeyDope@piefed.social ⁨3⁩ ⁨weeks⁩ ago
  You know the thing is that they know the character is a problem/annoyance, thats how they grease the wheel on selling subscription access to a commecial version with different branding.
  
  https://anubis.techaro.lol/docs/admin/botstopper/
  
  pricing from site
  
  Commercial support and an unbranded version
  
  If you want to use Anubis but organizational policies prevent you from using the branding that the open source project ships, we offer a commercial version of Anubis named BotStopper. BotStopper builds off of the open source core of Anubis and offers organizations more control over the branding, including but not limited to:
  
  Custom images for different states of the challenge process (in process, success, failure)
  
  Custom CSS and fonts
  
  Custom titles for the challenge and error pages
  
  “Anubis” replaced with “BotStopper” across the UI
  
  A private bug tracker for issues
  
  In the near future this will expand to:
  
  A private challenge implementation that does advanced fingerprinting to check if the client is a genuine browser or not
  
  Advanced fingerprinting via Thoth-based advanced checks
  
  In order to sign up for BotStopper, please do one of the following:
  
  Sign up on GitHub Sponsors at the $50 per month tier or higher
  
  Email <sales@techaro.lol> with your requirements for invoicing, please note that custom invoicing will cost more than using GitHub Sponsors for understandable overhead reasons
  
  I have to respect the play tbh its clever. Absolutely the kind of greasy shit play that Julian from the trailer park boys would do if he were an open source developer.
  
  Image
  
  source
  - webghost0101@sopuli.xyz ⁨3⁩ ⁨weeks⁩ ago
    I wish more projects did stuff like this.
    
    It just feels silly and unprofessional while being seriously useful. Exactly my flavour of software, makes the web feel less corporate.
    
    source
- CoyoteFacts@piefed.ca ⁨3⁩ ⁨weeks⁩ ago
  You can customize the images if you want: https://anubis.techaro.lol/docs/admin/botstopper#customizing-images
  
  source
Arghblarg@lemmy.ca ⁨2⁩ ⁨weeks⁩ ago
I have a script that watches apache or caddy logs for poison link hits and a set of bot user agents, adding IPs to an ipset blacklist, blocking with iptables. I should polish it up for others to try. My list of unique IPs is well over 10k in just a few days.

source
- JustTesting@lemmy.hogru.ch ⁨2⁩ ⁨weeks⁩ ago
  This is the way. I also have rules for hits to url, without a referer, that should never be hit without a referer, with some threshold to account for a user hitting F5. Plus a whitelist of real users (ones that got a 200 on a login endpoint).
  
  then there’s ratelimiting and banning ip’s that hit the ratelimit regularly.
  
  Dowloading abuse ip lists nightly and banning those, that’s around 60k abusive ip’s gone. At that point you probably need to use nftables though, for the sets, as having 60k rules would be a bad idea.
  
  there’s lists of all datacenter ip ranges out there, so you could block as well, though that’s a pretty nuclear option, so better make sure traffic you want is whitelisted. E.g. for lemmy, you can get a list of the ips of all other instances nightly, so you don’t accidentally block them. Lemmy traffic is very spammy…
  
  there’s so much that can be done with f2b and a bit of scripting/writing filters
  
  source
  - iopq@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    Can’t you just bookmark the page?
    
    source
    -> View More Comments
- quick_snail@feddit.nl ⁨2⁩ ⁨weeks⁩ ago
  You just described what wazuh does ootb
  
  source
- pedroapero@lemmy.ml ⁨2⁩ ⁨weeks⁩ ago
  Hi, there are pre-made ipset lists also, ex: github.com/ktsaou/blocklist-ipsets
  
  source
perishthethought@piefed.social ⁨3⁩ ⁨weeks⁩ ago
I don’t really understand what I am seeing here, so I have to ask – are these Security issues a concern?

https://github.com/TecharoHQ/anubis/security

I have a server running a few tiny web sites, so I am considering this, but I’m always concerned about the possibility that adding more things to it could make it less secure, versus more. Thanks for any thoughts.

source
- lime@feddit.nu ⁨2⁩ ⁨weeks⁩ ago
  all of the issues listed are closed so any recent version is fine.
  
  also, you probably don’t need to deploy this unless you have a problem with bots.
  
  source
- SmokeyDope@piefed.social ⁨2⁩ ⁨weeks⁩ ago
  Security issues are always a concern the question is how much. Looking at it they seem to at most be ways to circumvent the Anubis redirect system to get to your page using very specific exploits. These are marked as m low to moderate priority and I do not see anything that implies like system level access which is the big concern. Obviously do what you feel is best but IMO its not worth sweating about. Nice thing about open source projects is that anyone can look through and fix, if this gets more popular you can expect bug bounties and professional pen testing submissions.
  
  source
mrbn@lemmy.ca ⁨3⁩ ⁨weeks⁩ ago
When I visit sites on my cellphone, Anubis often doesn’t let me through.

source
Goretantath@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
My phone hates anubis.

source
Deathray5@lemmynsfw.com ⁨2⁩ ⁨weeks⁩ ago
Unrelated but one day I won’t get gender envy from random cartoon woman

source
Fizz@lemmy.nz ⁨2⁩ ⁨weeks⁩ ago
Its a fun little project and I like the little character but it doesnt actually do anything at this point.

source
sixty@sh.itjust.works ⁨2⁩ ⁨weeks⁩ ago
Yeah im not gonna use this anime stuff

source
quick_snail@feddit.nl ⁨2⁩ ⁨weeks⁩ ago
It’s amazing how few people here are familiar with caching

source
krooklochurm@lemmy.ca ⁨2⁩ ⁨weeks⁩ ago
“Anubis has risen, Wendell”

“Are you Jane’s addiction”?

source
tux0r@feddit.org ⁨2⁩ ⁨weeks⁩ ago
I use it with OpenBSD’s relayd and I find it amazing how little maintenance it needs.

source
Appoxo@lemmy.dbzer0.com ⁨2⁩ ⁨weeks⁩ ago
Maybe you know the answer to my question:
If I’d want to use any app that doesnt run in a webbrowser (e.g. the native jellyfin app), how would that work? Does it still work then?

source
turdas@suppo.fi ⁨2⁩ ⁨weeks⁩ ago
Inspired by this post I spent a couple of hours today trying to set this up on my toy server, only to immediately run into what seems to be a bug where <video> tags loading a simple WebM video from right next to index.html broke because the media response got Anubis’s HTML bot check instead of media.

I suppose my use-case was just too complicated.

source
quick_snail@feddit.nl ⁨2⁩ ⁨weeks⁩ ago

getting fail2ban to read caddy logs

You should look into wazuh

source
termaxima@slrpnk.net ⁨2⁩ ⁨weeks⁩ ago
I am very annoyed that I have to enable cloudflare’s JavaScript on so many websites, I would much prefer if more of them used Anubis so I didn’t have third-party JavaScript running as often.

( coming from an annoying user who tries to enable the fewest things possible in NoScript )

source
silmarine@discuss.tchncs.de ⁨2⁩ ⁨weeks⁩ ago
Thanks for this! In going to set this up for myself.

source

Anubis is awesome and I want to talk aout it

Comments

Meta Refresh (No JavaScript)

Commercial support and an unbranded version