How do you all handle security and monitoring for your publicly accessible services?

Submitted ⁨⁨4⁩ ⁨days⁩ ago⁩ by ⁨a_fancy_kiwi@lemmy.world⁩ to ⁨selfhosted@lemmy.world⁩

This is a continuation of my other post

I now have homeassistant, immich, and authentik docker containers exposed to the open internet. Homeassistant has built in 2FA and authentik is being used as the authentication for immich which supports 2FA. I went ahead and blocked connections from every country except for my own via cloudlfare (I’m aware this does almost nothing but I feel better about it).

At the moment, if my machine became compromised, I wouldn’t know. How do I monitor these docker containers? What’s a good way to block IPs based on failed login attempts? Is there a tool that could alert me if my machine was compromised? Any recommendations?

source

Comments

Sort:hotnew top

Xanza@lemm.ee ⁨4⁩ ⁨days⁩ ago
By not making them publicly accessible. With Wireguard there’s really no reason.

Setup service to be active on a subnet, enable Wireguard to VPN into the subnet and use the services.

source
- peregus@lemmy.world ⁨4⁩ ⁨days⁩ ago
  
  With Wireguard there’s really no reason.
  
  Well, that’s kinda of a personal choice. If somebody needs to have services accessible by someone else besides him, that service can’t be behind a VPN (let’s face the truth: we know that we can’t ask all out relatives and friends to use a VPN).
  
  source
  - KairuByte@lemmy.dbzer0.com ⁨3⁩ ⁨days⁩ ago
    There’s also something to be said about some services being cordoned off in a VPN while leaving some public with security. I don’t necessarily want everyone within my full network if all I want is to share one service with them.
    
    source
    -> View More Comments
  - Xanza@lemm.ee ⁨3⁩ ⁨days⁩ ago
    
    If somebody needs to have services accessible by someone else besides him, that service can’t be behind a VPN
    
    Again, this is the reason VPS’ exist. If that person needs access, then setup Wireguard…
    
    It’s like saying you don’t need a front gate with an access code because then you would have to give out your own access code. But I mean, the lock has the ability to setup more access codes. And you’re saying the only viable option is the leave the gate open and hire a guard to manage access. It’s just… Weird and wrong.
    
    source
    -> View More Comments
- Anivia@feddit.org ⁨3⁩ ⁨days⁩ ago
  Yeah, I’m not gonna tell the 50 users of my plex server to set up wireguard on their devices so they can request movies and TV series on my overseer, when I can instead just use NPM to make it publically accessible with a password prompt
  
  source
  - Xanza@lemm.ee ⁨3⁩ ⁨days⁩ ago
    Your use case, and OPs, are completely different scenarios. I can’t tell if you’re being purposefully disingenuous or just flippantly stupid.
    
    source
- Slax@sh.itjust.works ⁨4⁩ ⁨days⁩ ago
  I agree with WG however I need https for a few locally hosted items like actual budget so I have that through nginx proxy manager. I was debating adding Authelia in front with some of my others (audiobook shelf, home assistant and music assistant) as sometimes I disconnect from my home network and forget to reconnect.
  
  source
  - peregus@lemmy.world ⁨3⁩ ⁨days⁩ ago
    Just out of curiosity, why do you disconnect from your home VPN?
    
    source
  - Xanza@lemm.ee ⁨3⁩ ⁨days⁩ ago
    Why not swap from nginx-proxy-manager to Caddy2, which can handle everything? SSL and reverse_proxy?
    
    source
  - ikidd@lemmy.world ⁨3⁩ ⁨days⁩ ago
    There should be an option in your phone VPN setup to reconnect if app X is being used.
    
    source
    -> View More Comments
jeena@piefed.jeena.net ⁨4⁩ ⁨days⁩ ago
So there is https://en.wikipedia.org/wiki/Fail2ban which helps already to some degree.

But what are you trying to prevent? You have your services in a docker container, hopefully not running as root, which already makes it difficult to break out even if through a bug someone would be able to get access to the docker container.

I mean its not like your stuff is very important for someone to break in like the pentagon, you probably just have some photos from your phone on it, some lights can be switched on and off and some temperatures read.

I'm not trying to say that you should not care about it but I'm trying to figure out what your threat model is.

source
- a_fancy_kiwi@lemmy.world ⁨4⁩ ⁨days⁩ ago
  I feel weird about having those apps on the internet and basically being blind to threats. I mean yeah, I’m not a target on anyone’s list and most IPs visiting the site are bots. But I would still like to know what’s going on.
  
  I don’t work in tech for a living, this is just a hobby for me so I have limited time to work on this stuff and do research. It’s very possible I fucked something up and don’t know it. I figured if I at least got an alert that said “hey, your immich server db was dumped and sent to <insert IP>”, I could at least turn it off
  
  source
  - ikidd@lemmy.world ⁨3⁩ ⁨days⁩ ago
    Yah, it’s just a hobby for you, but it’s also a hobby for script kiddies to use Shodan to find people with out of date web interfaces and pop them. I tell you right now, the Immich team would be the first to say not to put their application publicly accessible.
    
    Just don’t get into this practice, it ends in tears and is way more maintenance to stay protected than just setting up tailscale and using that.
    
    source
  - AtariDump@lemmy.world ⁨3⁩ ⁨days⁩ ago
    By the time you get the alert and act on it, it’s too late.
    
    Don’t expose these things to the open internet; VPN back into your network and access them.
    
    source
- credo@lemmy.world ⁨4⁩ ⁨days⁩ ago
  IOT botnets are a thing. And if someone wanted to fire sell the US, all the vulnerable home networks would be on the table too. Great for a bit if extra chaos.
  
  source
truthfultemporarily@feddit.org ⁨4⁩ ⁨days⁩ ago
I would put this stuff behind VPN.

source
beerclue@lemmy.world ⁨4⁩ ⁨days⁩ ago
Check out crowdsec. Like fail2ban, but with crowdsourced lists on top.

source
- a_fancy_kiwi@lemmy.world ⁨4⁩ ⁨days⁩ ago
  will do, thanks
  
  source
foggy@lemmy.world ⁨4⁩ ⁨days⁩ ago
Auth portal for VPN tunnell -> Authelia -> fail2ban -> VLAN with services only.

Keep that VLAN segmented. You’re good unless you’re a DOGE employee, then I’d recommend quite a bit more security.

source
- pezhore@infosec.pub ⁨4⁩ ⁨days⁩ ago
  This is the way. Layer 3 separation for services you wish to access outside of the home network and the rest of your stuff, with a VPN endpoint exposed for remote access.
  
  It may be overkill, but I have several VLANs for specific traffic:
  
  DMZ - for Wireguard (and if I ever want to stand up a Honeypot)
  
  Services - *arr stack, some Kubes things for remote development
  
  IoT - any smart things like thermostat, home assistant, etc
  
  Trusted - primary at home network for laptops, HTPCs, etc
  
  There are two new additions: a ext-vpn VLAN and a egress-vpn VLAN. I spun up a VM that’s dual homed running its own Wireguard/OpenVPN client on the egress side, serving DHCP on the ext-vpn side. The latter has its own wireless ssid so that anyone who connects to it is automatically on a VPN into a non-US country.
  
  source
- a_fancy_kiwi@lemmy.world ⁨4⁩ ⁨days⁩ ago
  I’ve seen a bunch of people recommend Authelia. Do you mind if I ask why you went with it over other software? I only went with authentik because I found a tutorial on it first
  
  source
krash@lemmy.ml ⁨3⁩ ⁨days⁩ ago
I’ve tried different approaches with fail2ban, crowdsec, VPNs, etc. What I settled on is to divide the data of my services in two categories: confidential and “I can live with it leaking”.

The ones that host confidential data is behind a VPN and has some basic monitoring on them.

The ones that are out in the public are behind a WAF from cloudflare with pretty restrictive rules.

Yes, cloudflare suck etc., but the value of stopping potential attacks before they reach your services is hard to match.

Just keep in mind: you need layers of different security measures to protect your services (such as backups, control of network traffic, monitoring and detection, and so on).

source
- a_fancy_kiwi@lemmy.world ⁨3⁩ ⁨days⁩ ago
  
  has some basic monitoring on them.
  
  What monitoring software are you using?
  
  I feel like the other measures you talked about (backups, condom of network traffic, etc) I’m doing ok on. Its really just the monitoring where I’m stuck. There’s so many options
  
  source
  - krash@lemmy.ml ⁨2⁩ ⁨days⁩ ago
    There are so many monitoring tools with various degrees of complicated setup / configuration or the amount of information you get. And honestly, I’ve looked into various tools: checkmk, monit, Prometheus… And realised that I rarely look into that information anyway. Of all “fancy” tools, I liked the ease of Netdata to set up and the amount of information that you get. However, beware that their in the process to make their free / homelad offering worse. I’ve been eyeing beszel and don’t forget CLI based tools that are avaible such as atop, btop, htop or glances.
    
    If you want to delve deeper into the rabbit hole of monitoring, I can recommend you to read this article below: matduggan.com/were-all-doing-metrics-wrong/
    
    source
smiletolerantly@awful.systems ⁨4⁩ ⁨days⁩ ago
We expose about a dozen services to the open web. Haven’t bothered with something like Authentik yet, just strong passwords.

We use a solid OPNSense Firewall config with rather fine-grained permissions to allow/forbid traffic to the respective VMs, between the VMs, between VMs and the NAS, and so on.

We also have a wireguard tunnel to home for all the services that don’t need to be available on the internet publicly. That one also allows access to the management interface of the firewall.

In OPNSense, you get quite good logging capabilities, should you suspect someone is trying to gain access, you’ll be able to read it from there.

I am also considering setting up Prometheus and Grafana for all our services, which could point out some anomalies, though that would not be the main usecase.

Lastly, I also have a server at a hoster for some stuff that is not practical to host at home. The hoster provided a very rudimentary firewall, so I’m using that to only open necessary ports, and then Fail2Ban to insta-ban IPs for a week on the first offense. Have also set it up so they get banned on Cloudflare’s side, so before another malicious request ever reaches me.

Have not had any issues, ever.

source
- a_fancy_kiwi@lemmy.world ⁨4⁩ ⁨days⁩ ago
  
  Have also set it up so they get banned on Cloudflare’s side, so before another malicious request ever reaches me.
  
  How did you end up setting that up?
  
  source
  - smiletolerantly@awful.systems ⁨4⁩ ⁨days⁩ ago
    Fail2ban allows you set different actions for different infringements, as well as multiple ones. So in addition to being put in a “local” jail, the offending IP also gets added to the cloudflare rules (? Is that what its called?) via their API. It’s a premade action called “cloudflare-token-multi”
    
    source
    -> View More Comments
jagged_circle@feddit.nl ⁨3⁩ ⁨days⁩ ago
Wazuh

source
- a_fancy_kiwi@lemmy.world ⁨3⁩ ⁨days⁩ ago
  I’ll look into it, thank you
  
  source
just_another_person@lemmy.world ⁨4⁩ ⁨days⁩ ago
Why? Not every service is meant to be exposed to the open internet. Immich is the only one of what you listed that makes sense to have out in the open.

source
- jeena@piefed.jeena.net ⁨4⁩ ⁨days⁩ ago
  What about home assistant? Me and the Family quite often use the HA app on the go, sometimes even from other computers like at my parents or in a hotel to check on the house and the cat. I also gave my dad access to it so he can see if we're at home and things like that.
  
  Same with my dads HA.
  
  source
  - hellothere@sh.itjust.works ⁨4⁩ ⁨days⁩ ago
    Set up a VPN to get back inside your network if you’re outside it. Then you don’t need HA (or anything else for that matter) to be public.
    
    source
    -> View More Comments
  - ShortN0te@lemmy.ml ⁨3⁩ ⁨days⁩ ago
    HA had 2 security audits. I would not worry too much. Always depends on what you can control with it. home-assistant.io/…/security-audits-of-home-assis…
    
    source
- a_fancy_kiwi@lemmy.world ⁨4⁩ ⁨days⁩ ago
  I’ve been playing around with the voice assistant stuff in homeassistant and it seemingly needs a public url to get all the features. I could be wrong about that though?
  
  I put authentik in front of immich to handle authentication so that I would need need a 2FA code
  
  source
  - just_another_person@lemmy.world ⁨4⁩ ⁨days⁩ ago
    Most definitely does not need a public URL for Assist in HA. Not sure where you read that.
    
    It sounds like you need a VPN to your internal services if you’re concerned about security.
    
    source
    -> View More Comments
j4k3@lemmy.world ⁨4⁩ ⁨days⁩ ago
I’ve half ass thought about this but never have tried to actually self host. If you have access to all devices, why not just use your own self signed certificates to encrypt everything and require the certificate for all connections? Then there is never a way to log in or connect right? The only reason for any authentication is to make it possible to use any connection to dial into your server. So is that a bug or a feature. Maybe I’m missing something fundamental in this abstract concept that someone will tell me?

source
- a_fancy_kiwi@lemmy.world ⁨4⁩ ⁨days⁩ ago
  
  If you have access to all devices, why not just use your own self signed certificates to encrypt everything and require the certificate for all connections?
  
  Sounds like you are describing a VPN. I was using that setup before but small stuff like immich album sharing via a link won’t work properly. Also, having to ensure a vpn is on and connected is a little to much to ask of my partner; they would turn it off and forget about it and then ask why their app wasn’t working :/
  
  source
  - j4k3@lemmy.world ⁨4⁩ ⁨days⁩ ago
    I mean more like a self signed TLS certificate with your own host manually set in the browser. Then only make the TLS port available, or something like that. If you have access to both(all) devices, you should be able to fully encrypt by bruit force and without registering the certificate with anyone. That is what I do with AI at home.
    
    source
    -> View More Comments
  - peregus@lemmy.world ⁨4⁩ ⁨days⁩ ago
    Beside the fact that you would like to understand if you’ve done everything properly (that’s good, but I can’t help you here), a VPN on a smartphone can be always active. Mine is always on and I’ve never noticed any battery problem. If you prefer something simpler there’s Tailscale.
    
    source
    -> View More Comments
q7mJI7tk1@lemmy.world ⁨4⁩ ⁨days⁩ ago
So among my services I self host, a few need to be publicly accessible for work. For those I wish to remain private, I use Caddy allowing only private IP ranges, plus then Authelia as auth which is set to 30 days. There is then the login of each service being Authelia as well. It’s as good as it needs to be for my needs.

If I were only self hosting private services, then as others have said, I would put all access through a VPN.

source
- a_fancy_kiwi@lemmy.world ⁨4⁩ ⁨days⁩ ago
  
  Caddy only allows private IP ranges
  
  Do you mind telling me more about this? How does that work; a VPN?
  
  source
  - q7mJI7tk1@lemmy.world ⁨4⁩ ⁨days⁩ ago
    Sure, so I use Caddy as a reverse proxy for all my subdomains, the public ones direct straight to whatever service(s) are on IP:port etc, then the private ones only allow private IP ranges of which one is my VPN subnet, therefore only allowing LAN and VPN access. I then also have a section for each of the private subdomains with Authelia authentication which is omitted here:
    
    (allowed) { @allowed client_ip 192.168.1.0/24 192.168.10.0/24 192.168.20.0/28 } sub.domain.com { import allowed handle @allowed { reverse_proxy 192.168.80.8:8080 } handle { abort } }
    
    source
chirping@infosec.pub ⁨4⁩ ⁨days⁩ ago
Some of these you’re already doing, but writing a complete* list. *almost garuanteed not to be complete, suggestions welcome

Have everything behind the same reverse proxy, so that you have only one endpoint to worry about. Run it through ssllabs or similar to check your config.

On your reverse proxy, add one or more layers of authentication if possible. Many possibilities here: If one app supports client certificates, while another has limited capabilities, you could probably tie together something where IPs are whitelisted to the ither services based on that certificate auth.

Geoblock all countries you won’t be accessing from

crowdsec is pretty nice, this detects/blocks threats. kinda like fail2ban but on steroids.

if you use one of those 5$/month VPSes, with a VPN tunnel to your backend services, that adds one layer of “if it’s compromised, they’re not in your house”.

lastly consider if these things need to be publically avilable at all. I’m happy with 95% of my services only being available through Tailscale (mesh VPN, paid service with good enough free tier, open source+free alternatives available), and I’ve got tailscale on all my devices
source
- a_fancy_kiwi@lemmy.world ⁨4⁩ ⁨days⁩ ago
  check
  
  check
  
  check
  
  I saw someone else recommend crowdsec. I’ll look into it, thanks
  
  if you use one of those 5$/month VPSes, with a VPN tunnel to your backend services, that adds one layer of “if it’s compromised, they’re not in your house”.
  
  I’ve heard this mentioned before but I don’t really understand how this works in practice. If the VPS was compromised, couldn’t they use the VPN to then connect to my home?
  
  source
  - skysurfer@lemmy.world ⁨1⁩ ⁨day⁩ ago
    I set the VPN tunnel from the VPS to deny everything to the internal network by default, then put the services that need to be accessed on the allow list in the firewall. So the VPN endpoint from the VPS can only hit the very specific IPs/ports/protocols that were explicitly allowed. There is still the possibility of a compromise chain of VPS->service->container/VM->hypervisor->internal network access, but I feel comfortable with those layers.
    
    You could also setup an IDS such as Snort to pick up on that exploit traffic between the services and internal VPN endpoint if extra security is necessary on top of fail2ban and log alerts on the VPS.
    
    source