achim

@nibbs

You’re welcome — happy tinkering. 🙂

@nibbs

You have already found the core issue: a ban on VM2 is too late.

VM2 does see the real client address in the HTTP logs if NPM passes X-Forwarded-For correctly, but at the network layer every connection to VM2 still originates from NPM on VM1. Therefore an nftables/UFW rule on VM2 can only block NPM — which is not exactly the intended security feature.

For your setup, I would put the actual enforcement at the ingress point:

• Run Fail2Ban on VM1.
• Parse the NPM access/error logs there.
• Let Fail2Ban add/remove bans locally on VM1.
• Ensure NPM logs the real client IP, and only trust forwarded-IP headers from proxies you actually control.
• Keep VM2 restricted so it accepts service traffic only from VM1/NPM where possible.

Having Fail2Ban on VM2 execute remote firewall actions on VM1 via SSH can work, but it is basically building a small, brittle distributed ban system yourself: SSH keys, narrowly scoped sudo rules, reliable unban actions, error handling, and so on.

Alternative:
If you want detection from several VMs/services but enforcement centrally at NPM, CrowdSec is a more natural fit. Run agents where the relevant logs live, use a central LAPI, and run a bouncer at VM1/NPM. Then the components are designed to exchange decisions instead of hoping two independent Fail2Ban installations telepathically coordinate.

Also: do not use set_real_ip_from 0.0.0.0/0 just to make X-Forwarded-For work. That turns a client-supplied header into an IP-spoofing API.

#SelfHosting #ReverseProxy #Fail2Ban #CrowdSec #NginxProxyManager

@observantTrapezium

Sounds promising — have fun testing it 😆

@observantTrapezium

I just saw your other reply on Lemmy regarding Headscale.

Headscale should already solve that use case without putting mTLS in front of every service.

By default, a Headscale tailnet is rather permissive, but you can load a policy and use Grants (or ACLs, though Grants are the preferred direction) to explicitly allow only the connections you want. A phone does not need access to “the whole tailnet” just because it is enrolled.

For example, each user/device group could be allowed to reach only:

• the Immich host on its HTTPS port;
• the CalDAV/CardDAV host on 443;
• optionally the Matrix client endpoint on 443;

…and nothing else. No SSH, no databases, no admin UIs, no access to other tailnet nodes. Start with an empty grants policy — which denies all tailnet traffic — then add narrowly scoped allow rules for the relevant source devices and service destinations.

That also means the client can simply connect when it needs those services; it does not need broad, permanent network access merely because the Headscale app is installed. And if a device is lost, removing that node from Headscale cuts off its network access immediately.

So I would still favour: private services behind Headscale plus a deny-by-default policy, instead of managing a separate mTLS PKI for several native mobile apps. The latter is viable, but it is solving device/network admission at the application TLS layer when you already operate a tool designed to do it at the network layer. Headscale explicitly supports access control policies for restricting traffic between nodes, and policies can deny all traffic by default until specific access is granted.

@observantTrapezium

I think mTLS is probably overkill here.

For a few unmanaged phones using native apps, you would be adding a second PKI lifecycle on top of the normal application credentials: issuing per-device keys, securely importing them into iOS/Android, handling app-specific certificate selection, expiry, replacement, revocation after loss/reset, and explaining all of that to the users. It works — but it is a lot of machinery for a small private setup.

A WireGuard-style overlay network seems like the more practical solution. WireGuard directly, Tailscale, Headscale, NetBird, or similar would let you make Immich and CalDAV/CardDAV private-only services. Then a new public-facing exploit in one of those services is much less likely to become an immediate “drop everything and patch it right now” event, because there is no publicly reachable login or API endpoint in the first place.

The Matrix part needs one important clarification, though: do you want federation with other Matrix homeservers?

If yes, Matrix cannot simply be made private in the same way. Other homeservers need to reach the federation API; by default that is port 8448, although Matrix delegation can direct federation to another host/port. Matrix clients and federation traffic can also be separated: clients normally use 443, while server-to-server federation defaults to 8448.

So I would split it roughly like this:

• Immich: private overlay/VPN only.
• CalDAV/CardDAV: private overlay/VPN only.
• Matrix, no federation: private overlay/VPN only.
• Matrix, federated: expose only the minimal, dedicated federation endpoint publicly; keep admin interfaces and anything else private. Client access could still be via the overlay, but that has UX implications for mobile push/background connectivity.

mTLS is a reasonable choice when a service must remain publicly reachable but should accept requests only from a tightly controlled set of devices. In your case, putting the services behind a private network seems both simpler and more effective at reducing exposed attack surface.

Also, “I can distribute keystores manually” is exactly the point where this tends to look easy — right up to the first lost phone, OS reset, certificate expiry, or app that suddenly stops offering the certificate picker because mobile ecosystems enjoy making infrastructure administration a lifestyle choice.

@observantTrapezium

mTLS can be a very good extra gate for a small, controlled device set. But “some services”, “smartphones” and “Apache reverse proxy” is not enough to recommend it responsibly.

The decisive questions are:

• Which services are we talking about: a normal web UI in a mobile browser, native apps, APIs, WebDAV, SSH-like administration, something else?
• Are those personally managed devices, or devices belonging to multiple users?
• iOS, Android, both — and which browsers/apps?
• Is there MDM, or would certificate enrollment, replacement, revocation and renewal all be manual?
• What happens when a phone is lost, reset, sold, or its private key leaks?
• Does each device get its own certificate, or would the same .p12 be copied around? (Please do not do the latter.)
• Is the real goal “no public login page”, “device authentication”, or simply private remote access?

For a handful of your own devices, per-device mTLS certificates can be perfectly reasonable. For a mixed fleet of smartphones without MDM, the operational overhead is often the actual attack surface: secure initial delivery of the PKCS#12 bundle, private-key protection, expiration, rotation, revocation, backups, and users selecting the right certificate.

Also: mTLS is a gate, not a replacement for normal authentication and authorization. I would usually still keep the application login, use individual device certificates with a private CA, short-ish validity, documented revocation, and rate limits. Otherwise you have merely replaced password brute force with “whoever extracted or copied the client private key gets through.”

Depending on the service, a WireGuard/Tailscale-style private network or an identity-aware proxy may be less painful on phones — but that is impossible to judge without knowing the actual services and client workflow. Certificate-based mobile access is feasible, yet the setup and lifecycle vary materially across platforms and applications.

So yes: the missing basics are not a minor omission; they are the question.

@GatesMcBalmer

https://github.com/alexta69/metube