Many Network Interfaces per VM/CT - Good Practice?

Submitted ⁨⁨6⁩ ⁨months⁩ ago⁩ by ⁨anon2963@infosec.pub⁩ to ⁨selfhosted@lemmy.world⁩

I am currently setting up a Proxmox box that has the usual selfhosted stuff (Nextcloud, Jellyfin, etc) and I want all of these services in different containers/VMs. I am planning to start sharing this with family/friends who are not tech savvy, so I want excellent security.

I was thinking of restricting certain services to certain VLANs, and only plugging those VLANs into the CT/VMs that need them.

Currently, each CT/VM has a network interface (for example eth0) which gives them internet access (for updates and whatnot) and an interface that I use for SSH and management (for example eth1). These interfaces are both on different VLANs and I must use Wireguard to get onto the management network.

I am thinking of adding another interface just for “consumption” which my users would get onto via a separate Wireguard server, and they would use this to actually use the services.

I could also add another network just to connect to an internal NFS server to share files between CT/VMs, and this would have its own VLAN and require an additional interface per host that connects to it.

I have lots of other ideas for networks which would require additional interfaces per CT/VM that uses them.

From my experience, using a “VLAN-Aware” bridge and assigning VLANs per interface via the GUI is best practice. However, Proxmox does not support multiple VLANs per interface using this method.

I have an IPv6-only network, so I could theoretically assign multiple IPs per interface. Then I would use Linux VLANs from within the guest OS. However, this is a huge pain and I do not want to do this. And it is less secure because a compromised VM/CT could change its VLAN tag itself.

I am asking if adding many virtual interfaces per CT/VM is good practice, or if there is a better way to separate internal networks. Or maybe I should rethink the whole thing and not use one network per use-case.

I am especially curious about performance impacts of multiple interfaces.

source

Comments

Sort:hotnew top

testfactor@lemmy.world ⁨6⁩ ⁨months⁩ ago
I wouldn’t let every VM have an interface into your management network, regardless of how you implement this. Your management network should be segregated with the ability to route to all the other VLANs with an appropriate firewall setup that only allows “related/established” connections back into it.

As for your services, having them on separate VLANs is fine, but it seems like you would benefit from having a reverse proxy to forward things to the appropriate VLAN, to reduce your management overhead.

But in general, having multiple interfaces per VM is fine. There shouldn’t be any performance hit or anything. But remember that if you have a compromised VM, it’ll be on any networks you give it an interface in, so minimizing that is key for security purposes. Ideally it would live in a VLAN that only has Internet access and/or direct access to your reverse proxy.

source
- Im_old@lemmy.world ⁨6⁩ ⁨months⁩ ago
  Having multiple interfaces in each vm can lead to issues with routing if you screw something up.
  
  Like you said I’d expose the services via reverse proxy in the public vlan, and enable ssh access on the firewall only from a jumpbox or the ip of your pc (or maybe the vlan you are in).
  
  source
  - testfactor@lemmy.world ⁨6⁩ ⁨months⁩ ago
    Yeah, it can for sure. Definitely worth mentioning. Gotta watch what interface is set as the default router, or you’re bound to have a bad time. That said, the same is true with his originally proposed solution of pushing a trunk port to the VM, so it’s not any worse in that regard.
    
    But yeah, full agreement on the correct solution. Keep it simple.
    
    source
  - pyrosis@lemmy.world ⁨6⁩ ⁨months⁩ ago
    I agree with this. The only vm I have that has multiple interfaces is an opnsense router vm heavily optimized for kvm to reach 10gb speeds.
    
    One of the interfaces beyond wan and lan is an interface that links to a proxmox services bridge. It’s a proxbridge I gave to a container and is just a gateway in opnsense. It points traffic destined for services directly at the container ip. It keeps the service traffic on the bridge instead of having to hit the physical network.
    
    source
  - realbadat@programming.dev ⁨6⁩ ⁨months⁩ ago
    Agreed, I prefer trunk with native to the vlan for services, each container that the reverse proxy will hit in its own vlan (or multiples for differing sets of services, but I can be excessive).
    
    I’d block any traffic initiated from that vlan to all others, and I’d also only allow the specific ports needed for the services. Then fully open initiated from the general internal vlan.
    
    source
Trainguyrom@reddthat.com ⁨6⁩ ⁨months⁩ ago
It really sounds like you need to dive into firewall rules. Generally you lean on your firewall to allow and restrict access to services. Probably the easiest place to start is to setup pfsense/opnsense since it has a really clean interface for setting up rules. And as you learn more about firewalls learning how subnetting works will allow for more efficient rules (for example, if you have 192.168.0.0/23 192.168.2.0/24 and 192.168.3.0/ 24 for your networks that you’re allowing traffic to/from you can just enter a firewall rule for 192.168.0.0/22)

source
- anon2963@infosec.pub ⁨6⁩ ⁨months⁩ ago
  Thank you, that is a very good point, I never thought of that. Just to confirm, best standard practice is for every connection, even as simple as a Nextcloud server accessing an NFS server, to go through the firewall?
  
  Then I could just have one interface per host but use Proxmox host ID as the VLAN so they are all unique. Then, I would make a trunk on the guest OPNsense VM. In that way it is a router on a stick.
  
  I was a bit hesitant to do firewall rules based off of IP addresses, as a compromised host could change its IP address. However, if each host is on its own VLAN, then I could add a firewall rule to only allow through the 1 “legitimate” IP per VLAN. The rules per subnet would still work though.
  
  I feel like I may have to allow a couple CT/VMs to communicate without going through the firewall simply for performance reasons. Has that ever been a concern for you? None of the routing or switching would be hardware accelerated.
  
  source
Decronym@lemmy.decronym.xyz ⁨6⁩ ⁨months⁩ ago
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

Fewer Letters More Letters

DNS Domain Name Service/System

IP Internet Protocol

NFS Network File System, a Unix-based file-sharing protocol known for performance and efficiency

[Thread #748 for this sub, first seen 15th May 2024, 03:15] [FAQ] [Full list] [Contact] [Source code]

source
pyrosis@lemmy.world ⁨6⁩ ⁨months⁩ ago
I use using docker networks but that’s me. They are created for every service and it’s easy to target the gateway. Just make sure DNS is correct for your hostnames.

Lately I’ve been optimizing remote services for reverse proxy passthru. Did you know that it can break streams momentarily and make your proxy work a little harder if your host names don’t match outside and in?

So in other words if you want full passthru of a tcp or udp stream to your server without the proxy breaking it then opening a new stream you would have to make sure the internal network and external network are using the same fqdn for the service you are targeting.

It actually can break passthru via sni if they don’t use the same hostname and cause a slight delay. Kinda matters for things like streaming videos. Especially if you are using a reverse proxy and the service supports quic or http2.

So a reverse proxy entry that simply passes without breaking the stream and resending it might ook like…

Obviously you would need to get the http port working on jellyfin and have ipv6 working with internal DNS in this example.

server { listen 443 ssl; listen [::]:443 ssl; # Listen on IPv6 address server_name jellyfin.example.net; ssl_certificate /path/to/ssl_certificate.crt; ssl_certificate_key /path/to/ssl_certificate.key; location / { proxy_pass https://jellyfin.example.net:8920; # Use FQDN ... } }
source
- raldone01@lemmy.world ⁨4⁩ ⁨months⁩ ago
  Full pass through has no advantage when my reverse proxy terminates ssl and internal services are http only right?
  
  Regardless of fqdn nginx has to decrypt and restream anyways.
  
  source
  - pyrosis@lemmy.world ⁨3⁩ ⁨months⁩ ago
    This would be correct if you are terminating ssl at the proxy and it’s just passing it to http. However, if you can enable SSL on the service it’s possible to take advantage of full passthru if you care about such things.
    
    source
    -> View More Comments

Fewer Letters	More Letters
DNS	Domain Name Service/System
IP	Internet Protocol
NFS	Network File System, a Unix-based file-sharing protocol known for performance and efficiency