Comment on Multiple Kubernetes Services Using Same Port Without SNI
Findmysec@infosec.pub 3 weeks agoIngress controllers like Traefik come across as LB services to IPAM modules like MetalLB (I’ve never used Kube-VIP but I suppose it’s the same story). These plug-ins assign IP addresses to these LB services.
You can assign a specific IP to an instance of an “outward-facing route” with labels. I don’t remember technical terms relevant to Ingresses because I’ve been messing with the Gateway API recently.
wireless_purposely832@lemmy.world 3 weeks ago
That all makes sense and tried setting it up that way but could not get it to work. I am not sure if it was an issue with my network, k3s, Kube-VIP, or Traefik (or some combination of them). I will try getting it to work again.
Even if I do though, I would run into an issue if I publicly exposed these services (I understand there are security implications of doing so). How would I route traffic received externally/publicly on port 22 to more than one IP address? I think I would only be able to do this for local/internal traffic by managing the local DNS.
Findmysec@infosec.pub 3 weeks ago
You’d receive traffic on IP:PORT, that’s segregation right there. Slap on a DNS name for convenience.
I might have my MetalLB config laying around somewhere (it’s super easy, I copied most of it from their website), I can probably paste it here if you’d like.
wireless_purposely832@lemmy.world 3 weeks ago
Maybe I was not clear, but I do not think that you understand what I was trying to say with the second part of my last message.
Assume that multiple VIPs are setup and there is a load balancer IP for the SFTP entry point (eg: 192.168.1.40:22) and a different load balancer IP for the Forgejo SSH entry point (eg: 192.168.1.50:22). My local DNS can be setup so that sftp.my.domain points to 192.168.1.40 and ssh.forgejo.my.domain points to 192.168.1.50. When I make a request within my network, the DNS lookup will appropriately route sftp.my.domain:22 to 192.168.1.40:22 and ssh.forgejo.my.domain:22 to 192.168.1.50. I believe this is what you are recommending and exactly what I want. I will need to get the multiple VIP part of this setup worked out so I can do this.
However, this will not work when the traffic is received from outside of my network even if the above configuration is setup correctly. If you were to try to connect to either sftp.my.domain:22 or ssh.forgejo.my.domain:22, your traffic would be routed to my public IP address. My firewall/router would receive the traffic on port 22 and port forward the traffic to the single IP address assigned to that port forwarding rule. When k3s receives the traffic from my firewall/router, k3s will not have any SNI information (ie: it will not know whether you were using sftp.my.domain or ssh.forgejo.my.domain - or any other domain for that matter). Even if I were able to setup multiple port forwarding rules for port 22 on the firewall/router, I would still be unable to appropriately route the traffic because the firewall/router would also not know if the traffic was intended for sftp.my.domain or ssh.forgejo.my.domain. As a result, at most you would only be able to use one of the services because external traffic for both sftp.my.domain and ssh.forgejo.my.domain will be routed to the same IP address and k3s would have no idea what domain (if any) is being used.
There are a few solutions (eg: use different ports for each SSH or non-TLS trafficked service, wrap the SSH traffic in TLS to give k3s SNI information to route traffic to the appropriate endpoint, configure SSH on the node to route traffic to the appropriate IP address based on SSH user, require each client to use the local network or VPN, etc.), but none of them are as seamless and easy as routing TLS traffic which can use SNI information.
Findmysec@infosec.pub 3 weeks ago
In short, you need a reverse-proxy + traffic segregation with domain names (SNI).
I don’t remember much about ingresses, but this can be super easy to set up with Gateway API (I’m looking at it right now).
Basically, you can set up
sftp.my.domain/sshto192.168.1.40:22,sftp.my.domain/sftpto192.168.1.40:121(for example). Same with Forgejo,forgejo.my.domain/sshwill point to192.168.1.50:22andforgejo.my.domain/guiwill point to192.168.1.50:443.The Gateway API will simply send it over to the right k8s service.
About your home network: I think you could in theory open up a DMZ and everything should work. I would personally use a cheap VPS as a VPN server and NAT all traffic through it. About traffic from your router maintaining the SNI, that’s a different problem depending on your network setup.