Comment on Multiple Kubernetes Services Using Same Port Without SNI
wireless_purposely832@lemmy.world 3 weeks agoI guess I need to dig in a little deeper. I am currently only using Kube-VIP to provide a single IP address for the control plane. I think I may have it configured wrong though since that same IP address is the single load balancer IP used by Traefik.
I have struggled finding good documentation, hints, tutorials, etc. setting up Kube-VIP with Virtual IPs. Is there anything that you are aware of that might provide some assistant in setting that up correctly?
simonmicro@programming.dev 3 weeks ago
Okay, I’ll try explaining it. Yes there is especially for this very little documentation, so… Yeah.
You start by installing kube-vip into your cluster. Make sure to configure it correctly, so the uplink interface of you workers is being used for the vip, but not e.g. internal ones. Then you create a new loadbalancer-service in k8s, on which you also set the externalIP field. Maybe you also needed some annotations, I’m not so sure anymore. Theoretically that’s it. Now one of you nodes will start serving the service-port under the vip. The service may target everything, not only Traefik.
There is one more thing: Another field on the LB-service allows you to disable any kind of internal routing via your CNI, so you will even be able to see the real source IPv4 of your clients.
I WILL UPDATE THIS SOON WITH MORE SPECIFIC VALUES, HANG ON
wireless_purposely832@lemmy.world 3 weeks ago
I am unsure if I understood everything correctly, but I believe I am already doing everything that you mentioned. I followed the Kube-VIP’s ARP daemonset’s documentation. The leader election works. I am not using Kube-VIP for load balancing though. Instead, I am using Traefik, which is using the same IP address that was assigned to the control plane during both k3s’s and Kube-VIP’s setup. However, I am unable to get any additional VIP addresses to properly route to Traefik.
Even if I did get the additional VIP addresses working, I think I still have one last issue to overcome. I can control the local network’s DNS so that service#1 is assigned VIP#1 and service#2 assigned VIP#2. However, how would this be handled for traffic received externally? If the external/public DNS has service#1 and service#2 assigned to the network’s public IP address, both service’s traffic would be received by the router/firewall on port 22. The router/firewall could forward traffic on port 22 to (presumably) a single IP address, which would only allow service#1 or service#2 (but not both) to receive traffic publicly, correct?
simonmicro@programming.dev 3 weeks ago
Ah yes, I see. Because TCP has no SNI built-in this is not really possible.
You could try IPv6, as within even a single /64 routable prefix you can choose the address section freely. Also take a look at overlay-vpn solutions like Netbird: They allow you to offer you multiple clients, which you could use to assign multiple IPv4 to your server and then routing them differently…
wireless_purposely832@lemmy.world 3 weeks ago
I had not thought about using IPv6 for this. It’s definitely something that I would need to research more as I know that this would expose my attack surface and may require an overhaul of the network (or at least a very thorough review).
I’m not sure I understand the concern about Traefik. I am using it as a reverse proxy and forcing HTTPS for all applicable services (which unfortunately does not apply to this particular situation). I am honestly a little confused about the control plane, tls-san, gateway, load balancer, ingress, etc. and how they all work together. I may not be using Traefik as the Load Balancer and instead have Kube-VIP as the LoadBalancer. I did not configure Kube-VIP any particular way for Load Balancing, but I did configure Traefik with a few Load Balancer specific options. When I tried to setup Kube-VIP with the additional IP addresses for load balancing, I was unable to get k3s to work correctly so I assumed that Traefik was my Load Balancer instead of Kube-VIP.