I am running a bare metal Kubernetes cluster on k3s with Kube-VIP and Traefik. This works great for services that use SSL/TLS as Server Name Indication (SNI) can be used to reverse proxy multiple services listening on the same port. Consequently, getting Traefik to route multiple web servers receiving traffic on ports 80 or 443 is not a problem at all. However, I am stuck trying to accomplish the same thing for services that just use TCP or UDP without SSL/TLS since SNI is not included in TCP or UDP traffic.
I tried to setup Forgejo where clients will expect to use commands like git clone git@my.forgejo.instance.... which would ultimately use SSH on port 22. Since SSH uses TCP and Traefik supports TCPRoutes, I should be able to route traffic to Forgejo’s SSH entry point using port 22, but I ran into an issue where the SSH service on the node would receive/process all traffic received by the node instead of allowing Traefik to receive the traffic and route it. I believe that I should be able to change the port that the node’s SSH service is listening on or restrict the IP address that the node’s SSH service is listening on. This should allow Traefik to receive the traffic on port 22 and route that traffic to Forgejo’s SSH entry point while also allowing me to SSH directly into the node.
However, even if I get that to work correctly, I will run into another issue when other services that typically run on port 22 are stood up. For example, I would not be able to have Traefik reverse proxy both Forgejo’s SSH entry point and an SFTP’s entry point on port 22 since Traefik would only be able to route all traffic on port 22 to just one service due to the lack of SNI details.
The only viable solution that I can find is to only run one service’s entry point on port 22 and run each of the other services’ entry points on various ports. For instance, Forgejo’s SSH entry point could be port 22 and the SFTP’s entry point could be port 2222 (mapped to the pod’s port 22). This would require multiple additional ports be opened on the firewall and each client would need its configuration and/or commands modified to connect to the service’s a non-standard port.
Another solution that I have seen is to use other services like stunnel to wrap traffic in TLS (similar to how HTTPS works), but I believe this will likely lead to even more problems than using multiple ports as every client would likely need to be compatible with those wrapper services.
Is there some other solution that I am missing? Is there something that I could do with Virtual IP addresses, multiple load balancer IP addresses, etc.? Maybe I could route traffic on Load_Balancer#1_IP_Address:22 to Forgejo’s SSH entry point and Load_Balancer#2_IP_Address:22 to SFTP’s entry point?
tl;dr: Is it possible to host multiple services that do not use SSL/TLS (ie: cannot use SNI) on the same port in a single Kubernetes cluster without using non-standard ports and port mapping?
In short, you need a reverse-proxy + traffic segregation with domain names (SNI).
I don’t remember much about ingresses, but this can be super easy to set up with Gateway API (I’m looking at it right now).
Basically, you can set up
sftp.my.domain/sshto192.168.1.40:22,sftp.my.domain/sftpto192.168.1.40:121(for example). Same with Forgejo,forgejo.my.domain/sshwill point to192.168.1.50:22andforgejo.my.domain/guiwill point to192.168.1.50:443.The Gateway API will simply send it over to the right k8s service.
About your home network: I think you could in theory open up a DMZ and everything should work. I would personally use a cheap VPS as a VPN server and NAT all traffic through it. About traffic from your router maintaining the SNI, that’s a different problem depending on your network setup. Yes, you’ll have to deal with port-mapping because at the end of the day, even Gateway API is NodePort-esque when exposing traffic outside.
I am comfortable routing traffic via domain name through a reverse proxy. I am doing that via Traefik and can setup rules so that different sub-domains, domains, and/or path is routed to the appropriate end point (IP address and port). The issue is that k3s does not receive that information for SSH traffic since SNI (ie: the sub-domain, domain, etc.) is not included in SSH traffic. If SSH traffic provided SNI information, this issue would be much less complicated as I would only need to make sure that the port 22 traffic intended for k3s did not get processed by SSH or any other service on the node.
If I were to setup a DMZ, I think I would need to setup one unique public IP per SSH service. So I would need to create a public DNS record for sftp.my.domain to VPS#1’s public IP and ssh.forgejo.my.domain to VPS#2’s public IP. Assuming both VPS#1 and VPS#2 have some means of accessing the internal network (eg: VPN) then I could port forward traffic received by VPS#1 on port 22 to 192.168.1.40:22 and traffic received by VPS#2 on port 22 to 192.168.1.50:22. I had not considered this and based on what I have seen so far, I think that this would be the only solution to allow external traffic to access these services using the normal port 22 when there is more than one service in k3s expecting traffic on port 22 (or any other port that receives traffic without SNI details).
If you can only use port 22 for multiple SSH endpoints (for example), then yes your going to need multiple IPs. Or Port-mapping as a compromise