Kubernetes API Load-Balancer using HAProxy
For a simple Kubernetes cluster, with perhaps just a single master node, pointing your kubectl configuration directly at the master node is not a problem, and in fact is usually the standard configuration. When you have a HA (High Availability) cluster though, pointing your kubectl configuration at one node could be problematic if that particular master node has failed or is down for maintenance. While it is relatively simple to change the IP in the kubectl configuration to that of a servicable master node, a better way to set things up is to use a load-balancer in front of the Kubernetes cluster, and have it control access to all of the master nodes of the cluster.
In the case of my homelab, I've set up HAProxy under docker to act as my Kubernetes API load-balancer for my k3s based cluster.
Nothing too special about this docker-compose file. As I often do, I chose to explicitly set the image version being used. I've been bitten by using the ':latest' tag in the past when upstream image updates mysteriously break existing deployments.
$ cat docker-compose.yml --- version: '3' services: haproxy: container_name: haproxy image: haproxytech/haproxy-alpine:2.4.7 volumes: - ./config:/usr/local/etc/haproxy:ro environment: - PUID=1000 - PGID=1000 - TZ=America/Toronto restart: unless-stopped ports: - "6443:6443" - "8404:8404" # EOF
What follows is a fairly straightforward haproxy.cfg file, although I did have to dig through a few sources and consolidate them into a configuration that worked for me.
$ cat config/haproxy.cfg global stats socket /var/run/api.sock user haproxy group haproxy mode 660 level admin expose-fd listeners log stdout format raw local0 info defaults log global mode http option httplog option dontlognull timeout client 10s timeout connect 5s timeout server 10s timeout http-request 10s frontend stats bind *:8404 stats enable stats uri / stats refresh 10s frontend k8s-api bind *:6443 mode tcp option tcplog option forwardfor default_backend k8s-api backend k8s-api mode tcp option ssl-hello-chk option log-health-checks default-server inter 10s fall 2 server node-1-rpi4 192.168.7.51:6443 check server node-2-lxc 192.168.7.52:6443 check server node-3-lxc 192.168.7.53:6443 check
Adding HAProxy IP to k3s SAN
After creating the haproxy.cfg and starting the haproxy container, the next step is to change the endpoint IP to that of the haproxy server in your kubeconfig file. When you do this, you're likely to receive the following error;
$ kubectl get nodes Unable to connect to the server: x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.7.51, 192.168.7.52, 192.168.7.53, not 192.168.7.32
You have to add the haproxy IP to the k3s.service file, in the ExecStart line. This will need to be done on all master nodes. The ExecStart line should end up looking like this;
- /etc/systemd/system/k3s.service clip ------------------ ExecStart=/usr/local/bin/k3s \ server \ '--disable=traefik' \ '--disable=servicelb' \ '--tls-san=192.168.7.32' \ ------------------
After making this change, run the following to get the change to take effect;
# systemctl daemon-reload # systemctl restart k3s # curl -vk --resolve 192.168.7.32:6443:127.0.0.1 https://192.168.7.32:6443/ping
After this, run this check on a workstation (use the original endpoint IP in the kubeconfig);
$kubectl -n kube-system get secret k3s-serving -o yaml
If everything went well, the haproxy IP should now show up in the list of "listener.cattle.io" entries. At this point, the endpoint IP in the kubeconfig file should be able to be changed to that of the haproxy server.
This option can be specified at k3s installation time by specifying "–tls-san 192.168.7.32" on the installation command line.
Statistics Web Page
HAProxy has a very nice statistics page that can also be enabled in the haproxy.cfg.
Created: 2021-10-14 14:56
Last update: 2021-10-22 13:10