Kubernetes API Load-Balancer using HAProxy

References

Introduction

For a simple Kubernetes cluster, with perhaps just a single master node, pointing your kubectl configuration directly at the master node is not a problem, and in fact is usually the standard configuration. When you have a HA (High Availability) cluster though, pointing your kubectl configuration at one node could be problematic if that particular master node has failed or is down for maintenance. While it is relatively simple to change the IP in the kubectl configuration to that of a servicable master node, a better way to set things up is to use a load-balancer in front of the Kubernetes cluster, and have it control access to all of the master nodes of the cluster.

In the case of my homelab, I've set up HAProxy under docker to act as my Kubernetes API load-balancer for my k3s based cluster.

Docker-Compose File

Nothing too special about this docker-compose file. As I often do, I chose to explicitly set the image version being used. I've been bitten by using the ':latest' tag in the past when upstream image updates mysteriously break existing deployments.

$ cat docker-compose.yml

---
version: '3'

services:

  haproxy:
    container_name: haproxy
    image: haproxytech/haproxy-alpine:2.4.7
    volumes:
      - ./config:/usr/local/etc/haproxy:ro
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/Toronto
    restart: unless-stopped
    ports:
        - "6443:6443"
        - "8404:8404"

# EOF

HAProxy Config

What follows is a fairly straightforward haproxy.cfg file, although I did have to dig through a few sources and consolidate them into a configuration that worked for me.

$ cat config/haproxy.cfg

global
  stats socket /var/run/api.sock user haproxy group haproxy mode 660 level admin expose-fd listeners
  log stdout format raw local0 info

defaults
  log global
  mode http
  option  httplog
  option  dontlognull
  timeout client 10s
  timeout connect 5s
  timeout server 10s
  timeout http-request 10s

frontend stats
  bind *:8404
  stats enable
  stats uri /
  stats refresh 10s

frontend k8s-api
    bind *:6443
    mode tcp
    option tcplog
    option forwardfor
    default_backend k8s-api

backend k8s-api
    mode tcp
    option ssl-hello-chk
    option log-health-checks
    default-server inter 10s fall 2
    server node-1-rpi4 192.168.7.51:6443 check
    server node-2-lxc  192.168.7.52:6443 check
    server node-3-lxc  192.168.7.53:6443 check

Adding HAProxy IP to k3s SAN

After creating the haproxy.cfg and starting the haproxy container, the next step is to change the endpoint IP to that of the haproxy server in your kubeconfig file. When you do this, you're likely to receive the following error;

$ kubectl get nodes
Unable to connect to the server: x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.7.51, 192.168.7.52, 192.168.7.53, not 192.168.7.32

You have to add the haproxy IP to the k3s.service file, in the ExecStart line. This will need to be done on all master nodes. The ExecStart line should end up looking like this;

- /etc/systemd/system/k3s.service clip

------------------

ExecStart=/usr/local/bin/k3s \
    server \
        '--disable=traefik' \
        '--disable=servicelb' \
        '--tls-san=192.168.7.32' \

------------------

After making this change, run the following to get the change to take effect;

# systemctl daemon-reload
# systemctl restart k3s
# curl -vk --resolve 192.168.7.32:6443:127.0.0.1 https://192.168.7.32:6443/ping

After this, run this check on a workstation (use the original endpoint IP in the kubeconfig);

$kubectl -n kube-system get secret k3s-serving -o yaml

If everything went well, the haproxy IP should now show up in the list of "listener.cattle.io" entries. At this point, the endpoint IP in the kubeconfig file should be able to be changed to that of the haproxy server.

This option can be specified at k3s installation time by specifying "–tls-san 192.168.7.32" on the installation command line.

Statistics Web Page

HAProxy has a very nice statistics page that can also be enabled in the haproxy.cfg.

Created: 2021-10-14 14:56
Last update: 2021-10-22 13:10