K3S Nodes in LXD Containers
The page describes the process to enable running k3s nodes in an LXD container. The benefits of running an application under an LXD container instead of a virtual machine should be clear. No virtualization overhead, better deployment density, configuration flexibility; these are only a few examples.
Container storage can't be on a ZFS or BTRFS storage pool
It seems that k3s has issues allocating storage when the backing storage for an LXD container is a ZFS or BTRFS storage pool. The simplest way to solve this is to make a new pool of type LVM, and use that for k3s LXD containers. You could also use a DIR type storage pool, but be aware that there are performance issues and limitations with DIR based storage pools.
$ lxc storage create k3s lvm size=50GiB
New profile for k3s containers
A number of configuration parameters need to be added to the k3s LXD container. The easiest way to do this is by using a profile. As we're using a custom pool, and a bridged profile, we'll create a profile that encapsulates both the custom settings required for a k3s node, as well as the bridged network configuration.
$ cat k3s.cnf config: security.nesting: "true" security.privileged: "true" limits.cpu: "2" limits.memory: 4GB limits.memory.swap: "false" linux.kernel_modules: overlay,nf_nat,ip_tables,ip6_tables,netlink_diag,br_netfilter,xt_conntrack,nf_conntrack,ip_vs,vxlan raw.lxc: | lxc.apparmor.profile = unconfined lxc.cgroup.devices.allow = a lxc.mount.auto=proc:rw sys:rw lxc.cap.drop = description: Profile settings for a bridged k3s container devices: eth0: name: eth0 nictype: bridged parent: br0 type: nic kmsg: path: /dev/kmsg source: /dev/kmsg type: unix-char root: path: / pool: k3s type: disk name: k3s used_by:
To create the new profile, execute the following commands;
$ lxc profile create k3s $ lxc profile edit k3s <k3s.cnf
Packages to add to container after launch
This step is partially unique to my setup. If your k3s nodes are running Ubuntu then you'll probably also require the apparmor-utils package.
- need to add the following packages to get k3s running
- curl (to install k3s)
- nfs-common (to access nfs storage)
- apparmor-utils (if running Ubuntu on the k3s node container)
At this point the LXD container should be ready for deploying a k3s node using the standard procedure.
Created: 2021-10-07 17:21
Last update: 2023-03-11 11:01