Contents

• References
• Introduction
• Container storage can't be on a ZFS or BTRFS storage pool
• New profile for k3s containers
• Conclusion

References

• https://github.com/ruanbekker/k3s-on-lxd
• https://github.com/corneliusweig/kubernetes-lxd

Introduction

The page describes the process to enable running k3s nodes in an LXD container. The benefits of running an application under an LXD container instead of a virtual machine should be clear. No virtualization overhead, better deployment density, configuration flexibility; these are only a few examples.

Container storage can't be on a ZFS or BTRFS storage pool

It seems that k3s has issues allocating storage when the backing storage for an LXD container is a ZFS or BTRFS storage pool. The simplest way to solve this is to make a new pool of type LVM, and use that for k3s LXD containers. You could also use a DIR type storage pool, but be aware that there are performance issues and limitations with DIR based storage pools.

# mkdir /opt/lxd_pool2

$ lxc storage create k3s lvm size=50GiB

New profile for k3s containers

A number of configuration parameters need to be added to the k3s LXD container. The easiest way to do this is by using a profile. As we're using a custom pool, and a bridged profile, we'll create a profile that encapsulates both the custom settings required for a k3s node, as well as the bridged network configuration.

$ cat k3s.cnf 

config:
  security.nesting: "true"
  security.privileged: "true"
  limits.cpu: "2"
  limits.memory: 4GB
  limits.memory.swap: "false"
  linux.kernel_modules: overlay,nf_nat,ip_tables,ip6_tables,netlink_diag,br_netfilter,xt_conntrack,nf_conntrack,ip_vs,vxlan
  raw.lxc: |
    lxc.apparmor.profile = unconfined
    lxc.cgroup.devices.allow = a
    lxc.mount.auto=proc:rw sys:rw
    lxc.cap.drop =
description: Profile settings for a bridged k3s container
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  kmsg:
    path: /dev/kmsg
    source: /dev/kmsg
    type: unix-char
  root:
    path: /
    pool: k3s
    type: disk
name: k3s
used_by:

To create the new profile, execute the following commands;

$ lxc profile create k3s
$ lxc profile edit k3s <k3s.cnf

Packages to add to container after launch

This step is partially unique to my setup. If your k3s nodes are running Ubuntu then you'll probably also require the apparmor-utils package.

• need to add the following packages to get k3s running
  • curl (to install k3s)
  • nfs-common (to access nfs storage)
  • apparmor-utils (if running Ubuntu on the k3s node container)

Conclusion

At this point the LXD container should be ready for deploying a k3s node using the standard procedure.

....................
(created: 2021-10-07, last modified: 2022-01-18 at 10:16:36)