Proxmox LXCs
Containers are a lightweight alternative to fully virtualized machines (VMs). They use the kernel of the host system that they run on, instead of emulating a full operating system (OS). This means that containers can access resources on the host system directly.
The runtime costs for containers is low, usually negligible. However, there are some drawbacks that need be considered:
Only Linux distributions can be run in Proxmox Containers. It is not possible to run other operating systems like, for example, FreeBSD or Microsoft Windows inside a container.
Danger
For security reasons, access to host resources needs to be restricted. Therefore, containers run in their own separate namespaces. Additionally some syscalls (user space requests to the Linux kernel) are not allowed within containers.
LXCs configurations on 3 nodes Proxmox Cluster
K3S-master-01
K3S-master-02
Ansible
Alto ( docker )
K3S-master-03
K3S-admin
Configuration for K3S
I spent many hours grappling with the challenge of getting K3S to work on LXCs, and at this point, I'd like to extend my heartfelt thanks to Garrett Mills. His insights were invaluable. Now, let's dive into how you can successfully run K3S on LXC.
Warning
LXC K3S
arch: amd64
cores: 4
features: fuse=1
hostname: lxc-k3s-ct-ready
memory: 4096
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=BC:24:11:78:33:4B,ip=dhcp,type=veth
ostype: ubuntu
rootfs: Storage:basevol-210-disk-0,size=50G
swap: 0
template: 1
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
lxc.mount.auto: "proc:rw sys:rw"
- Make sure you add above highlighted lines to your LXC config.
- Note: It's important that the container is stopped when you try to edit the file, otherwise Proxmox's network filesystem will prevent you from saving it.
- In order, these options (1) disable AppArmor, (2) allow the container's cgroup to access all devices, (3) prevent dropping any capabilities for the container, and (4) mount /proc and /sys as read-write in the container.
- Next, we need to publish the kernel boot configuration into the container. Normally, this isn't needed by the container since it runs using the host's kernel, but the Kubelet uses the configuration to determine various settings for the runtime, so we need to copy it into the container. To do this, first start the container using the Proxmox web UI, then run the following command on the Proxmox host:
- Execute:
- Finally, in each of the containers, we need to make sure that /dev/kmsg exists. Kubelet uses this for some logging functions, and it doesn't exist in the containers by default. For our purposes, we'll just alias it to /dev/console. In each container, create the file /usr/local/bin/conf-kmsg.sh with the following contents:
7.This script symlinks /dev/console as /dev/kmsg if the latter does not exist. Finally, we will configure it to run when the container starts with a SystemD one-shot service. Create the file /etc/systemd/system/conf-kmsg.service with the following contents:
8.Finally, enable the service by running the following: