Before I dive into Kubernetes The Hard Way and get my hands dirty by bringing up a new K8S cluster, I thought I’d create a few pages covering Kubernetes theory, architecture, and related concepts.

So, let’s start with the obvious first candidate: Kubernetes architecture.

K8S Architecture Overview

There are two types of Kubernetes nodes. Some nodes run applications, while others control when, how, and where those applications run. The first group of nodes, which runs applications (or containers), is called Worker nodes, while the second group consists of servers called Control Plane (or Master) nodes.

When it comes to the tasks each group performs, the workers’ responsibilities are fairly straightforward: they accept requests to run applications, spin up containers using the specified container runtime, and provide networking and storage. On the other hand, Control Plane nodes perform more complex tasks to ensure the requested state is achieved. Control Plane nodes need to:

• place containers on the appropriate worker nodes
• determine how to place those containers
• identify the most suitable worker node
• store information about each worker node and monitor its state and availability
• and more…

Let’s now look at the individual services that run on each group of hosts.

Components that run on Worker nodes

Worker nodes are responsible for running kubelet and the container runtime, as well as providing networking (CNI) and storage.

Kubelet is a service that runs on each worker node and manages all activities on that node:

• informs Control Plane nodes that a new worker node is ready to join the group
• listens for requests coming from Control Plane nodes, such as information of containers that will be placed on that node
• reports back to Control Plane nodes the status of each container (deployed, running, failed to run, destroyed, etc)

Of course, something needs to deploy containers. That responsibility belongs to the container runtime, such as containerd, rkt, or Docker.

Kube-proxy is another service that runs on Worker nodes. It ensures that containers can communicate with each other—for example, allowing a web server to send requests to a database server running in another container. Kube-proxy accomplishes this by maintaining networking rules using iptables.

Next, there are CNI plugins that provide… well… networking. More specifically, CNI plugins implement the Kubernetes networking model. There are many CNI plugins available, including Calico, Cilium, Flannel, and Weave.

An important note about networking: if a CNI plugin implements packet forwarding for Services itself and provides functionality equivalent to kube-proxy, then kube-proxy does not need to run on the nodes. In that case, kube-proxy becomes optional rather than required.

Lastly, worker nodes are also responsible for providing storage for running pods. Kubernetes supports a number of volume types:

• populating a configuration file based on a ConfigMap or a Secret
• providing some temporary scratch space for a Pod
• sharing a filesystem between two different containers in the same Pod
• sharing a filesystem between two different Pods (even if those Pods run on different nodes)
• durably storing data so that it stays available even if the Pod restarts or is replaced
• passing configuration information to an app running in a container, based on details of the Pod the container is - in (for example: telling a sidecar container what namespace the Pod is running in)
• providing read-only access to data in a different container image

Components that run on Control Plane nodes

There are four main services that run on Control Plane nodes.

The kube-apiserver is the primary management component and is responsible for orchestrating all operations within the cluster. It communicates with all other components on the Control Plane and also sends instructions to Worker nodes.

Next, there is the etcd cluster. etcd stores all cluster information in a key-value (KV) store.

Then, there is kube-scheduler, which is responsible for scheduling Pods. It watches for newly created Pods that have not yet been assigned to a worker node and determines the most suitable node for each Pod. Factors it considers include resource requirements, policy constraints, affinity rules, and more.

Lastly, kube-controller-manager runs different controller processes. Some examples of them are:

• Node controller: Responsible for noticing and responding when nodes go down.
• Job controller: Watches for Job objects that represent one-off tasks, then creates Pods to run those tasks to completion.
• EndpointSlice controller: Populates EndpointSlice objects (to provide a link between Services and Pods).
• ServiceAccount controller: Create default ServiceAccounts for new namespaces.

The following image was taken from the official Kubernetes architecture page:

I have also asked an AI image generator to visualise all components:

This post explores how to build QCOW2 images with Packer. Once built, they can be uploaded to different cloud providers and provision new instances.

This post is the first post in a series of Kubernetes The Hard Way. I built my own images to simplify the process of provisioning new Kubernetes clusters. I really did not want to repeat all pre-requisit steps every time a new host comes up (or I mess up some configs somewhere and therefore need to re-bootstrap a host).

All the steps bellow assume the Ubuntu 22.04.05 LTS release.

Qcow2 Packer all the way

Step 1: Install Packer

On a local machine:

sudo apt update
sudo apt install -y packer

Step 2: Set up the project and its structure

The project directory looks like this:

root_dir
|
|-- builds
   |
   |-- dev.pkrvars.hcl   
|
|-- files
   |
   |-- 99-custom.conf   
|
|-- http
   |
   |-- meta-data   
|
|-- packer.pkr.hcl
|-- variables.pkr.hcl
|-- user-data.tpl
|-- build_packer_image.sh

Step 3: the packer file

The main packer file is an HCL file, which in this case was ubuntu.pkr.hcl and it should be created with:

packer {
  required_plugins {
    qemu = {
      source  = "github.com/hashicorp/qemu"
      version = ">= 1.0.0"
    }
  }
}

# ---- QEMU SOURCE ----
source "qemu" "ubuntu" {
  iso_url      = "https://releases.ubuntu.com/22.04.5/ubuntu-22.04.5-live-server-amd64.iso"
  iso_checksum = "sha256:9bc6028870aef3f74f4e16b900008179e78b130e6b0b9a140635434a46aa98b0"

  output_directory = "output-${var.image_name}"
  vm_name          = "${var.image_name}.qcow2"

  disk_size   = "10G"
  format      = "qcow2"
  accelerator = "kvm"
  disk_interface = "virtio"
  net_device = "virtio-net"

  memory = 2048
  cpus   = 2

  ssh_username         = var.username
  ssh_password         = var.password
  ssh_timeout          = "20m"

  http_directory = "http"

  boot_wait = "20s"

  boot_command = [
    "<wait>",
    "c",
    "<wait>",
    "linux /casper/vmlinuz autoinstall ip=dhcp ds=nocloud-net\\;s=http://:/ ---",
    "<enter>",
    "initrd /casper/initrd",
    "<enter>",
    "boot",
    "<enter>"
  ]

  shutdown_command = "sudo shutdown -P now"
}

# ---- BUILD ----
build {
  sources = ["source.qemu.ubuntu"]

  provisioner "file" {
    source      = "files/99-custom.conf"
    destination = "/tmp/99-custom.conf"
  }

  # k8s setup
  provisioner "shell" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get install -y apt-transport-https ca-certificates curl wget curl vim openssl git net-tools",

      "echo 'datasource_list: [ OpenStack, ConfigDrive, NoCloud ]' | sudo tee /etc/cloud/cloud.cfg.d/99-datasource.cfg",

      # Remove installer config
      "sudo rm -f /etc/netplan/00-installer-config.yaml",

      # Disable swap
      "sudo swapoff -a",
      "sudo sed -i '/ swap / s/^/#/' /etc/fstab",

      # Load kernel modules
      "echo 'overlay' | sudo tee /etc/modules-load.d/containerd.conf",
      "echo 'br_netfilter' | sudo tee -a /etc/modules-load.d/containerd.conf",
      "sudo modprobe overlay",
      "sudo modprobe br_netfilter",

      # Sysctl settings
      "echo 'net.bridge.bridge-nf-call-iptables = 1' | sudo tee /etc/sysctl.d/k8s.conf",
      "echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/k8s.conf",
      "echo 'net.bridge.bridge-nf-call-ip6tables = 1' | sudo tee -a /etc/sysctl.d/k8s.conf",
      "sudo sysctl --system" 
    ]
  }

  # SSH hardening
  provisioner "shell" {
    inline = [
      "sudo mv /tmp/99-custom.conf /etc/ssh/sshd_config.d/99-custom.conf",
      "sudo chown root:root /etc/ssh/sshd_config.d/99-custom.conf",
      "sudo chmod 644 /etc/ssh/sshd_config.d/99-custom.conf",

      # validate config
      "sudo sshd -t"
    ]
  }

  # cleanup
  provisioner "shell" {
   inline = [
    # guest agent
    "sudo systemctl enable qemu-guest-agent",

    # cleanup
    "sudo apt-get clean",
    "sudo rm -rf /var/lib/apt/lists/*",

    # reset identity
    "sudo truncate -s 0 /etc/machine-id",
    "sudo rm -f /var/lib/dbus/machine-id",
    "sudo rm -f /etc/ssh/ssh_host_*",

    "sudo cloud-init clean --logs",
    "sudo rm -rf /var/lib/cloud/*",

    # UFW
    "sudo ufw default deny incoming",
    "sudo ufw default allow outgoing",
    "sudo ufw allow OpenSSH",
    "sudo ufw disable",

    "sudo rm -f /home/admin/.ssh/authorized_keys",

    "sync"
  ]
 }
}

Next, let us have some variables in variables.pkr.hcl file:

variable "ssh_ip" {
  type        = string
  description = "Allowed SSH source IP"
  default     = "1.1.1.1"
}

variable "hostname" {
  type    = string
  default = "ubuntu"
}

variable "username" {
  type    = string
  default = "username"
}

variable "password" {
  type    = string
  default = "topsecret"
}

variable "region" {
  type    = string
  default = "contabo"
}

variable "image_name" {
  type    = string
  default = "ubuntu-base"
}

variable "ssh_public_key_path" {
  type        = string
  description = "Path to SSH public key"
}

variable "shapassword" {
  type        = string
  default = ""
}

Variables can be overridden through builds/ files. In this example, let’s set up an override with:

hostname       = "ubuntu"
username       = "username"
password       = "topsecret"
image_name     = "dev-ubuntu-22.04-base"
ssh_public_key_path  = "~/.ssh/dev_id_rsa.pub"

Next, we need to create user-data and meta-data files in http/ directory, which will be served to cloud-init during the build phase. I decided to template a user-data file via an external script and a tpl file. The user-data.tpl looks like this:

#cloud-config
autoinstall:
  version: 1

  identity:
    hostname: $hostname
    username: $username
    password: "$shapassword"

  ssh:
    install-server: true
    allow-pw: true
    authorized_keys:
     - $ssh_public_key


  storage:
    layout:
      name: direct

  packages:
    - qemu-guest-agent
    - ufw

  late-commands:
    - curtin in-target -- systemctl enable ssh
    - curtin in-target -- sh -c "echo '$username ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/${username}"

# ---- runtime cloud-init (Contabo-compatible) ----

users:
  - default

disable_root: true
preserve_hostname: false

system_info:
  default_user:
    name: $username
    lock_passwd: false
    groups: [sudo]
    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
    shell: /bin/bash

ssh_pwauth: true

# -- network
network:
  version: 2
  ethernets:
    all:
      match:
        name: "*"
      dhcp4: true

# ---- firewall (SAFE stage) ----

runcmd:
  # reset UFW (clean state)
  - ufw --force reset

  # default policies
  - ufw default deny incoming
  - ufw default allow outgoing

  # TEMP safety rule (avoid lockout during setup)
  - ufw allow 22/tcp

  # allow SSH only from your IP(s)
  - ufw allow from 1.1.1.1/20 to any port 22 proto tcp
  - ufw allow from 10.0.0.0/8 to any port 22 proto tcp
  - ufw allow from 172.16.0.0/12 to any port 22 proto tcp
  - ufw allow from 192.168.0.0/16 to any port 22 proto tcp

  # enable firewall
  - ufw --force enable

  # remove temporary open SSH rule
  - ufw delete allow 22/tcp

And the shell script that would generate http/user-data file:

#!/usr/bin/env bash
set -euo pipefail

VARS_FILE="${1:-pkvars.hcl}"

# ========== RENDER USER-DATA ====================
echo "Rendering user-data from ${VARS_FILE}..."

VARS_JSON=$(hcl2json < "$VARS_FILE")

export hostname=$(echo "$VARS_JSON" | jq -r '.hostname')
export username=$(echo "$VARS_JSON" | jq -r '.username')

password=$(echo "$VARS_JSON" | jq -r '.password')
export shapassword=$(mkpasswd -m sha-512 $password)

ssh_key_path=$(echo "$VARS_JSON" | jq -r '.ssh_public_key_path')

ssh_key_path="${ssh_key_path/#\~/$HOME}"
export ssh_public_key=$(cat "$ssh_key_path")

envsubst < user-data.tpl > http/user-data

echo "http/user-data generated"

# ========== PACKER BUILD =======================
echo "Starting packer build from ${VARS_FILE}..."
packer build -var-file=${VARS_FILE} .

Lastly, the http/meta-data file looks fairly generic:

instance-id: iid-local01
local-hostname: ubuntu

Finally, to build an image, run the script as:

./build_packer_image.sh builds/dev.pkrvars.hcl

Once the build is complete, there will be a separate output/ directory created with a new qcow2 image.

Step 4: Compress & Optimize QCOW2

The final step is to compress and optimize the image and get it ready for uploading into a cloud provider. This process will reduce the size significantly.

qemu-img convert -O qcow2 -c output/ubuntu-custom.qcow2 output/ubuntu-final.qcow2

Step 5: Upload the image

If you are running your infrastructure in AWS or GCP, you can store your images within each given platform.

However, in my case, I was using Contabo to run my K8s learning clusters on a few VPS instances. In order to use my newly built images, I had to somehow upload them into Contabo. The problem was - to upload an image, you had to provide a public URL to it, which Contabo will then use to pull the image into their image store.

Since I had no other VPS laying around that would serve as network storage, I used an S3 bucket with additional features:

Step 5.1: Create a bucket and upload an image

First, log into AWS, create a public bucket and then upload an image there

aws s3 cp ubuntu-final.qcow2 s3://your-bucket/

Step 5.2: Generate a signed URL with a TTL on it

aws s3 presign s3://your-bucket/ubuntu-final.qcow2 --expires-in 3600

The result would look like:

https://your-bucket.s3.amazonaws.com/ubuntu-final.qcow2?X-Amz-Signature=...

Which I then gave to Contabo to pull my image into their local store.

It goes without saying that this approach should not be used for production-grade systems. I used Contabo VPS for my own learning and K8s test environments, so there was no actual production data nor senstive information.

5.3: A local http server

If you can serve an HTTP traffic from your local machine (port forwarding configured, etc etc), a small local http server could do the trick:

python3 -m http.server 8000

Summary

The steps above described a simple build that generated a single base image. This is fine for small setups. However, for any larger project, I strongly recommend using Jsonnet instead of HCL2. Jsonnet gives a lot more flexibility and templating options, while avoiding code repetitions that come with HCL.