path: root/ISO-BUILDER.md

# ISO Builder

## Overview

This document specifies how to build a **bootable ISO image** that contains the base OS, all binaries, configuration files, systemd units, and scripts required for cluster-from-systemd.

## Design Decision: Kickstart + Live Image

**Approach**: Create a Rocky Linux 9 live ISO with embedded kickstart for automated installation.

**Rationale**:
- **Kickstart**: Automates installation (partitioning, package selection, post-install scripts)
- **Live CD tooling**: Well-established RHEL ecosystem tools (lorax, livecd-tools)
- **Single ISO**: All nodes boot from the same image
- **Reproducible**: Version-controlled build process

**Alternative approaches considered**:
- **cloud-init/ignition**: Better for cloud, but requires external metadata service
- **PXE boot**: Network dependency, more complex infrastructure
- **Container-based build (mkosi)**: Bleeding edge, less RHEL-native

## Build System Requirements

**Host OS**: Fedora 39 or Rocky Linux 9 (build environment)

**Required packages**:
```bash
dnf install -y \
  lorax \
  anaconda \
  pykickstart \
  createrepo_c \
  genisoimage \
  isomd5sum \
  syslinux \
  git
```

**Disk space**: ~30 GB (for build artifacts, repos, and final ISO)

## Directory Structure

```
iso-build/
├── kickstart/
│   └── cluster-node.ks              # Main kickstart file
├── repo/
│   ├── binaries/                    # Downloaded binaries (k8s, etcd, etc.)
│   ├── rpms/                        # Custom RPM packages
│   └── repodata/                    # RPM repository metadata
├── overlay/
│   ├── etc/
│   │   ├── cluster-config/          # Cluster YAML configs
│   │   │   ├── cluster.yaml
│   │   │   ├── services/
│   │   │   └── nodes/
│   │   ├── systemd/system/          # Systemd units
│   │   └── kubernetes/
│   │       ├── pki/                 # Pre-generated certificates
│   │       └── manifests/
│   ├── usr/local/bin/               # Cluster scripts
│   └── var/www/html/                # Join info distribution
├── output/
│   └── cluster-node.iso             # Final bootable ISO
└── build.sh                         # Main build script
```

## Kickstart File

**File**: `iso-build/kickstart/cluster-node.ks`

```kickstart
#version=RHEL9

# Install mode (not upgrade)
install

# Use CDROM installation media
cdrom

# Text mode installation (no GUI)
text

# System language
lang en_US.UTF-8

# Keyboard layouts
keyboard us

# Network configuration (temporary DHCP, will be reconfigured on boot)
network --bootproto=dhcp --device=link --activate

# Root password (change this!)
rootpw --plaintext cluster-password

# System authorization
authselect select sssd

# SELinux mode
selinux --enforcing

# Firewall configuration
firewall --enabled --service=ssh

# System timezone
timezone America/New_York --utc

# Disk partitioning
# WARNING: This will ERASE the entire disk!
ignoredisk --only-use=sda
clearpart --all --initlabel --drives=sda
part /boot --fstype=xfs --size=1024
part / --fstype=xfs --size=20480 --grow
part swap --fstype=swap --size=4096

# Bootloader
bootloader --location=mbr --boot-drive=sda

# Reboot after installation
reboot

# Package selection
%packages
@core
@standard
kernel
systemd
systemd-networkd
NetworkManager
vim
tmux
htop
curl
wget
jq
python3
python3-pip
python3-pyyaml
openssl
iproute
iputils
bind-utils
iptables
nftables
%end

# Pre-installation script
%pre --log=/tmp/ks-pre.log
#!/bin/bash
echo "==> Starting pre-installation"
# Could add disk detection logic here
%end

# Post-installation script
%post --log=/root/ks-post.log --erroronfail
#!/bin/bash
set -x

echo "==> Starting post-installation configuration"

# 1. Copy cluster configuration files
echo "==> Installing cluster configuration..."
mkdir -p /etc/cluster-config/{services,nodes,environment}
cp -r /mnt/install/repo/overlay/etc/cluster-config/* /etc/cluster-config/

# 2. Copy systemd units
echo "==> Installing systemd units..."
cp /mnt/install/repo/overlay/etc/systemd/system/* /etc/systemd/system/
systemctl daemon-reload

# 3. Install cluster scripts
echo "==> Installing cluster scripts..."
cp /mnt/install/repo/overlay/usr/local/bin/* /usr/local/bin/
chmod +x /usr/local/bin/*.sh

# 4. Install binaries
echo "==> Installing Kubernetes binaries..."
BINARIES="kubelet kubectl kubeadm kube-apiserver kube-controller-manager kube-scheduler"
for binary in $BINARIES; do
  if [[ -f /mnt/install/repo/binaries/$binary ]]; then
    cp /mnt/install/repo/binaries/$binary /usr/local/bin/
    chmod +x /usr/local/bin/$binary
  fi
done

echo "==> Installing etcd..."
if [[ -f /mnt/install/repo/binaries/etcd ]]; then
  cp /mnt/install/repo/binaries/etcd /usr/local/bin/
  cp /mnt/install/repo/binaries/etcdctl /usr/local/bin/
  chmod +x /usr/local/bin/etcd /usr/local/bin/etcdctl
fi

echo "==> Installing CoreDNS..."
if [[ -f /mnt/install/repo/binaries/coredns ]]; then
  cp /mnt/install/repo/binaries/coredns /usr/local/bin/
  chmod +x /usr/local/bin/coredns
fi

echo "==> Installing CNI plugins..."
if [[ -d /mnt/install/repo/binaries/cni ]]; then
  mkdir -p /opt/cni/bin
  cp -r /mnt/install/repo/binaries/cni/* /opt/cni/bin/
  chmod +x /opt/cni/bin/*
fi

echo "==> Installing Kafka..."
if [[ -d /mnt/install/repo/binaries/kafka ]]; then
  mkdir -p /opt
  cp -r /mnt/install/repo/binaries/kafka /opt/
  ln -sf /opt/kafka /opt/kafka-current
fi

# 5. Install certificates
echo "==> Installing certificates..."
mkdir -p /etc/kubernetes/pki/etcd
if [[ -d /mnt/install/repo/overlay/etc/kubernetes/pki ]]; then
  cp -r /mnt/install/repo/overlay/etc/kubernetes/pki/* /etc/kubernetes/pki/
  chmod 600 /etc/kubernetes/pki/*.key
  chmod 600 /etc/kubernetes/pki/etcd/*.key
fi

# 6. Create data directories
echo "==> Creating data directories..."
mkdir -p /var/lib/{kubelet,etcd,kafka,ceph/{mon,osd},mosquitto}
mkdir -p /var/lib/cluster-state

# 7. Install container runtime (containerd)
echo "==> Installing containerd..."
dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
dnf install -y containerd.io runc

mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

systemctl enable containerd

# 8. Install Ceph packages
echo "==> Installing Ceph packages..."
cat > /etc/yum.repos.d/ceph.repo <<'CEPH_REPO'
[ceph]
name=Ceph packages for $basearch
baseurl=https://download.ceph.com/rpm-reef/el9/$basearch
enabled=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-reef/el9/noarch
enabled=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
CEPH_REPO

dnf install -y ceph-common ceph-mon ceph-osd ceph-mds ceph-mgr

# 9. Install Mosquitto
echo "==> Installing Mosquitto..."
dnf install -y epel-release
dnf install -y mosquitto mosquitto-clients

# 10. Install Java (for Kafka)
echo "==> Installing Java..."
dnf install -y java-17-openjdk-headless

# 11. Enable cluster-detect service
echo "==> Enabling cluster services..."
systemctl enable cluster-detect.service
systemctl enable cluster-bootstrap.service

# 12. Disable unwanted services
systemctl disable firewalld
systemctl mask firewalld

# 13. Configure sysctl for Kubernetes
cat >> /etc/sysctl.d/99-kubernetes.conf <<'SYSCTL'
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
SYSCTL

# 14. Load required kernel modules
cat > /etc/modules-load.d/kubernetes.conf <<'MODULES'
overlay
br_netfilter
MODULES

# 15. Install yq (YAML processor)
echo "==> Installing yq..."
YQ_VERSION="v4.40.5"
curl -L "https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/yq_linux_amd64" \
  -o /usr/local/bin/yq
chmod +x /usr/local/bin/yq

# 16. Create version info
cat > /etc/cluster-version <<VERSION
CLUSTER_ISO_VERSION=0.1.0
BUILD_DATE=$(date -Iseconds)
KUBERNETES_VERSION=1.29.1
CEPH_VERSION=18.2.1
KAFKA_VERSION=3.6.1
VERSION

echo "==> Post-installation complete!"
echo "==> System will reboot and detect node identity on first boot"

%end
```

## Build Script

**File**: `iso-build/build.sh`

```bash
#!/bin/bash
# Main ISO build script

set -euo pipefail

BUILD_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
OUTPUT_DIR="$BUILD_DIR/output"
REPO_DIR="$BUILD_DIR/repo"
OVERLAY_DIR="$BUILD_DIR/overlay"
KICKSTART_FILE="$BUILD_DIR/kickstart/cluster-node.ks"

ISO_NAME="cluster-node.iso"
ISO_LABEL="ClusterNode"
ISO_VERSION="0.1.0"

echo "==> Cluster-from-SystemD ISO Builder v$ISO_VERSION"
echo "==> Build directory: $BUILD_DIR"

# Check prerequisites
command -v lorax >/dev/null || { echo "ERROR: lorax not found. Install with: dnf install lorax"; exit 1; }
command -v createrepo_c >/dev/null || { echo "ERROR: createrepo_c not found"; exit 1; }

# Create directories
mkdir -p "$OUTPUT_DIR" "$REPO_DIR"/{binaries,rpms} "$OVERLAY_DIR"

# Step 1: Download binaries
echo ""
echo "==> Step 1: Downloading binaries..."
bash "$BUILD_DIR/../tools/download-kubernetes.sh" "$REPO_DIR/binaries"
bash "$BUILD_DIR/../tools/download-etcd.sh" "$REPO_DIR/binaries"
bash "$BUILD_DIR/../tools/download-cni-plugins.sh" "$REPO_DIR/binaries"
bash "$BUILD_DIR/../tools/download-coredns.sh" "$REPO_DIR/binaries"
bash "$BUILD_DIR/../tools/install-kafka.sh" "$REPO_DIR/binaries"

# Step 2: Copy overlay files
echo ""
echo "==> Step 2: Copying overlay files..."

# Copy configs
mkdir -p "$OVERLAY_DIR/etc/cluster-config"
cp -r "$BUILD_DIR/../configs"/* "$OVERLAY_DIR/etc/cluster-config/"

# Copy systemd units
mkdir -p "$OVERLAY_DIR/etc/systemd/system"
cp "$BUILD_DIR/../systemd"/*.service "$OVERLAY_DIR/etc/systemd/system/"
cp "$BUILD_DIR/../systemd"/*.target "$OVERLAY_DIR/etc/systemd/system/"

# Copy scripts
mkdir -p "$OVERLAY_DIR/usr/local/bin"
cp "$BUILD_DIR/../tools"/*.sh "$OVERLAY_DIR/usr/local/bin/"
cp "$BUILD_DIR/../tools"/*.py "$OVERLAY_DIR/usr/local/bin/"

# Step 3: Generate certificates
echo ""
echo "==> Step 3: Generating certificates..."
bash "$BUILD_DIR/../tools/generate-build-certs.sh" "$OVERLAY_DIR/etc/kubernetes/pki"

# Step 4: Create local repository
echo ""
echo "==> Step 4: Creating local repository..."
createrepo_c "$REPO_DIR/rpms"

# Step 5: Build ISO using lorax
echo ""
echo "==> Step 5: Building ISO with lorax..."

# Download Rocky Linux boot files
ROCKY_VERSION="9.3"
ROCKY_BOOT_ISO="Rocky-${ROCKY_VERSION}-x86_64-boot.iso"
ROCKY_URL="https://download.rockylinux.org/pub/rocky/${ROCKY_VERSION}/isos/x86_64/${ROCKY_BOOT_ISO}"

if [[ ! -f "$REPO_DIR/$ROCKY_BOOT_ISO" ]]; then
  echo "==> Downloading Rocky Linux boot ISO..."
  curl -L "$ROCKY_URL" -o "$REPO_DIR/$ROCKY_BOOT_ISO"
fi

# Mount boot ISO to extract vmlinuz and initrd
MOUNT_DIR="/tmp/rocky-mount-$$"
mkdir -p "$MOUNT_DIR"
mount -o loop "$REPO_DIR/$ROCKY_BOOT_ISO" "$MOUNT_DIR"

# Create working directory for ISO build
ISO_WORK_DIR="/tmp/iso-build-$$"
mkdir -p "$ISO_WORK_DIR"/{isolinux,images,LiveOS}

# Copy boot files
cp "$MOUNT_DIR/isolinux/vmlinuz" "$ISO_WORK_DIR/isolinux/"
cp "$MOUNT_DIR/isolinux/initrd.img" "$ISO_WORK_DIR/isolinux/"
cp "$MOUNT_DIR/isolinux/isolinux.bin" "$ISO_WORK_DIR/isolinux/"
cp "$MOUNT_DIR/isolinux/ldlinux.c32" "$ISO_WORK_DIR/isolinux/"
cp "$MOUNT_DIR/isolinux/libcom32.c32" "$ISO_WORK_DIR/isolinux/"
cp "$MOUNT_DIR/isolinux/libutil.c32" "$ISO_WORK_DIR/isolinux/"

umount "$MOUNT_DIR"
rmdir "$MOUNT_DIR"

# Create isolinux.cfg with kickstart
cat > "$ISO_WORK_DIR/isolinux/isolinux.cfg" <<'ISOLINUX'
default vesamenu.c32
timeout 100

display boot.msg

label install
  menu label ^Install Cluster Node
  kernel vmlinuz
  append initrd=initrd.img inst.stage2=hd:LABEL=ClusterNode inst.ks=hd:LABEL=ClusterNode:/ks.cfg

label rescue
  menu label ^Rescue installed system
  kernel vmlinuz
  append initrd=initrd.img inst.stage2=hd:LABEL=ClusterNode rescue
ISOLINUX

# Copy kickstart file to ISO root
cp "$KICKSTART_FILE" "$ISO_WORK_DIR/ks.cfg"

# Copy overlay and repo to ISO
cp -r "$OVERLAY_DIR" "$ISO_WORK_DIR/overlay"
cp -r "$REPO_DIR" "$ISO_WORK_DIR/repo"

# Create ISO
echo "==> Creating bootable ISO..."
genisoimage \
  -o "$OUTPUT_DIR/$ISO_NAME" \
  -b isolinux/isolinux.bin \
  -c isolinux/boot.cat \
  -no-emul-boot \
  -boot-load-size 4 \
  -boot-info-table \
  -J -R -v \
  -V "$ISO_LABEL" \
  "$ISO_WORK_DIR"

# Make ISO bootable
isohybrid "$OUTPUT_DIR/$ISO_NAME"

# Add MD5 checksum
implantisomd5 "$OUTPUT_DIR/$ISO_NAME"

# Cleanup
rm -rf "$ISO_WORK_DIR"

echo ""
echo "==> ISO build complete!"
echo "==> Output: $OUTPUT_DIR/$ISO_NAME"
echo "==> Size: $(du -h "$OUTPUT_DIR/$ISO_NAME" | cut -f1)"
echo ""
echo "Test with:"
echo "  qemu-system-x86_64 -cdrom $OUTPUT_DIR/$ISO_NAME -m 4096 -smp 2"
```

## Simplified Build Script (Alternative: mkosi)

For a more modern, declarative approach:

**File**: `iso-build/mkosi.conf`

```ini
[Output]
ImageId=cluster-node
ImageVersion=0.1.0
Format=disk

[Distribution]
Distribution=rocky
Release=9

[Content]
Packages=
  systemd
  kubernetes-kubeadm
  etcd
  vim
  tmux

ExtraTree=/path/to/overlay/

[Host]
QemuHeadless=yes
```

**Note**: mkosi is cleaner but less mature for RHEL-based distros. Recommend starting with lorax/kickstart.

## Modified Download Scripts for Build

Update download scripts to accept output directory:

**Example**: `tools/download-kubernetes.sh`

```bash
#!/bin/bash
# Modified to accept output directory for ISO build

KUBE_VERSION="v1.29.1"
ARCH="amd64"
BASE_URL="https://dl.k8s.io/release/${KUBE_VERSION}/bin/linux/${ARCH}"
OUTPUT_DIR="${1:-/usr/local/bin}"

mkdir -p "$OUTPUT_DIR"

BINARIES=(
  "kubeadm"
  "kubectl"
  "kubelet"
  "kube-apiserver"
  "kube-controller-manager"
  "kube-scheduler"
)

for binary in "${BINARIES[@]}"; do
  echo "Downloading $binary to $OUTPUT_DIR..."
  curl -L "${BASE_URL}/${binary}" -o "$OUTPUT_DIR/${binary}"
  chmod +x "$OUTPUT_DIR/${binary}"
done

echo "==> Kubernetes binaries downloaded to $OUTPUT_DIR"
```

## Testing the ISO

### QEMU Testing

```bash
#!/bin/bash
# tools/test-iso-qemu.sh

ISO_PATH="iso-build/output/cluster-node.iso"
DISK_IMG="test-disk.qcow2"

# Create virtual disk
qemu-img create -f qcow2 "$DISK_IMG" 40G

# Boot from ISO
qemu-system-x86_64 \
  -cdrom "$ISO_PATH" \
  -hda "$DISK_IMG" \
  -m 4096 \
  -smp 2 \
  -boot d \
  -netdev user,id=net0 \
  -device virtio-net-pci,netdev=net0 \
  -serial stdio \
  -display none

# After installation, boot from disk
# qemu-system-x86_64 -hda "$DISK_IMG" -m 4096 -smp 2 -boot c
```

### VirtualBox Testing

```bash
# Create VM
VBoxManage createvm --name "cluster-test" --ostype RedHat_64 --register

# Configure VM
VBoxManage modifyvm "cluster-test" \
  --memory 4096 \
  --cpus 2 \
  --nic1 bridged \
  --bridgeadapter1 eth0

# Create disk
VBoxManage createhd --filename cluster-test.vdi --size 40960

# Attach storage
VBoxManage storagectl "cluster-test" --name "SATA" --add sata --controller IntelAhci
VBoxManage storageattach "cluster-test" --storagectl "SATA" --port 0 --device 0 --type hdd --medium cluster-test.vdi
VBoxManage storageattach "cluster-test" --storagectl "SATA" --port 1 --device 0 --type dvddrive --medium cluster-node.iso

# Start VM
VBoxManage startvm "cluster-test"
```

### Multi-Node Test

```bash
#!/bin/bash
# tools/test-multi-node.sh
# Create 3 VMs for full cluster test

for i in 1 2 3; do
  DISK="test-node-${i}.qcow2"
  qemu-img create -f qcow2 "$DISK" 40G

  qemu-system-x86_64 \
    -name "node-${i}" \
    -cdrom iso-build/output/cluster-node.iso \
    -hda "$DISK" \
    -m 4096 \
    -smp 2 \
    -boot d \
    -netdev tap,id=net0,ifname=tap${i},script=no \
    -device virtio-net-pci,netdev=net0,mac=52:54:00:12:34:1${i} \
    -daemonize \
    -vnc :${i}

  echo "Node $i started on VNC port 590${i}"
done
```

## Build Optimization

### Caching Downloaded Files

```bash
# Use a persistent cache directory
CACHE_DIR="$HOME/.cache/cluster-iso-build"
mkdir -p "$CACHE_DIR"/{binaries,rpms}

# In build.sh, check cache before downloading
if [[ ! -f "$CACHE_DIR/binaries/kubelet" ]]; then
  bash tools/download-kubernetes.sh "$CACHE_DIR/binaries"
fi

cp -r "$CACHE_DIR/binaries"/* "$REPO_DIR/binaries/"
```

### Parallel Downloads

```bash
# Download binaries in parallel
bash tools/download-kubernetes.sh "$REPO_DIR/binaries" &
bash tools/download-etcd.sh "$REPO_DIR/binaries" &
bash tools/download-coredns.sh "$REPO_DIR/binaries" &
wait

echo "==> All downloads complete"
```

## Reproducible Builds

### Version Lock File

**File**: `iso-build/versions.lock`

```yaml
kubernetes:
  version: "1.29.1"
  sha256:
    kubelet: "abc123..."
    kubectl: "def456..."

etcd:
  version: "3.5.11"
  sha256: "789ghi..."

ceph:
  version: "18.2.1"

kafka:
  version: "3.6.1"
  scala_version: "2.13"

base_iso:
  name: "Rocky-9.3-x86_64-boot.iso"
  sha256: "..."
```

### Checksum Verification

```bash
#!/bin/bash
# tools/verify-downloads.sh

set -euo pipefail

VERSIONS_FILE="iso-build/versions.lock"
BINARIES_DIR="$1"

while read -r binary expected_sha; do
  actual_sha=$(sha256sum "$BINARIES_DIR/$binary" | awk '{print $1}')

  if [[ "$actual_sha" != "$expected_sha" ]]; then
    echo "ERROR: Checksum mismatch for $binary"
    echo "  Expected: $expected_sha"
    echo "  Actual:   $actual_sha"
    exit 1
  fi

  echo "✓ $binary: checksum OK"
done < <(yq eval '.kubernetes.sha256 | to_entries | .[] | .key + " " + .value' "$VERSIONS_FILE")
```

## CI/CD Integration

### GitHub Actions Example

```yaml
# .github/workflows/build-iso.yml
name: Build ISO

on:
  push:
    tags:
      - 'v*'

jobs:
  build:
    runs-on: ubuntu-latest
    container:
      image: rockylinux:9

    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Install build dependencies
        run: |
          dnf install -y lorax createrepo_c genisoimage isomd5sum

      - name: Run build
        run: |
          bash iso-build/build.sh

      - name: Upload ISO artifact
        uses: actions/upload-artifact@v3
        with:
          name: cluster-node-iso
          path: iso-build/output/cluster-node.iso

      - name: Create release
        uses: softprops/action-gh-release@v1
        with:
          files: iso-build/output/cluster-node.iso
```

## Troubleshooting

### Build Fails: "lorax command not found"

```bash
# On Fedora
dnf install lorax anaconda-tui

# On Rocky Linux
dnf install epel-release
dnf install lorax
```

### ISO Won't Boot: "Missing operating system"

```bash
# Ensure isohybrid was run
isohybrid output/cluster-node.iso

# Verify boot sector
file output/cluster-node.iso
# Should show: "DOS/MBR boot sector"
```

### Kickstart Fails During Post-Install

```bash
# Boot ISO in rescue mode
# Check logs
cat /tmp/ks-post.log
cat /tmp/anaconda.log

# Common issues:
# - Missing files in overlay
# - Incorrect paths in kickstart
# - Network not available during post-install
```

## Next Steps

1. Create `iso-build/` directory structure
2. Implement build.sh script
3. Test ISO build on clean Fedora VM
4. Test installation in QEMU
5. Verify cluster-detect runs on first boot
6. Test multi-node cluster formation
7. Document customization procedures

## Customization Guide

### Adding Custom Packages

Edit kickstart file:
```kickstart
%packages
@core
my-custom-package
%end
```

### Changing Root Password

In kickstart:
```kickstart
rootpw --iscrypted $6$rounds=4096$salt$hash
```

Generate hash:
```bash
python3 -c 'import crypt; print(crypt.crypt("your-password", crypt.mkmeth(crypt.METHOD_SHA512)))'
```

### Embedding Custom Files

Add to overlay directory:
```bash
mkdir -p overlay/etc/my-app/
cp my-config.yaml overlay/etc/my-app/
```

## Maintenance

### Updating Base OS

```bash
# Update ROCKY_VERSION in build.sh
ROCKY_VERSION="9.4"  # When Rocky 9.4 is released

# Re-download boot ISO
rm repo/Rocky-*.iso
bash iso-build/build.sh
```

### Updating Kubernetes Version

```bash
# Edit versions.lock
kubernetes:
  version: "1.30.0"

# Re-run build
bash iso-build/build.sh
```

---

**P.S.** This is the glue that makes everything real—from pile of configs to bootable artifact. Kickstart is ugly but battle-tested; lorax is RHEL-native. The build script is bash-heavy but transparent. You can see exactly what's happening, which beats "magic container build tool" when debugging at 2 AM. Test in QEMU first, VirtualBox second, bare metal third.