more claudeHEADmain

1 files changed, 849 insertions, 0 deletions

diff --git a/ISO-BUILDER.md b/ISO-BUILDER.md
new file mode 100644
index 0000000..a1c5ebd
--- /dev/null
+++ b/ISO-BUILDER.md
@@ -0,0 +1,849 @@
+# ISO Builder
+
+## Overview
+
+This document specifies how to build a **bootable ISO image** that contains the base OS, all binaries, configuration files, systemd units, and scripts required for cluster-from-systemd.
+
+## Design Decision: Kickstart + Live Image
+
+**Approach**: Create a Rocky Linux 9 live ISO with embedded kickstart for automated installation.
+
+**Rationale**:
+- **Kickstart**: Automates installation (partitioning, package selection, post-install scripts)
+- **Live CD tooling**: Well-established RHEL ecosystem tools (lorax, livecd-tools)
+- **Single ISO**: All nodes boot from the same image
+- **Reproducible**: Version-controlled build process
+
+**Alternative approaches considered**:
+- **cloud-init/ignition**: Better for cloud, but requires external metadata service
+- **PXE boot**: Network dependency, more complex infrastructure
+- **Container-based build (mkosi)**: Bleeding edge, less RHEL-native
+
+## Build System Requirements
+
+**Host OS**: Fedora 39 or Rocky Linux 9 (build environment)
+
+**Required packages**:
+```bash
+dnf install -y \
+  lorax \
+  anaconda \
+  pykickstart \
+  createrepo_c \
+  genisoimage \
+  isomd5sum \
+  syslinux \
+  git
+```
+
+**Disk space**: ~30 GB (for build artifacts, repos, and final ISO)
+
+## Directory Structure
+
+```
+iso-build/
+├── kickstart/
+│   └── cluster-node.ks              # Main kickstart file
+├── repo/
+│   ├── binaries/                    # Downloaded binaries (k8s, etcd, etc.)
+│   ├── rpms/                        # Custom RPM packages
+│   └── repodata/                    # RPM repository metadata
+├── overlay/
+│   ├── etc/
+│   │   ├── cluster-config/          # Cluster YAML configs
+│   │   │   ├── cluster.yaml
+│   │   │   ├── services/
+│   │   │   └── nodes/
+│   │   ├── systemd/system/          # Systemd units
+│   │   └── kubernetes/
+│   │       ├── pki/                 # Pre-generated certificates
+│   │       └── manifests/
+│   ├── usr/local/bin/               # Cluster scripts
+│   └── var/www/html/                # Join info distribution
+├── output/
+│   └── cluster-node.iso             # Final bootable ISO
+└── build.sh                         # Main build script
+```
+
+## Kickstart File
+
+**File**: `iso-build/kickstart/cluster-node.ks`
+
+```kickstart
+#version=RHEL9
+
+# Install mode (not upgrade)
+install
+
+# Use CDROM installation media
+cdrom
+
+# Text mode installation (no GUI)
+text
+
+# System language
+lang en_US.UTF-8
+
+# Keyboard layouts
+keyboard us
+
+# Network configuration (temporary DHCP, will be reconfigured on boot)
+network --bootproto=dhcp --device=link --activate
+
+# Root password (change this!)
+rootpw --plaintext cluster-password
+
+# System authorization
+authselect select sssd
+
+# SELinux mode
+selinux --enforcing
+
+# Firewall configuration
+firewall --enabled --service=ssh
+
+# System timezone
+timezone America/New_York --utc
+
+# Disk partitioning
+# WARNING: This will ERASE the entire disk!
+ignoredisk --only-use=sda
+clearpart --all --initlabel --drives=sda
+part /boot --fstype=xfs --size=1024
+part / --fstype=xfs --size=20480 --grow
+part swap --fstype=swap --size=4096
+
+# Bootloader
+bootloader --location=mbr --boot-drive=sda
+
+# Reboot after installation
+reboot
+
+# Package selection
+%packages
+@core
+@standard
+kernel
+systemd
+systemd-networkd
+NetworkManager
+vim
+tmux
+htop
+curl
+wget
+jq
+python3
+python3-pip
+python3-pyyaml
+openssl
+iproute
+iputils
+bind-utils
+iptables
+nftables
+%end
+
+# Pre-installation script
+%pre --log=/tmp/ks-pre.log
+#!/bin/bash
+echo "==> Starting pre-installation"
+# Could add disk detection logic here
+%end
+
+# Post-installation script
+%post --log=/root/ks-post.log --erroronfail
+#!/bin/bash
+set -x
+
+echo "==> Starting post-installation configuration"
+
+# 1. Copy cluster configuration files
+echo "==> Installing cluster configuration..."
+mkdir -p /etc/cluster-config/{services,nodes,environment}
+cp -r /mnt/install/repo/overlay/etc/cluster-config/* /etc/cluster-config/
+
+# 2. Copy systemd units
+echo "==> Installing systemd units..."
+cp /mnt/install/repo/overlay/etc/systemd/system/* /etc/systemd/system/
+systemctl daemon-reload
+
+# 3. Install cluster scripts
+echo "==> Installing cluster scripts..."
+cp /mnt/install/repo/overlay/usr/local/bin/* /usr/local/bin/
+chmod +x /usr/local/bin/*.sh
+
+# 4. Install binaries
+echo "==> Installing Kubernetes binaries..."
+BINARIES="kubelet kubectl kubeadm kube-apiserver kube-controller-manager kube-scheduler"
+for binary in $BINARIES; do
+  if [[ -f /mnt/install/repo/binaries/$binary ]]; then
+    cp /mnt/install/repo/binaries/$binary /usr/local/bin/
+    chmod +x /usr/local/bin/$binary
+  fi
+done
+
+echo "==> Installing etcd..."
+if [[ -f /mnt/install/repo/binaries/etcd ]]; then
+  cp /mnt/install/repo/binaries/etcd /usr/local/bin/
+  cp /mnt/install/repo/binaries/etcdctl /usr/local/bin/
+  chmod +x /usr/local/bin/etcd /usr/local/bin/etcdctl
+fi
+
+echo "==> Installing CoreDNS..."
+if [[ -f /mnt/install/repo/binaries/coredns ]]; then
+  cp /mnt/install/repo/binaries/coredns /usr/local/bin/
+  chmod +x /usr/local/bin/coredns
+fi
+
+echo "==> Installing CNI plugins..."
+if [[ -d /mnt/install/repo/binaries/cni ]]; then
+  mkdir -p /opt/cni/bin
+  cp -r /mnt/install/repo/binaries/cni/* /opt/cni/bin/
+  chmod +x /opt/cni/bin/*
+fi
+
+echo "==> Installing Kafka..."
+if [[ -d /mnt/install/repo/binaries/kafka ]]; then
+  mkdir -p /opt
+  cp -r /mnt/install/repo/binaries/kafka /opt/
+  ln -sf /opt/kafka /opt/kafka-current
+fi
+
+# 5. Install certificates
+echo "==> Installing certificates..."
+mkdir -p /etc/kubernetes/pki/etcd
+if [[ -d /mnt/install/repo/overlay/etc/kubernetes/pki ]]; then
+  cp -r /mnt/install/repo/overlay/etc/kubernetes/pki/* /etc/kubernetes/pki/
+  chmod 600 /etc/kubernetes/pki/*.key
+  chmod 600 /etc/kubernetes/pki/etcd/*.key
+fi
+
+# 6. Create data directories
+echo "==> Creating data directories..."
+mkdir -p /var/lib/{kubelet,etcd,kafka,ceph/{mon,osd},mosquitto}
+mkdir -p /var/lib/cluster-state
+
+# 7. Install container runtime (containerd)
+echo "==> Installing containerd..."
+dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
+dnf install -y containerd.io runc
+
+mkdir -p /etc/containerd
+containerd config default > /etc/containerd/config.toml
+sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
+
+systemctl enable containerd
+
+# 8. Install Ceph packages
+echo "==> Installing Ceph packages..."
+cat > /etc/yum.repos.d/ceph.repo <<'CEPH_REPO'
+[ceph]
+name=Ceph packages for $basearch
+baseurl=https://download.ceph.com/rpm-reef/el9/$basearch
+enabled=1
+gpgcheck=1
+gpgkey=https://download.ceph.com/keys/release.asc
+
+[ceph-noarch]
+name=Ceph noarch packages
+baseurl=https://download.ceph.com/rpm-reef/el9/noarch
+enabled=1
+gpgcheck=1
+gpgkey=https://download.ceph.com/keys/release.asc
+CEPH_REPO
+
+dnf install -y ceph-common ceph-mon ceph-osd ceph-mds ceph-mgr
+
+# 9. Install Mosquitto
+echo "==> Installing Mosquitto..."
+dnf install -y epel-release
+dnf install -y mosquitto mosquitto-clients
+
+# 10. Install Java (for Kafka)
+echo "==> Installing Java..."
+dnf install -y java-17-openjdk-headless
+
+# 11. Enable cluster-detect service
+echo "==> Enabling cluster services..."
+systemctl enable cluster-detect.service
+systemctl enable cluster-bootstrap.service
+
+# 12. Disable unwanted services
+systemctl disable firewalld
+systemctl mask firewalld
+
+# 13. Configure sysctl for Kubernetes
+cat >> /etc/sysctl.d/99-kubernetes.conf <<'SYSCTL'
+net.bridge.bridge-nf-call-iptables = 1
+net.bridge.bridge-nf-call-ip6tables = 1
+net.ipv4.ip_forward = 1
+SYSCTL
+
+# 14. Load required kernel modules
+cat > /etc/modules-load.d/kubernetes.conf <<'MODULES'
+overlay
+br_netfilter
+MODULES
+
+# 15. Install yq (YAML processor)
+echo "==> Installing yq..."
+YQ_VERSION="v4.40.5"
+curl -L "https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/yq_linux_amd64" \
+  -o /usr/local/bin/yq
+chmod +x /usr/local/bin/yq
+
+# 16. Create version info
+cat > /etc/cluster-version <<VERSION
+CLUSTER_ISO_VERSION=0.1.0
+BUILD_DATE=$(date -Iseconds)
+KUBERNETES_VERSION=1.29.1
+CEPH_VERSION=18.2.1
+KAFKA_VERSION=3.6.1
+VERSION
+
+echo "==> Post-installation complete!"
+echo "==> System will reboot and detect node identity on first boot"
+
+%end
+```
+
+## Build Script
+
+**File**: `iso-build/build.sh`
+
+```bash
+#!/bin/bash
+# Main ISO build script
+
+set -euo pipefail
+
+BUILD_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+OUTPUT_DIR="$BUILD_DIR/output"
+REPO_DIR="$BUILD_DIR/repo"
+OVERLAY_DIR="$BUILD_DIR/overlay"
+KICKSTART_FILE="$BUILD_DIR/kickstart/cluster-node.ks"
+
+ISO_NAME="cluster-node.iso"
+ISO_LABEL="ClusterNode"
+ISO_VERSION="0.1.0"
+
+echo "==> Cluster-from-SystemD ISO Builder v$ISO_VERSION"
+echo "==> Build directory: $BUILD_DIR"
+
+# Check prerequisites
+command -v lorax >/dev/null || { echo "ERROR: lorax not found. Install with: dnf install lorax"; exit 1; }
+command -v createrepo_c >/dev/null || { echo "ERROR: createrepo_c not found"; exit 1; }
+
+# Create directories
+mkdir -p "$OUTPUT_DIR" "$REPO_DIR"/{binaries,rpms} "$OVERLAY_DIR"
+
+# Step 1: Download binaries
+echo ""
+echo "==> Step 1: Downloading binaries..."
+bash "$BUILD_DIR/../tools/download-kubernetes.sh" "$REPO_DIR/binaries"
+bash "$BUILD_DIR/../tools/download-etcd.sh" "$REPO_DIR/binaries"
+bash "$BUILD_DIR/../tools/download-cni-plugins.sh" "$REPO_DIR/binaries"
+bash "$BUILD_DIR/../tools/download-coredns.sh" "$REPO_DIR/binaries"
+bash "$BUILD_DIR/../tools/install-kafka.sh" "$REPO_DIR/binaries"
+
+# Step 2: Copy overlay files
+echo ""
+echo "==> Step 2: Copying overlay files..."
+
+# Copy configs
+mkdir -p "$OVERLAY_DIR/etc/cluster-config"
+cp -r "$BUILD_DIR/../configs"/* "$OVERLAY_DIR/etc/cluster-config/"
+
+# Copy systemd units
+mkdir -p "$OVERLAY_DIR/etc/systemd/system"
+cp "$BUILD_DIR/../systemd"/*.service "$OVERLAY_DIR/etc/systemd/system/"
+cp "$BUILD_DIR/../systemd"/*.target "$OVERLAY_DIR/etc/systemd/system/"
+
+# Copy scripts
+mkdir -p "$OVERLAY_DIR/usr/local/bin"
+cp "$BUILD_DIR/../tools"/*.sh "$OVERLAY_DIR/usr/local/bin/"
+cp "$BUILD_DIR/../tools"/*.py "$OVERLAY_DIR/usr/local/bin/"
+
+# Step 3: Generate certificates
+echo ""
+echo "==> Step 3: Generating certificates..."
+bash "$BUILD_DIR/../tools/generate-build-certs.sh" "$OVERLAY_DIR/etc/kubernetes/pki"
+
+# Step 4: Create local repository
+echo ""
+echo "==> Step 4: Creating local repository..."
+createrepo_c "$REPO_DIR/rpms"
+
+# Step 5: Build ISO using lorax
+echo ""
+echo "==> Step 5: Building ISO with lorax..."
+
+# Download Rocky Linux boot files
+ROCKY_VERSION="9.3"
+ROCKY_BOOT_ISO="Rocky-${ROCKY_VERSION}-x86_64-boot.iso"
+ROCKY_URL="https://download.rockylinux.org/pub/rocky/${ROCKY_VERSION}/isos/x86_64/${ROCKY_BOOT_ISO}"
+
+if [[ ! -f "$REPO_DIR/$ROCKY_BOOT_ISO" ]]; then
+  echo "==> Downloading Rocky Linux boot ISO..."
+  curl -L "$ROCKY_URL" -o "$REPO_DIR/$ROCKY_BOOT_ISO"
+fi
+
+# Mount boot ISO to extract vmlinuz and initrd
+MOUNT_DIR="/tmp/rocky-mount-$$"
+mkdir -p "$MOUNT_DIR"
+mount -o loop "$REPO_DIR/$ROCKY_BOOT_ISO" "$MOUNT_DIR"
+
+# Create working directory for ISO build
+ISO_WORK_DIR="/tmp/iso-build-$$"
+mkdir -p "$ISO_WORK_DIR"/{isolinux,images,LiveOS}
+
+# Copy boot files
+cp "$MOUNT_DIR/isolinux/vmlinuz" "$ISO_WORK_DIR/isolinux/"
+cp "$MOUNT_DIR/isolinux/initrd.img" "$ISO_WORK_DIR/isolinux/"
+cp "$MOUNT_DIR/isolinux/isolinux.bin" "$ISO_WORK_DIR/isolinux/"
+cp "$MOUNT_DIR/isolinux/ldlinux.c32" "$ISO_WORK_DIR/isolinux/"
+cp "$MOUNT_DIR/isolinux/libcom32.c32" "$ISO_WORK_DIR/isolinux/"
+cp "$MOUNT_DIR/isolinux/libutil.c32" "$ISO_WORK_DIR/isolinux/"
+
+umount "$MOUNT_DIR"
+rmdir "$MOUNT_DIR"
+
+# Create isolinux.cfg with kickstart
+cat > "$ISO_WORK_DIR/isolinux/isolinux.cfg" <<'ISOLINUX'
+default vesamenu.c32
+timeout 100
+
+display boot.msg
+
+label install
+  menu label ^Install Cluster Node
+  kernel vmlinuz
+  append initrd=initrd.img inst.stage2=hd:LABEL=ClusterNode inst.ks=hd:LABEL=ClusterNode:/ks.cfg
+
+label rescue
+  menu label ^Rescue installed system
+  kernel vmlinuz
+  append initrd=initrd.img inst.stage2=hd:LABEL=ClusterNode rescue
+ISOLINUX
+
+# Copy kickstart file to ISO root
+cp "$KICKSTART_FILE" "$ISO_WORK_DIR/ks.cfg"
+
+# Copy overlay and repo to ISO
+cp -r "$OVERLAY_DIR" "$ISO_WORK_DIR/overlay"
+cp -r "$REPO_DIR" "$ISO_WORK_DIR/repo"
+
+# Create ISO
+echo "==> Creating bootable ISO..."
+genisoimage \
+  -o "$OUTPUT_DIR/$ISO_NAME" \
+  -b isolinux/isolinux.bin \
+  -c isolinux/boot.cat \
+  -no-emul-boot \
+  -boot-load-size 4 \
+  -boot-info-table \
+  -J -R -v \
+  -V "$ISO_LABEL" \
+  "$ISO_WORK_DIR"
+
+# Make ISO bootable
+isohybrid "$OUTPUT_DIR/$ISO_NAME"
+
+# Add MD5 checksum
+implantisomd5 "$OUTPUT_DIR/$ISO_NAME"
+
+# Cleanup
+rm -rf "$ISO_WORK_DIR"
+
+echo ""
+echo "==> ISO build complete!"
+echo "==> Output: $OUTPUT_DIR/$ISO_NAME"
+echo "==> Size: $(du -h "$OUTPUT_DIR/$ISO_NAME" | cut -f1)"
+echo ""
+echo "Test with:"
+echo "  qemu-system-x86_64 -cdrom $OUTPUT_DIR/$ISO_NAME -m 4096 -smp 2"
+```
+
+## Simplified Build Script (Alternative: mkosi)
+
+For a more modern, declarative approach:
+
+**File**: `iso-build/mkosi.conf`
+
+```ini
+[Output]
+ImageId=cluster-node
+ImageVersion=0.1.0
+Format=disk
+
+[Distribution]
+Distribution=rocky
+Release=9
+
+[Content]
+Packages=
+  systemd
+  kubernetes-kubeadm
+  etcd
+  vim
+  tmux
+
+ExtraTree=/path/to/overlay/
+
+[Host]
+QemuHeadless=yes
+```
+
+**Note**: mkosi is cleaner but less mature for RHEL-based distros. Recommend starting with lorax/kickstart.
+
+## Modified Download Scripts for Build
+
+Update download scripts to accept output directory:
+
+**Example**: `tools/download-kubernetes.sh`
+
+```bash
+#!/bin/bash
+# Modified to accept output directory for ISO build
+
+KUBE_VERSION="v1.29.1"
+ARCH="amd64"
+BASE_URL="https://dl.k8s.io/release/${KUBE_VERSION}/bin/linux/${ARCH}"
+OUTPUT_DIR="${1:-/usr/local/bin}"
+
+mkdir -p "$OUTPUT_DIR"
+
+BINARIES=(
+  "kubeadm"
+  "kubectl"
+  "kubelet"
+  "kube-apiserver"
+  "kube-controller-manager"
+  "kube-scheduler"
+)
+
+for binary in "${BINARIES[@]}"; do
+  echo "Downloading $binary to $OUTPUT_DIR..."
+  curl -L "${BASE_URL}/${binary}" -o "$OUTPUT_DIR/${binary}"
+  chmod +x "$OUTPUT_DIR/${binary}"
+done
+
+echo "==> Kubernetes binaries downloaded to $OUTPUT_DIR"
+```
+
+## Testing the ISO
+
+### QEMU Testing
+
+```bash
+#!/bin/bash
+# tools/test-iso-qemu.sh
+
+ISO_PATH="iso-build/output/cluster-node.iso"
+DISK_IMG="test-disk.qcow2"
+
+# Create virtual disk
+qemu-img create -f qcow2 "$DISK_IMG" 40G
+
+# Boot from ISO
+qemu-system-x86_64 \
+  -cdrom "$ISO_PATH" \
+  -hda "$DISK_IMG" \
+  -m 4096 \
+  -smp 2 \
+  -boot d \
+  -netdev user,id=net0 \
+  -device virtio-net-pci,netdev=net0 \
+  -serial stdio \
+  -display none
+
+# After installation, boot from disk
+# qemu-system-x86_64 -hda "$DISK_IMG" -m 4096 -smp 2 -boot c
+```
+
+### VirtualBox Testing
+
+```bash
+# Create VM
+VBoxManage createvm --name "cluster-test" --ostype RedHat_64 --register
+
+# Configure VM
+VBoxManage modifyvm "cluster-test" \
+  --memory 4096 \
+  --cpus 2 \
+  --nic1 bridged \
+  --bridgeadapter1 eth0
+
+# Create disk
+VBoxManage createhd --filename cluster-test.vdi --size 40960
+
+# Attach storage
+VBoxManage storagectl "cluster-test" --name "SATA" --add sata --controller IntelAhci
+VBoxManage storageattach "cluster-test" --storagectl "SATA" --port 0 --device 0 --type hdd --medium cluster-test.vdi
+VBoxManage storageattach "cluster-test" --storagectl "SATA" --port 1 --device 0 --type dvddrive --medium cluster-node.iso
+
+# Start VM
+VBoxManage startvm "cluster-test"
+```
+
+### Multi-Node Test
+
+```bash
+#!/bin/bash
+# tools/test-multi-node.sh
+# Create 3 VMs for full cluster test
+
+for i in 1 2 3; do
+  DISK="test-node-${i}.qcow2"
+  qemu-img create -f qcow2 "$DISK" 40G
+
+  qemu-system-x86_64 \
+    -name "node-${i}" \
+    -cdrom iso-build/output/cluster-node.iso \
+    -hda "$DISK" \
+    -m 4096 \
+    -smp 2 \
+    -boot d \
+    -netdev tap,id=net0,ifname=tap${i},script=no \
+    -device virtio-net-pci,netdev=net0,mac=52:54:00:12:34:1${i} \
+    -daemonize \
+    -vnc :${i}
+
+  echo "Node $i started on VNC port 590${i}"
+done
+```
+
+## Build Optimization
+
+### Caching Downloaded Files
+
+```bash
+# Use a persistent cache directory
+CACHE_DIR="$HOME/.cache/cluster-iso-build"
+mkdir -p "$CACHE_DIR"/{binaries,rpms}
+
+# In build.sh, check cache before downloading
+if [[ ! -f "$CACHE_DIR/binaries/kubelet" ]]; then
+  bash tools/download-kubernetes.sh "$CACHE_DIR/binaries"
+fi
+
+cp -r "$CACHE_DIR/binaries"/* "$REPO_DIR/binaries/"
+```
+
+### Parallel Downloads
+
+```bash
+# Download binaries in parallel
+bash tools/download-kubernetes.sh "$REPO_DIR/binaries" &
+bash tools/download-etcd.sh "$REPO_DIR/binaries" &
+bash tools/download-coredns.sh "$REPO_DIR/binaries" &
+wait
+
+echo "==> All downloads complete"
+```
+
+## Reproducible Builds
+
+### Version Lock File
+
+**File**: `iso-build/versions.lock`
+
+```yaml
+kubernetes:
+  version: "1.29.1"
+  sha256:
+    kubelet: "abc123..."
+    kubectl: "def456..."
+
+etcd:
+  version: "3.5.11"
+  sha256: "789ghi..."
+
+ceph:
+  version: "18.2.1"
+
+kafka:
+  version: "3.6.1"
+  scala_version: "2.13"
+
+base_iso:
+  name: "Rocky-9.3-x86_64-boot.iso"
+  sha256: "..."
+```
+
+### Checksum Verification
+
+```bash
+#!/bin/bash
+# tools/verify-downloads.sh
+
+set -euo pipefail
+
+VERSIONS_FILE="iso-build/versions.lock"
+BINARIES_DIR="$1"
+
+while read -r binary expected_sha; do
+  actual_sha=$(sha256sum "$BINARIES_DIR/$binary" | awk '{print $1}')
+
+  if [[ "$actual_sha" != "$expected_sha" ]]; then
+    echo "ERROR: Checksum mismatch for $binary"
+    echo "  Expected: $expected_sha"
+    echo "  Actual:   $actual_sha"
+    exit 1
+  fi
+
+  echo "✓ $binary: checksum OK"
+done < <(yq eval '.kubernetes.sha256 | to_entries | .[] | .key + " " + .value' "$VERSIONS_FILE")
+```
+
+## CI/CD Integration
+
+### GitHub Actions Example
+
+```yaml
+# .github/workflows/build-iso.yml
+name: Build ISO
+
+on:
+  push:
+    tags:
+      - 'v*'
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    container:
+      image: rockylinux:9
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+
+      - name: Install build dependencies
+        run: |
+          dnf install -y lorax createrepo_c genisoimage isomd5sum
+
+      - name: Run build
+        run: |
+          bash iso-build/build.sh
+
+      - name: Upload ISO artifact
+        uses: actions/upload-artifact@v3
+        with:
+          name: cluster-node-iso
+          path: iso-build/output/cluster-node.iso
+
+      - name: Create release
+        uses: softprops/action-gh-release@v1
+        with:
+          files: iso-build/output/cluster-node.iso
+```
+
+## Troubleshooting
+
+### Build Fails: "lorax command not found"
+
+```bash
+# On Fedora
+dnf install lorax anaconda-tui
+
+# On Rocky Linux
+dnf install epel-release
+dnf install lorax
+```
+
+### ISO Won't Boot: "Missing operating system"
+
+```bash
+# Ensure isohybrid was run
+isohybrid output/cluster-node.iso
+
+# Verify boot sector
+file output/cluster-node.iso
+# Should show: "DOS/MBR boot sector"
+```
+
+### Kickstart Fails During Post-Install
+
+```bash
+# Boot ISO in rescue mode
+# Check logs
+cat /tmp/ks-post.log
+cat /tmp/anaconda.log
+
+# Common issues:
+# - Missing files in overlay
+# - Incorrect paths in kickstart
+# - Network not available during post-install
+```
+
+## Next Steps
+
+1. Create `iso-build/` directory structure
+2. Implement build.sh script
+3. Test ISO build on clean Fedora VM
+4. Test installation in QEMU
+5. Verify cluster-detect runs on first boot
+6. Test multi-node cluster formation
+7. Document customization procedures
+
+## Customization Guide
+
+### Adding Custom Packages
+
+Edit kickstart file:
+```kickstart
+%packages
+@core
+my-custom-package
+%end
+```
+
+### Changing Root Password
+
+In kickstart:
+```kickstart
+rootpw --iscrypted $6$rounds=4096$salt$hash
+```
+
+Generate hash:
+```bash
+python3 -c 'import crypt; print(crypt.crypt("your-password", crypt.mkmeth(crypt.METHOD_SHA512)))'
+```
+
+### Embedding Custom Files
+
+Add to overlay directory:
+```bash
+mkdir -p overlay/etc/my-app/
+cp my-config.yaml overlay/etc/my-app/
+```
+
+## Maintenance
+
+### Updating Base OS
+
+```bash
+# Update ROCKY_VERSION in build.sh
+ROCKY_VERSION="9.4"  # When Rocky 9.4 is released
+
+# Re-download boot ISO
+rm repo/Rocky-*.iso
+bash iso-build/build.sh
+```
+
+### Updating Kubernetes Version
+
+```bash
+# Edit versions.lock
+kubernetes:
+  version: "1.30.0"
+
+# Re-run build
+bash iso-build/build.sh
+```
+
+---
+
+**P.S.** This is the glue that makes everything real—from pile of configs to bootable artifact. Kickstart is ugly but battle-tested; lorax is RHEL-native. The build script is bash-heavy but transparent. You can see exactly what's happening, which beats "magic container build tool" when debugging at 2 AM. Test in QEMU first, VirtualBox second, bare metal third.