more claudeHEADmain

1 files changed, 680 insertions, 0 deletions

diff --git a/BINARIES.md b/BINARIES.md
new file mode 100644
index 0000000..9e9d21b
--- /dev/null
+++ b/BINARIES.md
@@ -0,0 +1,680 @@
+# Binary Acquisition and Packaging
+
+## Overview
+
+This document specifies exact versions, download sources, and installation paths for all software components in the cluster-from-systemd project.
+
+## Design Decision: Hybrid Package Strategy
+
+**Approach**: Use distro packages where stable, upstream binaries for latest features.
+
+**Rationale**:
+- **Distro packages** (dnf/yum): Base system, containerd, utilities (reliable, security updates)
+- **Upstream binaries**: Kubernetes, etcd (need specific versions, faster release cycle)
+- **Container images**: Kafka, Ceph (easier to version-lock, upstream-supported)
+- **Compiled from source**: Only if unavoidable (minimize complexity)
+
+## Base Distribution
+
+**Choice**: Rocky Linux 9.3 (or Fedora 39)
+
+**Rationale**:
+- RHEL-compatible (enterprise acceptance)
+- systemd 252+ (needed features)
+- SELinux integration
+- Long support lifecycle (Rocky: ~10 years)
+
+**Alternative**: Fedora 39 for bleeding-edge testing, migrate to Rocky for production.
+
+## Component Versions (2025-01-15 snapshot)
+
+### Kubernetes Cluster
+- **Kubernetes**: v1.29.1 (upstream binaries)
+- **etcd**: v3.5.11 (upstream binaries)
+- **containerd**: 1.7.11 (distro package)
+- **runc**: 1.1.10 (distro package)
+- **CNI plugins**: v1.4.0 (upstream binaries)
+- **Calico**: v3.27.0 (manifest/container images)
+
+### Storage (Ceph)
+- **Ceph**: Reef 18.2.1 (distro package from CentOS SIG)
+- **ceph-common**: 18.2.1
+- **ceph-mon**: 18.2.1
+- **ceph-osd**: 18.2.1
+- **ceph-mds**: 18.2.1
+
+### Messaging
+- **Kafka**: 3.6.1 (upstream tarball)
+- **OpenJDK**: 17.0.9 (distro package - required by Kafka)
+- **Mosquitto**: 2.0.18 (distro package)
+
+### DNS
+- **CoreDNS**: v1.11.1 (upstream binary)
+
+### Utilities
+- **yq**: v4.40.5 (YAML processor)
+- **jq**: 1.6 (distro package)
+- **curl**: latest (distro)
+- **openssl**: latest (distro)
+
+## Binary Installation Paths
+
+```
+/usr/local/bin/
+├── kubeadm                    # Kubernetes cluster bootstrapper
+├── kubectl                    # Kubernetes CLI
+├── kubelet                    # Kubernetes node agent
+├── kube-apiserver            # Kubernetes API server
+├── kube-controller-manager   # Kubernetes controller
+├── kube-scheduler            # Kubernetes scheduler
+├── etcd                      # etcd binary
+├── etcdctl                   # etcd CLI
+├── coredns                   # CoreDNS binary
+└── yq                        # YAML processor
+
+/usr/bin/ (via distro packages)
+├── containerd
+├── ctr
+├── runc
+├── ceph
+├── ceph-mon
+├── ceph-osd
+├── mosquitto
+├── mosquitto_passwd
+├── java (OpenJDK)
+└── jq
+
+/opt/
+├── kafka/                    # Kafka installation
+│   ├── bin/
+│   │   ├── kafka-server-start.sh
+│   │   ├── kafka-topics.sh
+│   │   └── ...
+│   ├── config/
+│   │   └── kraft/
+│   └── libs/
+└── cni/                      # CNI plugins
+    └── bin/
+        ├── bridge
+        ├── host-local
+        ├── loopback
+        └── ...
+```
+
+## Download Sources and Installation
+
+### 1. Kubernetes Binaries
+
+**Version**: v1.29.1
+**Source**: https://github.com/kubernetes/kubernetes/releases
+**Architecture**: amd64 (x86_64)
+
+```bash
+#!/bin/bash
+# tools/download-kubernetes.sh
+
+KUBE_VERSION="v1.29.1"
+ARCH="amd64"
+BASE_URL="https://dl.k8s.io/release/${KUBE_VERSION}/bin/linux/${ARCH}"
+
+BINARIES=(
+  "kubeadm"
+  "kubectl"
+  "kubelet"
+  "kube-apiserver"
+  "kube-controller-manager"
+  "kube-scheduler"
+  "kube-proxy"
+)
+
+for binary in "${BINARIES[@]}"; do
+  echo "Downloading $binary..."
+  curl -L "${BASE_URL}/${binary}" -o "/usr/local/bin/${binary}"
+  chmod +x "/usr/local/bin/${binary}"
+done
+
+# Verify
+kubelet --version
+kubectl version --client
+```
+
+**Checksums**: Download and verify
+```bash
+curl -L "${BASE_URL}/kubelet.sha256" -o kubelet.sha256
+echo "$(cat kubelet.sha256) /usr/local/bin/kubelet" | sha256sum --check
+```
+
+### 2. etcd
+
+**Version**: v3.5.11
+**Source**: https://github.com/etcd-io/etcd/releases
+
+```bash
+#!/bin/bash
+# tools/download-etcd.sh
+
+ETCD_VERSION="v3.5.11"
+ARCH="amd64"
+TARBALL="etcd-${ETCD_VERSION}-linux-${ARCH}.tar.gz"
+URL="https://github.com/etcd-io/etcd/releases/download/${ETCD_VERSION}/${TARBALL}"
+
+curl -L "$URL" -o "/tmp/${TARBALL}"
+tar xzf "/tmp/${TARBALL}" -C /tmp
+mv /tmp/etcd-${ETCD_VERSION}-linux-${ARCH}/etcd /usr/local/bin/
+mv /tmp/etcd-${ETCD_VERSION}-linux-${ARCH}/etcdctl /usr/local/bin/
+chmod +x /usr/local/bin/etcd /usr/local/bin/etcdctl
+
+# Verify
+etcd --version
+etcdctl version
+```
+
+### 3. containerd + runc (Distro Packages)
+
+**Source**: Rocky Linux 9 AppStream repository
+
+```bash
+#!/bin/bash
+# tools/install-container-runtime.sh
+
+dnf install -y \
+  containerd.io \
+  runc \
+  containernetworking-plugins
+
+# Or from Docker repo for newer version
+dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
+dnf install -y containerd.io
+
+# Configure containerd
+mkdir -p /etc/containerd
+containerd config default > /etc/containerd/config.toml
+
+# Enable systemd cgroup driver (required for Kubernetes)
+sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
+
+systemctl enable containerd
+```
+
+### 4. CNI Plugins
+
+**Version**: v1.4.0
+**Source**: https://github.com/containernetworking/plugins/releases
+
+```bash
+#!/bin/bash
+# tools/download-cni-plugins.sh
+
+CNI_VERSION="v1.4.0"
+ARCH="amd64"
+CNI_TARBALL="cni-plugins-linux-${ARCH}-${CNI_VERSION}.tgz"
+URL="https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/${CNI_TARBALL}"
+
+mkdir -p /opt/cni/bin
+curl -L "$URL" -o "/tmp/${CNI_TARBALL}"
+tar xzf "/tmp/${CNI_TARBALL}" -C /opt/cni/bin
+
+chmod +x /opt/cni/bin/*
+ls -lh /opt/cni/bin/
+```
+
+### 5. Calico CNI
+
+**Version**: v3.27.0
+**Source**: https://docs.tigera.io/calico/latest/getting-started/kubernetes/self-managed-onprem/onpremises
+
+**Installation method**: Kubernetes manifest (applied after cluster init)
+
+```bash
+#!/bin/bash
+# tools/install-calico.sh
+# Run AFTER first master is initialized
+
+CALICO_VERSION="v3.27.0"
+kubectl apply -f "https://raw.githubusercontent.com/projectcalico/calico/${CALICO_VERSION}/manifests/calico.yaml"
+```
+
+**Embedded in ISO**: Download manifest at build time, apply at bootstrap.
+
+```bash
+# At ISO build time
+mkdir -p /etc/kubernetes/manifests/calico
+curl -L "https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml" \
+  -o /etc/kubernetes/manifests/calico/calico.yaml
+```
+
+### 6. Ceph (Distro Packages)
+
+**Version**: Reef 18.2.1
+**Source**: CentOS Storage SIG repository
+
+```bash
+#!/bin/bash
+# tools/install-ceph.sh
+
+# Add Ceph repository
+cat > /etc/yum.repos.d/ceph.repo <<EOF
+[ceph]
+name=Ceph packages for \$basearch
+baseurl=https://download.ceph.com/rpm-reef/el9/\$basearch
+enabled=1
+gpgcheck=1
+gpgkey=https://download.ceph.com/keys/release.asc
+
+[ceph-noarch]
+name=Ceph noarch packages
+baseurl=https://download.ceph.com/rpm-reef/el9/noarch
+enabled=1
+gpgcheck=1
+gpgkey=https://download.ceph.com/keys/release.asc
+EOF
+
+# Install Ceph packages
+dnf install -y \
+  ceph-common \
+  ceph-mon \
+  ceph-osd \
+  ceph-mds \
+  ceph-mgr \
+  python3-ceph-argparse \
+  python3-cephfs
+
+# Verify
+ceph --version
+```
+
+### 7. Kafka
+
+**Version**: 3.6.1
+**Source**: https://kafka.apache.org/downloads
+
+```bash
+#!/bin/bash
+# tools/install-kafka.sh
+
+KAFKA_VERSION="3.6.1"
+SCALA_VERSION="2.13"
+TARBALL="kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz"
+URL="https://archive.apache.org/dist/kafka/${KAFKA_VERSION}/${TARBALL}"
+
+# Install Java (required)
+dnf install -y java-17-openjdk-headless
+
+# Download and extract Kafka
+curl -L "$URL" -o "/tmp/${TARBALL}"
+tar xzf "/tmp/${TARBALL}" -C /opt/
+mv "/opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION}" /opt/kafka
+
+# Create symlink for easy updates
+ln -sf /opt/kafka /opt/kafka-current
+
+# Verify
+/opt/kafka/bin/kafka-topics.sh --version
+```
+
+### 8. Mosquitto (Distro Package)
+
+**Version**: 2.0.18
+**Source**: EPEL repository
+
+```bash
+#!/bin/bash
+# tools/install-mosquitto.sh
+
+# Enable EPEL
+dnf install -y epel-release
+
+# Install Mosquitto
+dnf install -y mosquitto mosquitto-clients
+
+# Verify
+mosquitto -h | head -1
+```
+
+### 9. CoreDNS
+
+**Version**: v1.11.1
+**Source**: https://github.com/coredns/coredns/releases
+
+```bash
+#!/bin/bash
+# tools/download-coredns.sh
+
+COREDNS_VERSION="1.11.1"
+ARCH="amd64"
+TARBALL="coredns_${COREDNS_VERSION}_linux_${ARCH}.tgz"
+URL="https://github.com/coredns/coredns/releases/download/v${COREDNS_VERSION}/${TARBALL}"
+
+curl -L "$URL" -o "/tmp/${TARBALL}"
+tar xzf "/tmp/${TARBALL}" -C /tmp
+mv /tmp/coredns /usr/local/bin/
+chmod +x /usr/local/bin/coredns
+
+# Verify
+coredns -version
+```
+
+### 10. Utilities
+
+```bash
+#!/bin/bash
+# tools/install-utilities.sh
+
+# Install from distro repos
+dnf install -y \
+  jq \
+  curl \
+  openssl \
+  iproute \
+  iputils \
+  bind-utils \
+  wget \
+  vim \
+  tmux \
+  htop
+
+# Install yq (YAML processor)
+YQ_VERSION="v4.40.5"
+YQ_BINARY="yq_linux_amd64"
+curl -L "https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/${YQ_BINARY}" \
+  -o /usr/local/bin/yq
+chmod +x /usr/local/bin/yq
+
+# Verify
+yq --version
+jq --version
+```
+
+## Complete Installation Script
+
+**Script**: `tools/install-all-binaries.sh`
+
+```bash
+#!/bin/bash
+# Master script to install all binaries in correct order
+
+set -euo pipefail
+
+echo "==> Installing base system packages..."
+dnf update -y
+dnf install -y epel-release
+
+echo "==> Installing container runtime..."
+./tools/install-container-runtime.sh
+
+echo "==> Installing Kubernetes binaries..."
+./tools/download-kubernetes.sh
+
+echo "==> Installing etcd..."
+./tools/download-etcd.sh
+
+echo "==> Installing CNI plugins..."
+./tools/download-cni-plugins.sh
+
+echo "==> Installing Ceph..."
+./tools/install-ceph.sh
+
+echo "==> Installing Kafka..."
+./tools/install-kafka.sh
+
+echo "==> Installing Mosquitto..."
+./tools/install-mosquitto.sh
+
+echo "==> Installing CoreDNS..."
+./tools/download-coredns.sh
+
+echo "==> Installing utilities..."
+./tools/install-utilities.sh
+
+echo "==> Downloading Calico manifest..."
+mkdir -p /etc/kubernetes/manifests/calico
+curl -L "https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml" \
+  -o /etc/kubernetes/manifests/calico/calico.yaml
+
+echo "==> Binary installation complete!"
+echo ""
+echo "Installed versions:"
+kubelet --version
+etcd --version
+containerd --version
+ceph --version
+java -version 2>&1 | head -1
+mosquitto -h 2>&1 | head -1
+coredns -version
+```
+
+## Package List for ISO Builder
+
+**File**: `configs/package-list.txt`
+
+```
+# Base system
+@core
+@standard
+kernel
+kernel-modules
+kernel-modules-extra
+systemd
+systemd-networkd
+systemd-resolved
+
+# Networking
+NetworkManager
+iproute
+iputils
+bind-utils
+iptables
+nftables
+
+# Container runtime (from Docker repo)
+containerd.io
+
+# Development tools (for troubleshooting)
+strace
+tcpdump
+lsof
+vim
+tmux
+htop
+
+# Python (for cluster scripts)
+python3
+python3-pip
+python3-pyyaml
+
+# Java (for Kafka)
+java-17-openjdk-headless
+
+# MQTT
+mosquitto
+mosquitto-clients
+
+# Utilities
+jq
+curl
+wget
+openssl
+tar
+gzip
+```
+
+## Dependency Tree
+
+```
+Kubernetes
+├── containerd
+│   └── runc
+├── CNI plugins
+└── etcd
+
+Ceph
+├── python3-ceph-argparse
+└── ceph-common
+
+Kafka
+└── java-17-openjdk-headless
+
+Mosquitto
+└── (no dependencies)
+
+CoreDNS
+└── (no dependencies)
+```
+
+## Version Lock Strategy
+
+**Problem**: Prevent unexpected version changes between builds.
+
+**Solution**: Pin exact versions in scripts, not "latest".
+
+**Update process**:
+1. Test new versions in dev environment
+2. Update version variables in download scripts
+3. Rebuild ISO
+4. Run integration tests
+5. Deploy to production
+
+## Verification Checklist
+
+Run after binary installation:
+
+```bash
+#!/bin/bash
+# tools/verify-binaries.sh
+
+ERRORS=0
+
+check_binary() {
+  local binary=$1
+  local version_flag=${2:---version}
+
+  if ! command -v "$binary" &>/dev/null; then
+    echo "ERROR: Binary not found: $binary"
+    ((ERRORS++))
+  else
+    echo "✓ $binary: $($binary $version_flag 2>&1 | head -1)"
+  fi
+}
+
+echo "==> Verifying installed binaries..."
+
+# Kubernetes
+check_binary kubelet --version
+check_binary kubectl version --client
+check_binary kube-apiserver --version
+check_binary kube-controller-manager --version
+check_binary kube-scheduler --version
+
+# Container runtime
+check_binary containerd --version
+check_binary runc --version
+
+# etcd
+check_binary etcd --version
+check_binary etcdctl version
+
+# Ceph
+check_binary ceph --version
+
+# Kafka
+if [[ -x /opt/kafka/bin/kafka-topics.sh ]]; then
+  echo "✓ Kafka: $(/opt/kafka/bin/kafka-topics.sh --version 2>&1)"
+else
+  echo "ERROR: Kafka not found"
+  ((ERRORS++))
+fi
+
+# Mosquitto
+check_binary mosquitto -h
+
+# CoreDNS
+check_binary coredns -version
+
+# Utilities
+check_binary yq --version
+check_binary jq --version
+
+# CNI plugins
+if [[ -d /opt/cni/bin ]]; then
+  echo "✓ CNI plugins: $(ls /opt/cni/bin | wc -l) binaries installed"
+else
+  echo "ERROR: CNI plugins directory not found"
+  ((ERRORS++))
+fi
+
+if [[ $ERRORS -gt 0 ]]; then
+  echo "==> Verification FAILED with $ERRORS errors"
+  exit 1
+fi
+
+echo "==> Verification PASSED"
+```
+
+## Storage Requirements
+
+**Disk space needed for binaries**:
+
+```
+Kubernetes binaries:    ~500 MB
+etcd:                   ~30 MB
+containerd/runc:        ~50 MB
+CNI plugins:            ~40 MB
+Ceph packages:          ~150 MB
+Kafka:                  ~100 MB
+Mosquitto:              ~5 MB
+CoreDNS:                ~50 MB
+Base system:            ~2 GB
+-----------------------------------
+Total:                  ~3 GB
+
+Recommended ISO size:   8 GB (includes configs, logs, temp space)
+```
+
+## Update Strategy
+
+**Minor version updates** (e.g., 1.29.1 → 1.29.2):
+1. Update version variable in download script
+2. Rebuild ISO
+3. Test in staging
+
+**Major version updates** (e.g., 1.29 → 1.30):
+1. Review Kubernetes changelog
+2. Update API versions in manifests
+3. Test upgrade path (rolling update)
+4. Update documentation
+
+## Airgapped Installation
+
+For environments without internet access:
+
+```bash
+#!/bin/bash
+# tools/create-binary-bundle.sh
+# Download all binaries for offline installation
+
+BUNDLE_DIR="./binary-bundle"
+mkdir -p "$BUNDLE_DIR"
+
+# Download all binaries to local directory
+# Modify download scripts to save to $BUNDLE_DIR instead of /usr/local/bin
+
+# Create tarball
+tar czf binary-bundle.tar.gz "$BUNDLE_DIR"
+
+# On target system:
+# tar xzf binary-bundle.tar.gz
+# ./binary-bundle/install-offline.sh
+```
+
+## Next Steps
+
+1. Implement all download scripts in `tools/`
+2. Test binary installation on clean Rocky 9 VM
+3. Create version-lock manifest (SHA256 checksums)
+4. Integrate with ISO builder kickstart
+5. Add to validation workflow (`verify-binaries.sh`)
+6. Document upgrade procedures
+
+---
+
+**P.S.** This is the unglamorous but critical work—getting the right bits in the right place. Using distro packages where possible keeps maintenance sane; upstream binaries for Kubernetes give you version control. The scripts are copy-paste ready. Next doc should be **BOOTSTRAP.md** to define the initialization sequence.