path: root/BINARIES.md

# Binary Acquisition and Packaging

## Overview

This document specifies exact versions, download sources, and installation paths for all software components in the cluster-from-systemd project.

## Design Decision: Hybrid Package Strategy

**Approach**: Use distro packages where stable, upstream binaries for latest features.

**Rationale**:
- **Distro packages** (dnf/yum): Base system, containerd, utilities (reliable, security updates)
- **Upstream binaries**: Kubernetes, etcd (need specific versions, faster release cycle)
- **Container images**: Kafka, Ceph (easier to version-lock, upstream-supported)
- **Compiled from source**: Only if unavoidable (minimize complexity)

## Base Distribution

**Choice**: Rocky Linux 9.3 (or Fedora 39)

**Rationale**:
- RHEL-compatible (enterprise acceptance)
- systemd 252+ (needed features)
- SELinux integration
- Long support lifecycle (Rocky: ~10 years)

**Alternative**: Fedora 39 for bleeding-edge testing, migrate to Rocky for production.

## Component Versions (2025-01-15 snapshot)

### Kubernetes Cluster
- **Kubernetes**: v1.29.1 (upstream binaries)
- **etcd**: v3.5.11 (upstream binaries)
- **containerd**: 1.7.11 (distro package)
- **runc**: 1.1.10 (distro package)
- **CNI plugins**: v1.4.0 (upstream binaries)
- **Calico**: v3.27.0 (manifest/container images)

### Storage (Ceph)
- **Ceph**: Reef 18.2.1 (distro package from CentOS SIG)
- **ceph-common**: 18.2.1
- **ceph-mon**: 18.2.1
- **ceph-osd**: 18.2.1
- **ceph-mds**: 18.2.1

### Messaging
- **Kafka**: 3.6.1 (upstream tarball)
- **OpenJDK**: 17.0.9 (distro package - required by Kafka)
- **Mosquitto**: 2.0.18 (distro package)

### DNS
- **CoreDNS**: v1.11.1 (upstream binary)

### Utilities
- **yq**: v4.40.5 (YAML processor)
- **jq**: 1.6 (distro package)
- **curl**: latest (distro)
- **openssl**: latest (distro)

## Binary Installation Paths

```
/usr/local/bin/
├── kubeadm                    # Kubernetes cluster bootstrapper
├── kubectl                    # Kubernetes CLI
├── kubelet                    # Kubernetes node agent
├── kube-apiserver            # Kubernetes API server
├── kube-controller-manager   # Kubernetes controller
├── kube-scheduler            # Kubernetes scheduler
├── etcd                      # etcd binary
├── etcdctl                   # etcd CLI
├── coredns                   # CoreDNS binary
└── yq                        # YAML processor

/usr/bin/ (via distro packages)
├── containerd
├── ctr
├── runc
├── ceph
├── ceph-mon
├── ceph-osd
├── mosquitto
├── mosquitto_passwd
├── java (OpenJDK)
└── jq

/opt/
├── kafka/                    # Kafka installation
│   ├── bin/
│   │   ├── kafka-server-start.sh
│   │   ├── kafka-topics.sh
│   │   └── ...
│   ├── config/
│   │   └── kraft/
│   └── libs/
└── cni/                      # CNI plugins
    └── bin/
        ├── bridge
        ├── host-local
        ├── loopback
        └── ...
```

## Download Sources and Installation

### 1. Kubernetes Binaries

**Version**: v1.29.1
**Source**: https://github.com/kubernetes/kubernetes/releases
**Architecture**: amd64 (x86_64)

```bash
#!/bin/bash
# tools/download-kubernetes.sh

KUBE_VERSION="v1.29.1"
ARCH="amd64"
BASE_URL="https://dl.k8s.io/release/${KUBE_VERSION}/bin/linux/${ARCH}"

BINARIES=(
  "kubeadm"
  "kubectl"
  "kubelet"
  "kube-apiserver"
  "kube-controller-manager"
  "kube-scheduler"
  "kube-proxy"
)

for binary in "${BINARIES[@]}"; do
  echo "Downloading $binary..."
  curl -L "${BASE_URL}/${binary}" -o "/usr/local/bin/${binary}"
  chmod +x "/usr/local/bin/${binary}"
done

# Verify
kubelet --version
kubectl version --client
```

**Checksums**: Download and verify
```bash
curl -L "${BASE_URL}/kubelet.sha256" -o kubelet.sha256
echo "$(cat kubelet.sha256) /usr/local/bin/kubelet" | sha256sum --check
```

### 2. etcd

**Version**: v3.5.11
**Source**: https://github.com/etcd-io/etcd/releases

```bash
#!/bin/bash
# tools/download-etcd.sh

ETCD_VERSION="v3.5.11"
ARCH="amd64"
TARBALL="etcd-${ETCD_VERSION}-linux-${ARCH}.tar.gz"
URL="https://github.com/etcd-io/etcd/releases/download/${ETCD_VERSION}/${TARBALL}"

curl -L "$URL" -o "/tmp/${TARBALL}"
tar xzf "/tmp/${TARBALL}" -C /tmp
mv /tmp/etcd-${ETCD_VERSION}-linux-${ARCH}/etcd /usr/local/bin/
mv /tmp/etcd-${ETCD_VERSION}-linux-${ARCH}/etcdctl /usr/local/bin/
chmod +x /usr/local/bin/etcd /usr/local/bin/etcdctl

# Verify
etcd --version
etcdctl version
```

### 3. containerd + runc (Distro Packages)

**Source**: Rocky Linux 9 AppStream repository

```bash
#!/bin/bash
# tools/install-container-runtime.sh

dnf install -y \
  containerd.io \
  runc \
  containernetworking-plugins

# Or from Docker repo for newer version
dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
dnf install -y containerd.io

# Configure containerd
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml

# Enable systemd cgroup driver (required for Kubernetes)
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

systemctl enable containerd
```

### 4. CNI Plugins

**Version**: v1.4.0
**Source**: https://github.com/containernetworking/plugins/releases

```bash
#!/bin/bash
# tools/download-cni-plugins.sh

CNI_VERSION="v1.4.0"
ARCH="amd64"
CNI_TARBALL="cni-plugins-linux-${ARCH}-${CNI_VERSION}.tgz"
URL="https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/${CNI_TARBALL}"

mkdir -p /opt/cni/bin
curl -L "$URL" -o "/tmp/${CNI_TARBALL}"
tar xzf "/tmp/${CNI_TARBALL}" -C /opt/cni/bin

chmod +x /opt/cni/bin/*
ls -lh /opt/cni/bin/
```

### 5. Calico CNI

**Version**: v3.27.0
**Source**: https://docs.tigera.io/calico/latest/getting-started/kubernetes/self-managed-onprem/onpremises

**Installation method**: Kubernetes manifest (applied after cluster init)

```bash
#!/bin/bash
# tools/install-calico.sh
# Run AFTER first master is initialized

CALICO_VERSION="v3.27.0"
kubectl apply -f "https://raw.githubusercontent.com/projectcalico/calico/${CALICO_VERSION}/manifests/calico.yaml"
```

**Embedded in ISO**: Download manifest at build time, apply at bootstrap.

```bash
# At ISO build time
mkdir -p /etc/kubernetes/manifests/calico
curl -L "https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml" \
  -o /etc/kubernetes/manifests/calico/calico.yaml
```

### 6. Ceph (Distro Packages)

**Version**: Reef 18.2.1
**Source**: CentOS Storage SIG repository

```bash
#!/bin/bash
# tools/install-ceph.sh

# Add Ceph repository
cat > /etc/yum.repos.d/ceph.repo <<EOF
[ceph]
name=Ceph packages for \$basearch
baseurl=https://download.ceph.com/rpm-reef/el9/\$basearch
enabled=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc

[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-reef/el9/noarch
enabled=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
EOF

# Install Ceph packages
dnf install -y \
  ceph-common \
  ceph-mon \
  ceph-osd \
  ceph-mds \
  ceph-mgr \
  python3-ceph-argparse \
  python3-cephfs

# Verify
ceph --version
```

### 7. Kafka

**Version**: 3.6.1
**Source**: https://kafka.apache.org/downloads

```bash
#!/bin/bash
# tools/install-kafka.sh

KAFKA_VERSION="3.6.1"
SCALA_VERSION="2.13"
TARBALL="kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz"
URL="https://archive.apache.org/dist/kafka/${KAFKA_VERSION}/${TARBALL}"

# Install Java (required)
dnf install -y java-17-openjdk-headless

# Download and extract Kafka
curl -L "$URL" -o "/tmp/${TARBALL}"
tar xzf "/tmp/${TARBALL}" -C /opt/
mv "/opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION}" /opt/kafka

# Create symlink for easy updates
ln -sf /opt/kafka /opt/kafka-current

# Verify
/opt/kafka/bin/kafka-topics.sh --version
```

### 8. Mosquitto (Distro Package)

**Version**: 2.0.18
**Source**: EPEL repository

```bash
#!/bin/bash
# tools/install-mosquitto.sh

# Enable EPEL
dnf install -y epel-release

# Install Mosquitto
dnf install -y mosquitto mosquitto-clients

# Verify
mosquitto -h | head -1
```

### 9. CoreDNS

**Version**: v1.11.1
**Source**: https://github.com/coredns/coredns/releases

```bash
#!/bin/bash
# tools/download-coredns.sh

COREDNS_VERSION="1.11.1"
ARCH="amd64"
TARBALL="coredns_${COREDNS_VERSION}_linux_${ARCH}.tgz"
URL="https://github.com/coredns/coredns/releases/download/v${COREDNS_VERSION}/${TARBALL}"

curl -L "$URL" -o "/tmp/${TARBALL}"
tar xzf "/tmp/${TARBALL}" -C /tmp
mv /tmp/coredns /usr/local/bin/
chmod +x /usr/local/bin/coredns

# Verify
coredns -version
```

### 10. Utilities

```bash
#!/bin/bash
# tools/install-utilities.sh

# Install from distro repos
dnf install -y \
  jq \
  curl \
  openssl \
  iproute \
  iputils \
  bind-utils \
  wget \
  vim \
  tmux \
  htop

# Install yq (YAML processor)
YQ_VERSION="v4.40.5"
YQ_BINARY="yq_linux_amd64"
curl -L "https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/${YQ_BINARY}" \
  -o /usr/local/bin/yq
chmod +x /usr/local/bin/yq

# Verify
yq --version
jq --version
```

## Complete Installation Script

**Script**: `tools/install-all-binaries.sh`

```bash
#!/bin/bash
# Master script to install all binaries in correct order

set -euo pipefail

echo "==> Installing base system packages..."
dnf update -y
dnf install -y epel-release

echo "==> Installing container runtime..."
./tools/install-container-runtime.sh

echo "==> Installing Kubernetes binaries..."
./tools/download-kubernetes.sh

echo "==> Installing etcd..."
./tools/download-etcd.sh

echo "==> Installing CNI plugins..."
./tools/download-cni-plugins.sh

echo "==> Installing Ceph..."
./tools/install-ceph.sh

echo "==> Installing Kafka..."
./tools/install-kafka.sh

echo "==> Installing Mosquitto..."
./tools/install-mosquitto.sh

echo "==> Installing CoreDNS..."
./tools/download-coredns.sh

echo "==> Installing utilities..."
./tools/install-utilities.sh

echo "==> Downloading Calico manifest..."
mkdir -p /etc/kubernetes/manifests/calico
curl -L "https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml" \
  -o /etc/kubernetes/manifests/calico/calico.yaml

echo "==> Binary installation complete!"
echo ""
echo "Installed versions:"
kubelet --version
etcd --version
containerd --version
ceph --version
java -version 2>&1 | head -1
mosquitto -h 2>&1 | head -1
coredns -version
```

## Package List for ISO Builder

**File**: `configs/package-list.txt`

```
# Base system
@core
@standard
kernel
kernel-modules
kernel-modules-extra
systemd
systemd-networkd
systemd-resolved

# Networking
NetworkManager
iproute
iputils
bind-utils
iptables
nftables

# Container runtime (from Docker repo)
containerd.io

# Development tools (for troubleshooting)
strace
tcpdump
lsof
vim
tmux
htop

# Python (for cluster scripts)
python3
python3-pip
python3-pyyaml

# Java (for Kafka)
java-17-openjdk-headless

# MQTT
mosquitto
mosquitto-clients

# Utilities
jq
curl
wget
openssl
tar
gzip
```

## Dependency Tree

```
Kubernetes
├── containerd
│   └── runc
├── CNI plugins
└── etcd

Ceph
├── python3-ceph-argparse
└── ceph-common

Kafka
└── java-17-openjdk-headless

Mosquitto
└── (no dependencies)

CoreDNS
└── (no dependencies)
```

## Version Lock Strategy

**Problem**: Prevent unexpected version changes between builds.

**Solution**: Pin exact versions in scripts, not "latest".

**Update process**:
1. Test new versions in dev environment
2. Update version variables in download scripts
3. Rebuild ISO
4. Run integration tests
5. Deploy to production

## Verification Checklist

Run after binary installation:

```bash
#!/bin/bash
# tools/verify-binaries.sh

ERRORS=0

check_binary() {
  local binary=$1
  local version_flag=${2:---version}

  if ! command -v "$binary" &>/dev/null; then
    echo "ERROR: Binary not found: $binary"
    ((ERRORS++))
  else
    echo "✓ $binary: $($binary $version_flag 2>&1 | head -1)"
  fi
}

echo "==> Verifying installed binaries..."

# Kubernetes
check_binary kubelet --version
check_binary kubectl version --client
check_binary kube-apiserver --version
check_binary kube-controller-manager --version
check_binary kube-scheduler --version

# Container runtime
check_binary containerd --version
check_binary runc --version

# etcd
check_binary etcd --version
check_binary etcdctl version

# Ceph
check_binary ceph --version

# Kafka
if [[ -x /opt/kafka/bin/kafka-topics.sh ]]; then
  echo "✓ Kafka: $(/opt/kafka/bin/kafka-topics.sh --version 2>&1)"
else
  echo "ERROR: Kafka not found"
  ((ERRORS++))
fi

# Mosquitto
check_binary mosquitto -h

# CoreDNS
check_binary coredns -version

# Utilities
check_binary yq --version
check_binary jq --version

# CNI plugins
if [[ -d /opt/cni/bin ]]; then
  echo "✓ CNI plugins: $(ls /opt/cni/bin | wc -l) binaries installed"
else
  echo "ERROR: CNI plugins directory not found"
  ((ERRORS++))
fi

if [[ $ERRORS -gt 0 ]]; then
  echo "==> Verification FAILED with $ERRORS errors"
  exit 1
fi

echo "==> Verification PASSED"
```

## Storage Requirements

**Disk space needed for binaries**:

```
Kubernetes binaries:    ~500 MB
etcd:                   ~30 MB
containerd/runc:        ~50 MB
CNI plugins:            ~40 MB
Ceph packages:          ~150 MB
Kafka:                  ~100 MB
Mosquitto:              ~5 MB
CoreDNS:                ~50 MB
Base system:            ~2 GB
-----------------------------------
Total:                  ~3 GB

Recommended ISO size:   8 GB (includes configs, logs, temp space)
```

## Update Strategy

**Minor version updates** (e.g., 1.29.1 → 1.29.2):
1. Update version variable in download script
2. Rebuild ISO
3. Test in staging

**Major version updates** (e.g., 1.29 → 1.30):
1. Review Kubernetes changelog
2. Update API versions in manifests
3. Test upgrade path (rolling update)
4. Update documentation

## Airgapped Installation

For environments without internet access:

```bash
#!/bin/bash
# tools/create-binary-bundle.sh
# Download all binaries for offline installation

BUNDLE_DIR="./binary-bundle"
mkdir -p "$BUNDLE_DIR"

# Download all binaries to local directory
# Modify download scripts to save to $BUNDLE_DIR instead of /usr/local/bin

# Create tarball
tar czf binary-bundle.tar.gz "$BUNDLE_DIR"

# On target system:
# tar xzf binary-bundle.tar.gz
# ./binary-bundle/install-offline.sh
```

## Next Steps

1. Implement all download scripts in `tools/`
2. Test binary installation on clean Rocky 9 VM
3. Create version-lock manifest (SHA256 checksums)
4. Integrate with ISO builder kickstart
5. Add to validation workflow (`verify-binaries.sh`)
6. Document upgrade procedures

---

**P.S.** This is the unglamorous but critical work—getting the right bits in the right place. Using distro packages where possible keeps maintenance sane; upstream binaries for Kubernetes give you version control. The scripts are copy-paste ready. Next doc should be **BOOTSTRAP.md** to define the initialization sequence.