path: root/CERTIFICATES.md

# Certificate and Key Management

## Overview

This document specifies the complete PKI infrastructure required for the cluster-from-systemd project. All services requiring mutual TLS authentication or encrypted communication need certificates generated before they can start.

## Design Decision: Hybrid Approach

**Strategy**: Pre-generate CA and static certs at ISO build time, generate node-specific certs on first boot.

**Rationale**:
- **Build-time**: CA certificates, service account keys (shared across cluster)
- **Boot-time**: Node-specific certs (kubelet, etcd peer certs tied to IP/hostname)
- **Avoids**: Chicken-and-egg problem with multi-master coordination
- **Security**: CA private key embedded in ISO (acceptable for isolated clusters; can be removed post-bootstrap for production)

## Certificate Inventory

### Kubernetes PKI (10 certificate pairs + 1 key)

```
/etc/kubernetes/pki/
├── ca.crt                              # Kubernetes CA certificate
├── ca.key                              # Kubernetes CA private key
├── apiserver.crt                       # API server certificate
├── apiserver.key                       # API server private key
├── apiserver-kubelet-client.crt        # API server → kubelet client cert
├── apiserver-kubelet-client.key
├── front-proxy-ca.crt                  # Front proxy CA
├── front-proxy-ca.key
├── front-proxy-client.crt              # Front proxy client
├── front-proxy-client.key
├── sa.pub                              # Service account public key
├── sa.key                              # Service account private key (RSA)
└── etcd/
    ├── ca.crt                          # etcd CA certificate
    ├── ca.key                          # etcd CA private key
    ├── server.crt                      # etcd server certificate (PER NODE)
    ├── server.key                      # etcd server private key (PER NODE)
    ├── peer.crt                        # etcd peer certificate (PER NODE)
    ├── peer.key                        # etcd peer private key (PER NODE)
    ├── healthcheck-client.crt          # etcd health check client
    └── healthcheck-client.key
```

**Per-node certificates** (generated at boot):
```
/var/lib/kubelet/pki/
├── kubelet.crt                         # Node-specific kubelet server cert
├── kubelet.key                         # Node-specific kubelet private key
└── kubelet-client-current.pem          # Kubelet client cert (CSR auto-rotation)
```

### Ceph Authentication (4 keyrings)

```
/etc/ceph/
├── ceph.conf                           # Cluster configuration
├── ceph.client.admin.keyring           # Admin keyring
├── ceph.mon.keyring                    # Monitor keyring
├── ceph.bootstrap-osd.keyring          # OSD bootstrap keyring
└── ceph.bootstrap-mds.keyring          # MDS bootstrap keyring
```

### MQTT Authentication (1 password file)

```
/etc/mosquitto/
└── passwd                              # Password file (mosquitto_passwd format)
```

## Certificate Generation Scripts

### 1. Build-Time: Generate Cluster-Wide Certs

**Script**: `tools/generate-build-certs.sh`

```bash
#!/bin/bash
# Generates all CA certificates and shared keys at ISO build time
# Output: certs/ directory to be embedded in ISO

set -euo pipefail

CERT_DIR="${1:-./certs}"
mkdir -p "$CERT_DIR"/{kubernetes/pki/etcd,ceph,mqtt}

echo "==> Generating Kubernetes CA certificates..."

# 1. Kubernetes CA
openssl genrsa -out "$CERT_DIR/kubernetes/pki/ca.key" 2048
openssl req -x509 -new -nodes -key "$CERT_DIR/kubernetes/pki/ca.key" \
  -subj "/CN=kubernetes-ca" \
  -days 3650 -out "$CERT_DIR/kubernetes/pki/ca.crt"

# 2. Front Proxy CA
openssl genrsa -out "$CERT_DIR/kubernetes/pki/front-proxy-ca.key" 2048
openssl req -x509 -new -nodes -key "$CERT_DIR/kubernetes/pki/front-proxy-ca.key" \
  -subj "/CN=kubernetes-front-proxy-ca" \
  -days 3650 -out "$CERT_DIR/kubernetes/pki/front-proxy-ca.crt"

# 3. etcd CA
openssl genrsa -out "$CERT_DIR/kubernetes/pki/etcd/ca.key" 2048
openssl req -x509 -new -nodes -key "$CERT_DIR/kubernetes/pki/etcd/ca.key" \
  -subj "/CN=etcd-ca" \
  -days 3650 -out "$CERT_DIR/kubernetes/pki/etcd/ca.crt"

# 4. Service Account Key Pair (RSA for signing JWTs)
openssl genrsa -out "$CERT_DIR/kubernetes/pki/sa.key" 2048
openssl rsa -in "$CERT_DIR/kubernetes/pki/sa.key" \
  -pubout -out "$CERT_DIR/kubernetes/pki/sa.pub"

# 5. Front Proxy Client (static - same for all API servers)
openssl genrsa -out "$CERT_DIR/kubernetes/pki/front-proxy-client.key" 2048
openssl req -new -key "$CERT_DIR/kubernetes/pki/front-proxy-client.key" \
  -subj "/CN=front-proxy-client" \
  -out "$CERT_DIR/kubernetes/pki/front-proxy-client.csr"
openssl x509 -req -in "$CERT_DIR/kubernetes/pki/front-proxy-client.csr" \
  -CA "$CERT_DIR/kubernetes/pki/front-proxy-ca.crt" \
  -CAkey "$CERT_DIR/kubernetes/pki/front-proxy-ca.key" \
  -CAcreateserial -days 3650 \
  -out "$CERT_DIR/kubernetes/pki/front-proxy-client.crt"

echo "==> Generating Ceph keyrings..."

# Ceph uses cephx authentication (symmetric keys)
# These will be generated using ceph-authtool at first-boot of first monitor
# For now, create placeholder directory
touch "$CERT_DIR/ceph/.placeholder"

echo "==> Generating MQTT password file..."

# Default admin user (change password post-install!)
# Password: "admin" (hashed)
mosquitto_passwd -c -b "$CERT_DIR/mqtt/passwd" admin admin 2>/dev/null || {
  echo "WARNING: mosquitto_passwd not found. Creating empty file."
  touch "$CERT_DIR/mqtt/passwd"
}

echo "==> Build-time certificates generated in $CERT_DIR"
ls -lhR "$CERT_DIR"
```

### 2. First Boot (Master): Generate Node-Specific Certs

**Script**: `tools/generate-node-certs.sh`

```bash
#!/bin/bash
# Generates node-specific certificates on first boot
# Run by cluster-detect.service after node identity established

set -euo pipefail

CONFIG_DIR="${CONFIG_DIR:-/etc/cluster-config}"
NODE_NAME=$(cat "$CONFIG_DIR/node-identity")
NODE_CONFIG="$CONFIG_DIR/nodes/$NODE_NAME.yaml"

# Extract node IP from current-node.yaml
NODE_IP=$(yq eval '.node.ip' "$CONFIG_DIR/current-node.yaml")
CLUSTER_NAME=$(yq eval '.cluster.name' "$CONFIG_DIR/cluster.yaml")
SERVICE_CIDR=$(yq eval '.cluster.networking.service_cidr' "$CONFIG_DIR/cluster.yaml")

# Kubernetes API VIP (first IP in service CIDR)
KUBERNETES_SVC_IP=$(echo "$SERVICE_CIDR" | awk -F'[./]' '{print $1"."$2"."$3".1"}')

PKI_DIR="/etc/kubernetes/pki"
ETCD_PKI_DIR="$PKI_DIR/etcd"

echo "==> Generating certificates for node: $NODE_NAME ($NODE_IP)"

# Check if node is a master
ROLES=$(yq eval '.node.roles[]' "$NODE_CONFIG")
IS_MASTER=false
[[ "$ROLES" =~ "master" || "$ROLES" =~ "control-plane" ]] && IS_MASTER=true

if [[ "$IS_MASTER" == "true" ]]; then
  echo "==> Generating API server certificate..."

  # Create OpenSSL config for SAN
  cat > /tmp/apiserver-csr.conf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
DNS.5 = $NODE_NAME
DNS.6 = $CLUSTER_NAME
IP.1 = $KUBERNETES_SVC_IP
IP.2 = $NODE_IP
IP.3 = 127.0.0.1
EOF

  openssl genrsa -out "$PKI_DIR/apiserver.key" 2048
  openssl req -new -key "$PKI_DIR/apiserver.key" \
    -subj "/CN=kube-apiserver" \
    -config /tmp/apiserver-csr.conf \
    -out "$PKI_DIR/apiserver.csr"
  openssl x509 -req -in "$PKI_DIR/apiserver.csr" \
    -CA "$PKI_DIR/ca.crt" -CAkey "$PKI_DIR/ca.key" \
    -CAcreateserial -days 3650 \
    -extensions v3_req -extfile /tmp/apiserver-csr.conf \
    -out "$PKI_DIR/apiserver.crt"

  echo "==> Generating API server kubelet client certificate..."
  openssl genrsa -out "$PKI_DIR/apiserver-kubelet-client.key" 2048
  openssl req -new -key "$PKI_DIR/apiserver-kubelet-client.key" \
    -subj "/CN=kube-apiserver-kubelet-client/O=system:masters" \
    -out "$PKI_DIR/apiserver-kubelet-client.csr"
  openssl x509 -req -in "$PKI_DIR/apiserver-kubelet-client.csr" \
    -CA "$PKI_DIR/ca.crt" -CAkey "$PKI_DIR/ca.key" \
    -CAcreateserial -days 3650 \
    -out "$PKI_DIR/apiserver-kubelet-client.crt"

  echo "==> Generating etcd server certificate..."
  cat > /tmp/etcd-server-csr.conf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = $NODE_NAME
DNS.2 = localhost
IP.1 = $NODE_IP
IP.2 = 127.0.0.1
EOF

  openssl genrsa -out "$ETCD_PKI_DIR/server.key" 2048
  openssl req -new -key "$ETCD_PKI_DIR/server.key" \
    -subj "/CN=$NODE_NAME" \
    -config /tmp/etcd-server-csr.conf \
    -out "$ETCD_PKI_DIR/server.csr"
  openssl x509 -req -in "$ETCD_PKI_DIR/server.csr" \
    -CA "$ETCD_PKI_DIR/ca.crt" -CAkey "$ETCD_PKI_DIR/ca.key" \
    -CAcreateserial -days 3650 \
    -extensions v3_req -extfile /tmp/etcd-server-csr.conf \
    -out "$ETCD_PKI_DIR/server.crt"

  echo "==> Generating etcd peer certificate..."
  openssl genrsa -out "$ETCD_PKI_DIR/peer.key" 2048
  openssl req -new -key "$ETCD_PKI_DIR/peer.key" \
    -subj "/CN=$NODE_NAME" \
    -config /tmp/etcd-server-csr.conf \
    -out "$ETCD_PKI_DIR/peer.csr"
  openssl x509 -req -in "$ETCD_PKI_DIR/peer.csr" \
    -CA "$ETCD_PKI_DIR/ca.crt" -CAkey "$ETCD_PKI_DIR/ca.key" \
    -CAcreateserial -days 3650 \
    -extensions v3_req -extfile /tmp/etcd-server-csr.conf \
    -out "$ETCD_PKI_DIR/peer.crt"

  echo "==> Generating etcd healthcheck client certificate..."
  openssl genrsa -out "$ETCD_PKI_DIR/healthcheck-client.key" 2048
  openssl req -new -key "$ETCD_PKI_DIR/healthcheck-client.key" \
    -subj "/CN=kube-etcd-healthcheck-client/O=system:masters" \
    -out "$ETCD_PKI_DIR/healthcheck-client.csr"
  openssl x509 -req -in "$ETCD_PKI_DIR/healthcheck-client.csr" \
    -CA "$ETCD_PKI_DIR/ca.crt" -CAkey "$ETCD_PKI_DIR/ca.key" \
    -CAcreateserial -days 3650 \
    -out "$ETCD_PKI_DIR/healthcheck-client.crt"

  # Set permissions
  chmod 600 "$PKI_DIR"/*.key "$ETCD_PKI_DIR"/*.key
fi

echo "==> Generating kubelet certificate..."
mkdir -p /var/lib/kubelet/pki

cat > /tmp/kubelet-csr.conf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = $NODE_NAME
IP.1 = $NODE_IP
EOF

openssl genrsa -out /var/lib/kubelet/pki/kubelet.key 2048
openssl req -new -key /var/lib/kubelet/pki/kubelet.key \
  -subj "/CN=system:node:$NODE_NAME/O=system:nodes" \
  -config /tmp/kubelet-csr.conf \
  -out /var/lib/kubelet/pki/kubelet.csr
openssl x509 -req -in /var/lib/kubelet/pki/kubelet.csr \
  -CA "$PKI_DIR/ca.crt" -CAkey "$PKI_DIR/ca.key" \
  -CAcreateserial -days 3650 \
  -extensions v3_req -extfile /tmp/kubelet-csr.conf \
  -out /var/lib/kubelet/pki/kubelet.crt

chmod 600 /var/lib/kubelet/pki/kubelet.key

echo "==> Node certificates generated successfully"
```

### 3. First Boot (First Master): Generate Ceph Keys

**Script**: `tools/generate-ceph-keys.sh`

```bash
#!/bin/bash
# Generates Ceph authentication keys on first monitor node
# Only run on the FIRST ceph-mon node

set -euo pipefail

CEPH_DIR="/etc/ceph"
mkdir -p "$CEPH_DIR"

echo "==> Generating Ceph cluster FSID..."
FSID=$(uuidgen)
echo "$FSID" > "$CEPH_DIR/cluster.fsid"

echo "==> Generating Ceph monitor keyring..."
ceph-authtool --create-keyring "$CEPH_DIR/ceph.mon.keyring" \
  --gen-key -n mon. --cap mon 'allow *'

echo "==> Generating Ceph admin keyring..."
ceph-authtool --create-keyring "$CEPH_DIR/ceph.client.admin.keyring" \
  --gen-key -n client.admin \
  --cap mon 'allow *' \
  --cap osd 'allow *' \
  --cap mds 'allow *' \
  --cap mgr 'allow *'

echo "==> Generating OSD bootstrap keyring..."
ceph-authtool --create-keyring "$CEPH_DIR/ceph.bootstrap-osd.keyring" \
  --gen-key -n client.bootstrap-osd \
  --cap mon 'allow profile bootstrap-osd' \
  --cap mgr 'allow r'

echo "==> Generating MDS bootstrap keyring..."
ceph-authtool --create-keyring "$CEPH_DIR/ceph.bootstrap-mds.keyring" \
  --gen-key -n client.bootstrap-mds \
  --cap mon 'allow profile bootstrap-mds' \
  --cap mgr 'allow r'

# Import admin keyring into mon keyring
ceph-authtool "$CEPH_DIR/ceph.mon.keyring" --import-keyring "$CEPH_DIR/ceph.client.admin.keyring"

chmod 600 "$CEPH_DIR"/*.keyring

echo "==> Ceph keys generated. FSID: $FSID"
echo "==> These keys must be distributed to all ceph nodes!"
```

## Integration with Boot Flow

### Modified Boot Sequence

```
1. cluster-detect.service
   ├─> cluster-detect.sh (identify node)
   └─> generate-node-certs.sh (NEW - generate node-specific certs)

2. generate-environment-files.sh
   (reads certs, creates .env files with cert paths)

3. cluster-activate-roles.sh
   (enables targets based on roles)

4. Services start
   (now have valid certificates)
```

### Updated cluster-detect.service

```ini
[Unit]
Description=Cluster Node Detection and Certificate Generation
DefaultDependencies=no
Before=network-pre.target
Wants=network-pre.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/cluster-detect.sh
ExecStart=/usr/local/bin/generate-node-certs.sh
ExecStart=/usr/local/bin/generate-environment-files.sh
ExecStart=/usr/local/bin/cluster-activate-roles.sh

[Install]
WantedBy=multi-user.target
```

## Certificate Distribution for Multi-Master

**Problem**: Masters 2+ need the CA private keys to sign their own certs.

**Solutions** (pick one):

### Option A: Embed CA keys in ISO (simple, less secure)
- All CA private keys baked into ISO
- Each master generates its own node certs using shared CA
- **Remove CA keys after cluster init** via post-bootstrap script

### Option B: Fetch from first master (secure, complex)
- First master generates CA, serves via HTTPS
- Additional masters fetch CA bundle during bootstrap
- Requires authentication (shared secret in cluster.yaml)

### Option C: External secret store (production)
- Build-time: Upload CA to Vault/AWS Secrets Manager
- Boot-time: Fetch CA using node identity proof
- Requires external dependency

**Recommendation**: Start with Option A, migrate to Option C for production.

## Security Hardening Checklist

- [ ] Set file permissions: 600 for `.key`, 644 for `.crt`
- [ ] Remove CA private keys after all masters bootstrapped
- [ ] Rotate service account keys annually
- [ ] Enable Kubernetes certificate rotation (kubelet `--rotate-certificates`)
- [ ] Monitor certificate expiry (90 days before expiration)
- [ ] Store root CA offline (not in ISO for production)
- [ ] Use hardware security module (HSM) for CA keys (production)
- [ ] Implement certificate revocation list (CRL)

## Validation Script

**Script**: `tools/validate-certs.sh`

```bash
#!/bin/bash
# Validates all certificates before service startup

set -euo pipefail

PKI_DIR="/etc/kubernetes/pki"
ERRORS=0

check_cert() {
  local cert=$1
  local key=$2

  if [[ ! -f "$cert" ]]; then
    echo "ERROR: Certificate not found: $cert"
    ((ERRORS++))
    return
  fi

  if [[ ! -f "$key" ]]; then
    echo "ERROR: Private key not found: $key"
    ((ERRORS++))
    return
  fi

  # Check expiry
  if ! openssl x509 -in "$cert" -noout -checkend 86400; then
    echo "WARNING: Certificate expires within 24 hours: $cert"
  fi

  # Verify key matches cert
  cert_modulus=$(openssl x509 -in "$cert" -noout -modulus)
  key_modulus=$(openssl rsa -in "$key" -noout -modulus)

  if [[ "$cert_modulus" != "$key_modulus" ]]; then
    echo "ERROR: Certificate and key mismatch: $cert / $key"
    ((ERRORS++))
  fi
}

echo "==> Validating Kubernetes certificates..."
check_cert "$PKI_DIR/ca.crt" "$PKI_DIR/ca.key"
check_cert "$PKI_DIR/apiserver.crt" "$PKI_DIR/apiserver.key"

if [[ $ERRORS -gt 0 ]]; then
  echo "==> Validation FAILED with $ERRORS errors"
  exit 1
fi

echo "==> Validation PASSED"
```

## Next Steps

1. Implement `tools/generate-build-certs.sh`
2. Implement `tools/generate-node-certs.sh`
3. Implement `tools/generate-ceph-keys.sh`
4. Update `cluster-detect.service` to call cert generation
5. Update service units to validate certs in `ExecStartPre=`
6. Test certificate generation in VM
7. Document CA key removal procedure for production

---

**P.S.** This gives you a working PKI. It's not production-grade (CA keys in ISO is a compromise), but it unblocks development. You can swap to Vault/external PKI later without changing the service units—they just read from `/etc/kubernetes/pki/`.