diff options
Diffstat (limited to 'CERTIFICATES.md')
| -rw-r--r-- | CERTIFICATES.md | 506 |
1 files changed, 506 insertions, 0 deletions
diff --git a/CERTIFICATES.md b/CERTIFICATES.md new file mode 100644 index 0000000..26ca71b --- /dev/null +++ b/CERTIFICATES.md @@ -0,0 +1,506 @@ +# Certificate and Key Management + +## Overview + +This document specifies the complete PKI infrastructure required for the cluster-from-systemd project. All services requiring mutual TLS authentication or encrypted communication need certificates generated before they can start. + +## Design Decision: Hybrid Approach + +**Strategy**: Pre-generate CA and static certs at ISO build time, generate node-specific certs on first boot. + +**Rationale**: +- **Build-time**: CA certificates, service account keys (shared across cluster) +- **Boot-time**: Node-specific certs (kubelet, etcd peer certs tied to IP/hostname) +- **Avoids**: Chicken-and-egg problem with multi-master coordination +- **Security**: CA private key embedded in ISO (acceptable for isolated clusters; can be removed post-bootstrap for production) + +## Certificate Inventory + +### Kubernetes PKI (10 certificate pairs + 1 key) + +``` +/etc/kubernetes/pki/ +├── ca.crt # Kubernetes CA certificate +├── ca.key # Kubernetes CA private key +├── apiserver.crt # API server certificate +├── apiserver.key # API server private key +├── apiserver-kubelet-client.crt # API server → kubelet client cert +├── apiserver-kubelet-client.key +├── front-proxy-ca.crt # Front proxy CA +├── front-proxy-ca.key +├── front-proxy-client.crt # Front proxy client +├── front-proxy-client.key +├── sa.pub # Service account public key +├── sa.key # Service account private key (RSA) +└── etcd/ + ├── ca.crt # etcd CA certificate + ├── ca.key # etcd CA private key + ├── server.crt # etcd server certificate (PER NODE) + ├── server.key # etcd server private key (PER NODE) + ├── peer.crt # etcd peer certificate (PER NODE) + ├── peer.key # etcd peer private key (PER NODE) + ├── healthcheck-client.crt # etcd health check client + └── healthcheck-client.key +``` + +**Per-node certificates** (generated at boot): +``` +/var/lib/kubelet/pki/ +├── kubelet.crt # Node-specific kubelet server cert +├── kubelet.key # Node-specific kubelet private key +└── kubelet-client-current.pem # Kubelet client cert (CSR auto-rotation) +``` + +### Ceph Authentication (4 keyrings) + +``` +/etc/ceph/ +├── ceph.conf # Cluster configuration +├── ceph.client.admin.keyring # Admin keyring +├── ceph.mon.keyring # Monitor keyring +├── ceph.bootstrap-osd.keyring # OSD bootstrap keyring +└── ceph.bootstrap-mds.keyring # MDS bootstrap keyring +``` + +### MQTT Authentication (1 password file) + +``` +/etc/mosquitto/ +└── passwd # Password file (mosquitto_passwd format) +``` + +## Certificate Generation Scripts + +### 1. Build-Time: Generate Cluster-Wide Certs + +**Script**: `tools/generate-build-certs.sh` + +```bash +#!/bin/bash +# Generates all CA certificates and shared keys at ISO build time +# Output: certs/ directory to be embedded in ISO + +set -euo pipefail + +CERT_DIR="${1:-./certs}" +mkdir -p "$CERT_DIR"/{kubernetes/pki/etcd,ceph,mqtt} + +echo "==> Generating Kubernetes CA certificates..." + +# 1. Kubernetes CA +openssl genrsa -out "$CERT_DIR/kubernetes/pki/ca.key" 2048 +openssl req -x509 -new -nodes -key "$CERT_DIR/kubernetes/pki/ca.key" \ + -subj "/CN=kubernetes-ca" \ + -days 3650 -out "$CERT_DIR/kubernetes/pki/ca.crt" + +# 2. Front Proxy CA +openssl genrsa -out "$CERT_DIR/kubernetes/pki/front-proxy-ca.key" 2048 +openssl req -x509 -new -nodes -key "$CERT_DIR/kubernetes/pki/front-proxy-ca.key" \ + -subj "/CN=kubernetes-front-proxy-ca" \ + -days 3650 -out "$CERT_DIR/kubernetes/pki/front-proxy-ca.crt" + +# 3. etcd CA +openssl genrsa -out "$CERT_DIR/kubernetes/pki/etcd/ca.key" 2048 +openssl req -x509 -new -nodes -key "$CERT_DIR/kubernetes/pki/etcd/ca.key" \ + -subj "/CN=etcd-ca" \ + -days 3650 -out "$CERT_DIR/kubernetes/pki/etcd/ca.crt" + +# 4. Service Account Key Pair (RSA for signing JWTs) +openssl genrsa -out "$CERT_DIR/kubernetes/pki/sa.key" 2048 +openssl rsa -in "$CERT_DIR/kubernetes/pki/sa.key" \ + -pubout -out "$CERT_DIR/kubernetes/pki/sa.pub" + +# 5. Front Proxy Client (static - same for all API servers) +openssl genrsa -out "$CERT_DIR/kubernetes/pki/front-proxy-client.key" 2048 +openssl req -new -key "$CERT_DIR/kubernetes/pki/front-proxy-client.key" \ + -subj "/CN=front-proxy-client" \ + -out "$CERT_DIR/kubernetes/pki/front-proxy-client.csr" +openssl x509 -req -in "$CERT_DIR/kubernetes/pki/front-proxy-client.csr" \ + -CA "$CERT_DIR/kubernetes/pki/front-proxy-ca.crt" \ + -CAkey "$CERT_DIR/kubernetes/pki/front-proxy-ca.key" \ + -CAcreateserial -days 3650 \ + -out "$CERT_DIR/kubernetes/pki/front-proxy-client.crt" + +echo "==> Generating Ceph keyrings..." + +# Ceph uses cephx authentication (symmetric keys) +# These will be generated using ceph-authtool at first-boot of first monitor +# For now, create placeholder directory +touch "$CERT_DIR/ceph/.placeholder" + +echo "==> Generating MQTT password file..." + +# Default admin user (change password post-install!) +# Password: "admin" (hashed) +mosquitto_passwd -c -b "$CERT_DIR/mqtt/passwd" admin admin 2>/dev/null || { + echo "WARNING: mosquitto_passwd not found. Creating empty file." + touch "$CERT_DIR/mqtt/passwd" +} + +echo "==> Build-time certificates generated in $CERT_DIR" +ls -lhR "$CERT_DIR" +``` + +### 2. First Boot (Master): Generate Node-Specific Certs + +**Script**: `tools/generate-node-certs.sh` + +```bash +#!/bin/bash +# Generates node-specific certificates on first boot +# Run by cluster-detect.service after node identity established + +set -euo pipefail + +CONFIG_DIR="${CONFIG_DIR:-/etc/cluster-config}" +NODE_NAME=$(cat "$CONFIG_DIR/node-identity") +NODE_CONFIG="$CONFIG_DIR/nodes/$NODE_NAME.yaml" + +# Extract node IP from current-node.yaml +NODE_IP=$(yq eval '.node.ip' "$CONFIG_DIR/current-node.yaml") +CLUSTER_NAME=$(yq eval '.cluster.name' "$CONFIG_DIR/cluster.yaml") +SERVICE_CIDR=$(yq eval '.cluster.networking.service_cidr' "$CONFIG_DIR/cluster.yaml") + +# Kubernetes API VIP (first IP in service CIDR) +KUBERNETES_SVC_IP=$(echo "$SERVICE_CIDR" | awk -F'[./]' '{print $1"."$2"."$3".1"}') + +PKI_DIR="/etc/kubernetes/pki" +ETCD_PKI_DIR="$PKI_DIR/etcd" + +echo "==> Generating certificates for node: $NODE_NAME ($NODE_IP)" + +# Check if node is a master +ROLES=$(yq eval '.node.roles[]' "$NODE_CONFIG") +IS_MASTER=false +[[ "$ROLES" =~ "master" || "$ROLES" =~ "control-plane" ]] && IS_MASTER=true + +if [[ "$IS_MASTER" == "true" ]]; then + echo "==> Generating API server certificate..." + + # Create OpenSSL config for SAN + cat > /tmp/apiserver-csr.conf <<EOF +[req] +req_extensions = v3_req +distinguished_name = req_distinguished_name +[req_distinguished_name] +[v3_req] +basicConstraints = CA:FALSE +keyUsage = nonRepudiation, digitalSignature, keyEncipherment +extendedKeyUsage = serverAuth +subjectAltName = @alt_names +[alt_names] +DNS.1 = kubernetes +DNS.2 = kubernetes.default +DNS.3 = kubernetes.default.svc +DNS.4 = kubernetes.default.svc.cluster.local +DNS.5 = $NODE_NAME +DNS.6 = $CLUSTER_NAME +IP.1 = $KUBERNETES_SVC_IP +IP.2 = $NODE_IP +IP.3 = 127.0.0.1 +EOF + + openssl genrsa -out "$PKI_DIR/apiserver.key" 2048 + openssl req -new -key "$PKI_DIR/apiserver.key" \ + -subj "/CN=kube-apiserver" \ + -config /tmp/apiserver-csr.conf \ + -out "$PKI_DIR/apiserver.csr" + openssl x509 -req -in "$PKI_DIR/apiserver.csr" \ + -CA "$PKI_DIR/ca.crt" -CAkey "$PKI_DIR/ca.key" \ + -CAcreateserial -days 3650 \ + -extensions v3_req -extfile /tmp/apiserver-csr.conf \ + -out "$PKI_DIR/apiserver.crt" + + echo "==> Generating API server kubelet client certificate..." + openssl genrsa -out "$PKI_DIR/apiserver-kubelet-client.key" 2048 + openssl req -new -key "$PKI_DIR/apiserver-kubelet-client.key" \ + -subj "/CN=kube-apiserver-kubelet-client/O=system:masters" \ + -out "$PKI_DIR/apiserver-kubelet-client.csr" + openssl x509 -req -in "$PKI_DIR/apiserver-kubelet-client.csr" \ + -CA "$PKI_DIR/ca.crt" -CAkey "$PKI_DIR/ca.key" \ + -CAcreateserial -days 3650 \ + -out "$PKI_DIR/apiserver-kubelet-client.crt" + + echo "==> Generating etcd server certificate..." + cat > /tmp/etcd-server-csr.conf <<EOF +[req] +req_extensions = v3_req +distinguished_name = req_distinguished_name +[req_distinguished_name] +[v3_req] +basicConstraints = CA:FALSE +keyUsage = nonRepudiation, digitalSignature, keyEncipherment +extendedKeyUsage = serverAuth, clientAuth +subjectAltName = @alt_names +[alt_names] +DNS.1 = $NODE_NAME +DNS.2 = localhost +IP.1 = $NODE_IP +IP.2 = 127.0.0.1 +EOF + + openssl genrsa -out "$ETCD_PKI_DIR/server.key" 2048 + openssl req -new -key "$ETCD_PKI_DIR/server.key" \ + -subj "/CN=$NODE_NAME" \ + -config /tmp/etcd-server-csr.conf \ + -out "$ETCD_PKI_DIR/server.csr" + openssl x509 -req -in "$ETCD_PKI_DIR/server.csr" \ + -CA "$ETCD_PKI_DIR/ca.crt" -CAkey "$ETCD_PKI_DIR/ca.key" \ + -CAcreateserial -days 3650 \ + -extensions v3_req -extfile /tmp/etcd-server-csr.conf \ + -out "$ETCD_PKI_DIR/server.crt" + + echo "==> Generating etcd peer certificate..." + openssl genrsa -out "$ETCD_PKI_DIR/peer.key" 2048 + openssl req -new -key "$ETCD_PKI_DIR/peer.key" \ + -subj "/CN=$NODE_NAME" \ + -config /tmp/etcd-server-csr.conf \ + -out "$ETCD_PKI_DIR/peer.csr" + openssl x509 -req -in "$ETCD_PKI_DIR/peer.csr" \ + -CA "$ETCD_PKI_DIR/ca.crt" -CAkey "$ETCD_PKI_DIR/ca.key" \ + -CAcreateserial -days 3650 \ + -extensions v3_req -extfile /tmp/etcd-server-csr.conf \ + -out "$ETCD_PKI_DIR/peer.crt" + + echo "==> Generating etcd healthcheck client certificate..." + openssl genrsa -out "$ETCD_PKI_DIR/healthcheck-client.key" 2048 + openssl req -new -key "$ETCD_PKI_DIR/healthcheck-client.key" \ + -subj "/CN=kube-etcd-healthcheck-client/O=system:masters" \ + -out "$ETCD_PKI_DIR/healthcheck-client.csr" + openssl x509 -req -in "$ETCD_PKI_DIR/healthcheck-client.csr" \ + -CA "$ETCD_PKI_DIR/ca.crt" -CAkey "$ETCD_PKI_DIR/ca.key" \ + -CAcreateserial -days 3650 \ + -out "$ETCD_PKI_DIR/healthcheck-client.crt" + + # Set permissions + chmod 600 "$PKI_DIR"/*.key "$ETCD_PKI_DIR"/*.key +fi + +echo "==> Generating kubelet certificate..." +mkdir -p /var/lib/kubelet/pki + +cat > /tmp/kubelet-csr.conf <<EOF +[req] +req_extensions = v3_req +distinguished_name = req_distinguished_name +[req_distinguished_name] +[v3_req] +basicConstraints = CA:FALSE +keyUsage = nonRepudiation, digitalSignature, keyEncipherment +extendedKeyUsage = serverAuth +subjectAltName = @alt_names +[alt_names] +DNS.1 = $NODE_NAME +IP.1 = $NODE_IP +EOF + +openssl genrsa -out /var/lib/kubelet/pki/kubelet.key 2048 +openssl req -new -key /var/lib/kubelet/pki/kubelet.key \ + -subj "/CN=system:node:$NODE_NAME/O=system:nodes" \ + -config /tmp/kubelet-csr.conf \ + -out /var/lib/kubelet/pki/kubelet.csr +openssl x509 -req -in /var/lib/kubelet/pki/kubelet.csr \ + -CA "$PKI_DIR/ca.crt" -CAkey "$PKI_DIR/ca.key" \ + -CAcreateserial -days 3650 \ + -extensions v3_req -extfile /tmp/kubelet-csr.conf \ + -out /var/lib/kubelet/pki/kubelet.crt + +chmod 600 /var/lib/kubelet/pki/kubelet.key + +echo "==> Node certificates generated successfully" +``` + +### 3. First Boot (First Master): Generate Ceph Keys + +**Script**: `tools/generate-ceph-keys.sh` + +```bash +#!/bin/bash +# Generates Ceph authentication keys on first monitor node +# Only run on the FIRST ceph-mon node + +set -euo pipefail + +CEPH_DIR="/etc/ceph" +mkdir -p "$CEPH_DIR" + +echo "==> Generating Ceph cluster FSID..." +FSID=$(uuidgen) +echo "$FSID" > "$CEPH_DIR/cluster.fsid" + +echo "==> Generating Ceph monitor keyring..." +ceph-authtool --create-keyring "$CEPH_DIR/ceph.mon.keyring" \ + --gen-key -n mon. --cap mon 'allow *' + +echo "==> Generating Ceph admin keyring..." +ceph-authtool --create-keyring "$CEPH_DIR/ceph.client.admin.keyring" \ + --gen-key -n client.admin \ + --cap mon 'allow *' \ + --cap osd 'allow *' \ + --cap mds 'allow *' \ + --cap mgr 'allow *' + +echo "==> Generating OSD bootstrap keyring..." +ceph-authtool --create-keyring "$CEPH_DIR/ceph.bootstrap-osd.keyring" \ + --gen-key -n client.bootstrap-osd \ + --cap mon 'allow profile bootstrap-osd' \ + --cap mgr 'allow r' + +echo "==> Generating MDS bootstrap keyring..." +ceph-authtool --create-keyring "$CEPH_DIR/ceph.bootstrap-mds.keyring" \ + --gen-key -n client.bootstrap-mds \ + --cap mon 'allow profile bootstrap-mds' \ + --cap mgr 'allow r' + +# Import admin keyring into mon keyring +ceph-authtool "$CEPH_DIR/ceph.mon.keyring" --import-keyring "$CEPH_DIR/ceph.client.admin.keyring" + +chmod 600 "$CEPH_DIR"/*.keyring + +echo "==> Ceph keys generated. FSID: $FSID" +echo "==> These keys must be distributed to all ceph nodes!" +``` + +## Integration with Boot Flow + +### Modified Boot Sequence + +``` +1. cluster-detect.service + ├─> cluster-detect.sh (identify node) + └─> generate-node-certs.sh (NEW - generate node-specific certs) + +2. generate-environment-files.sh + (reads certs, creates .env files with cert paths) + +3. cluster-activate-roles.sh + (enables targets based on roles) + +4. Services start + (now have valid certificates) +``` + +### Updated cluster-detect.service + +```ini +[Unit] +Description=Cluster Node Detection and Certificate Generation +DefaultDependencies=no +Before=network-pre.target +Wants=network-pre.target + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/local/bin/cluster-detect.sh +ExecStart=/usr/local/bin/generate-node-certs.sh +ExecStart=/usr/local/bin/generate-environment-files.sh +ExecStart=/usr/local/bin/cluster-activate-roles.sh + +[Install] +WantedBy=multi-user.target +``` + +## Certificate Distribution for Multi-Master + +**Problem**: Masters 2+ need the CA private keys to sign their own certs. + +**Solutions** (pick one): + +### Option A: Embed CA keys in ISO (simple, less secure) +- All CA private keys baked into ISO +- Each master generates its own node certs using shared CA +- **Remove CA keys after cluster init** via post-bootstrap script + +### Option B: Fetch from first master (secure, complex) +- First master generates CA, serves via HTTPS +- Additional masters fetch CA bundle during bootstrap +- Requires authentication (shared secret in cluster.yaml) + +### Option C: External secret store (production) +- Build-time: Upload CA to Vault/AWS Secrets Manager +- Boot-time: Fetch CA using node identity proof +- Requires external dependency + +**Recommendation**: Start with Option A, migrate to Option C for production. + +## Security Hardening Checklist + +- [ ] Set file permissions: 600 for `.key`, 644 for `.crt` +- [ ] Remove CA private keys after all masters bootstrapped +- [ ] Rotate service account keys annually +- [ ] Enable Kubernetes certificate rotation (kubelet `--rotate-certificates`) +- [ ] Monitor certificate expiry (90 days before expiration) +- [ ] Store root CA offline (not in ISO for production) +- [ ] Use hardware security module (HSM) for CA keys (production) +- [ ] Implement certificate revocation list (CRL) + +## Validation Script + +**Script**: `tools/validate-certs.sh` + +```bash +#!/bin/bash +# Validates all certificates before service startup + +set -euo pipefail + +PKI_DIR="/etc/kubernetes/pki" +ERRORS=0 + +check_cert() { + local cert=$1 + local key=$2 + + if [[ ! -f "$cert" ]]; then + echo "ERROR: Certificate not found: $cert" + ((ERRORS++)) + return + fi + + if [[ ! -f "$key" ]]; then + echo "ERROR: Private key not found: $key" + ((ERRORS++)) + return + fi + + # Check expiry + if ! openssl x509 -in "$cert" -noout -checkend 86400; then + echo "WARNING: Certificate expires within 24 hours: $cert" + fi + + # Verify key matches cert + cert_modulus=$(openssl x509 -in "$cert" -noout -modulus) + key_modulus=$(openssl rsa -in "$key" -noout -modulus) + + if [[ "$cert_modulus" != "$key_modulus" ]]; then + echo "ERROR: Certificate and key mismatch: $cert / $key" + ((ERRORS++)) + fi +} + +echo "==> Validating Kubernetes certificates..." +check_cert "$PKI_DIR/ca.crt" "$PKI_DIR/ca.key" +check_cert "$PKI_DIR/apiserver.crt" "$PKI_DIR/apiserver.key" + +if [[ $ERRORS -gt 0 ]]; then + echo "==> Validation FAILED with $ERRORS errors" + exit 1 +fi + +echo "==> Validation PASSED" +``` + +## Next Steps + +1. Implement `tools/generate-build-certs.sh` +2. Implement `tools/generate-node-certs.sh` +3. Implement `tools/generate-ceph-keys.sh` +4. Update `cluster-detect.service` to call cert generation +5. Update service units to validate certs in `ExecStartPre=` +6. Test certificate generation in VM +7. Document CA key removal procedure for production + +--- + +**P.S.** This gives you a working PKI. It's not production-grade (CA keys in ISO is a compromise), but it unblocks development. You can swap to Vault/external PKI later without changing the service units—they just read from `/etc/kubernetes/pki/`. |