Cloud Cluster Architecture: Raft Quorum Math, etcd Latency, East-West MTU, and K3s Topology
On this page (6 sections)
- Raft Quorum Math: Why Two Nodes Have Worse Availability Than One
- etcd Storage Physics and the < 10 ms wal_fsync Budget
- East-West CNI Overlays and the 1420 Byte MTU Fragmentation Trap
- StatefulSets vs. Externalized Managed State
- Sizing Managed K3s and Pooled Virtual Datacenters on Servers Are Up
- Frequently Asked Questions
- Why is a 2-node cluster less reliable than a single Linux server?
- What K3s version runs on Servers Are Up Managed Kubernetes plans?
- Should I stretch etcd voting nodes across geographically distant clouds?
- Why do pods briefly return 502 Bad Gateway during a RollingUpdate even with ReadinessProbe configured?
The most counterintuitive failure in cloud infrastructure happens when a team upgrades from one virtual server to a "two-node cluster" for redundancy—and inadvertently doubles its outage probability. Moving from a single Linux KVM instance to a distributed container cluster trades single-machine hardware failure for consensus quorum math, write-ahead log (WAL) fsync budgets, east-west overlay MTU fragmentation, and network-partition split-brain edge cases. Architected rigorously around odd-numbered Raft quorums, bounded etcd storage latency, clamped 1420-byte WireGuard overlays, and externalized stateful storage, a CNCF-certified K3s cluster delivers automated pod self-healing and zero-downtime rolling deployments. Assembled without those invariants, a minor disk I/O burst triggers leader election thrashing across the entire control plane.
Raft Quorum Math: Why Two Nodes Have Worse Availability Than One
Every Kubernetes control plane relies on a strongly consistent key-value store—etcd in multi-master HA topologies—governed by the Raft consensus protocol. When a network link partitions a cluster into two isolated halves, both halves cannot accept writes simultaneously without corrupting cluster state (split-brain). Raft prevents divergence by requiring a strict majority quorum $Q$ of voting control-plane members to persist each log entry before committing:
$$Q = \left\lfloor \frac{N}{2} \right\rfloor + 1, \qquad F_{\text{max}} = N - Q = \left\lfloor \frac{N - 1}{2} \right\rfloor$$
Evaluating maximum tolerable node failures $F_{\text{max}}$ across voting cluster sizes $N$ reveals why even-numbered control planes are an operational trap:
- $N = 1$ (Single-Node Cluster): $Q = 1$, $F_{\text{max}} = 0$. If an individual VM experiences independent unscheduled unavailability with probability $p$, cluster failure probability equals $p$. Ideal for development, CI preview namespaces, and staging where simplicity and cost dominate.
- $N = 2$ (Two-Node Cluster — Anti-Pattern): $Q = \lfloor 2/2 \rfloor + 1 = 2$, $F_{\text{max}} = 0$. Losing either VM drops the survivor to $1$ vote—short of the required $2$-vote quorum—forcing
etcdread-only and halting allkube-apiservermutations, pod evictions, and endpoint slice updates. Unscheduled failure probability rises to $1 - (1 - p)^2 = 2p - p^2 \approx 2p$, cutting reliability in half while paying for twice the compute. - $N = 3$ (Three-Node HA Control Plane): $Q = 2$, $F_{\text{max}} = 1$. Any single control-plane VM can reboot for a kernel patch or fail unexpectedly while the remaining 2 voters preserve full API read/write availability ($P_{\text{fail}} = 3p^2 - 2p^3 \ll p$).
- $N = 5$ (Five-Node HA Control Plane): $Q = 3$, $F_{\text{max}} = 2$. Essential when strict change-management policies require surviving an unexpected hardware fault during a planned rolling OS upgrade: with 1 node intentionally drained for maintenance, 4 active voters remain, still tolerating 1 surprise crash before dropping below $Q = 3$.
Notice that $N = 4$ ($Q = 3, F_{\text{max}} = 1$) tolerates only 1 failure—no better than $N = 3$—while requiring an extra synchronous Raft acknowledgement on every write. Scale stateless worker nodes to any count your CPU/RAM demand requires, but keep voting etcd control-plane members strictly at 1, 3, or 5.
etcd Storage Physics and the < 10 ms wal_fsync Budget
Distributed consensus turns storage latency directly into network availability. Before an etcd voter replies to the Raft leader's AppendEntries RPC, it must flush the log entry to persistent storage via fdatasync(). By default, etcd sends leader heartbeats every 100 ms and triggers a follower election timeout at 1000 ms.
If co-located batch jobs, container image layer unpacks, or heavy log indexing saturate disk queues and push etcd_disk_wal_fsync_duration_seconds p99 above 10 ms (or etcd_disk_backend_commit_duration_seconds p99 above 25 ms), Raft followers begin missing heartbeats, increment the Raft term, strip leadership from the healthy leader, pause kube-apiserver watch caches, and cascade 504 timeouts through ingress controllers.
Protect your control plane with two concrete engineering habits:
- Isolate
etcdI/O from noisy workloads—keep heavy CI runners, Elasticsearch/Prometheus compaction, and unthrottled log shippers off control-plane nodes viaNoScheduletaints or dedicated worker VMs. - Instrument and audit live
etcdWAL sync histograms and Raft term stability directly:
# 1. Inspect embedded K3s etcd member health, DB size, Raft index, and leader term
sudo ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
--cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
--key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \
endpoint status --write-out=table
# 2. Verify p99 wal_fsync_duration_seconds stays below 0.010s (10 ms)
curl -sk https://127.0.0.1:2381/metrics \
| awk '/^etcd_disk_wal_fsync_duration_seconds_bucket\{le="(0\.008|0\.016|\+Inf)"\}/ || /^etcd_server_leader_changes_seen_total/'
# 3. Trigger an on-demand compacted snapshot before control-plane upgrades
sudo k3s etcd-snapshot save --name pre-upgrade-$(date -u +%Y%m%dT%H%M%SZ)
When deploying self-managed HA K3s (v1.36.3+k3s1) across three or five Linux KVM servers, configure /etc/rancher/k3s/config.yaml with explicit kube-reserved and system-reserved cgroup slices so pod memory bursts on converged nodes can never starve k3s server or etcd, and tighten Kubernetes' default 300s node-failure pod eviction delay down to 30s:
# /etc/rancher/k3s/config.yaml — HA Control-Plane Server Node
cluster-init: true
tls-san:
- "k8s-api.internal.example.com"
- "10.20.0.10"
flannel-backend: "wireguard-native"
etcd-expose-metrics: true
etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 14
kube-apiserver-arg:
- "default-not-ready-toleration-seconds=30"
- "default-unreachable-toleration-seconds=30"
kubelet-arg:
- "max-pods=110"
- "system-reserved=cpu=300m,memory=768Mi"
- "kube-reserved=cpu=300m,memory=768Mi"
- "eviction-hard=memory.available<256Mi,nodefs.available<10%"
East-West CNI Overlays and the 1420 Byte MTU Fragmentation Trap
Multi-node clusters connect pod subnets across hosts using an overlay Container Network Interface (CNI)—commonly VXLAN (UDP/8472, 50-byte header on IPv4) or K3s's wireguard-native backend (UDP/51820, 60-byte header on IPv4 and 80-byte header on IPv6). On standard 1500-byte virtual Ethernet NICs, encapsulating dual-stack pod packets inside WireGuard leaves a maximum inner payload MTU of exactly 1420 bytes (1500 - 80).
MTU mismatches produce the single most maddening symptom in Kubernetes networking: curl http://service/healthz (90 bytes) succeeds instantaneously every time, whereas POST bodies, TLS 1.3 certificate chains, or gRPC responses larger than 1420 bytes hang until ETIMEDOUT. This occurs whenever nested Docker-in-Docker builders, custom sidecar proxies, or cross-cloud VPN bridges assume a 1500 MTU with the IP DF (Don't Fragment) bit set while an intermediate firewall drops ICMP Fragmentation Needed packets.
Always verify inner CNI link MTUs with ip -d link, test unfragmented packet ceilings using ping -M do, and enforce nftables TCP Maximum Segment Size (MSS) clamping on cluster gateway interfaces:
# 1. Confirm WireGuard CNI interface reports mtu 1420
ip -d link show flannel-wg
# 2. Test maximum unfragmented ICMP payload between worker Pod IPs
# 1420 inner MTU - 20 byte IPv4 header - 8 byte ICMP header = 1392 bytes
ping -M do -s 1392 -c 4 10.42.2.18
# 3. Clamp TCP SYN MSS dynamically to route PMTU via nftables
sudo nft add table inet mangle
sudo nft 'add chain inet mangle forward { type filter hook forward priority mangle; policy accept; }'
sudo nft 'add rule inet mangle forward tcp flags syn / syn,rst tcp option maxseg size set rt mtu'
StatefulSets vs. Externalized Managed State
Kubernetes StatefulSet controllers provide stable network identities (db-0, db-1) and sticky PersistentVolumeClaim bindings, tempting teams to run distributed block storage (Rook-Ceph or Longhorn) plus self-managed PostgreSQL failover operators inside a small 3-node cluster. On compact clusters, hyperconverged storage replicates every single 4 KiB database page write three times across east-west WireGuard tunnels, amplifying disk write IOPS and network bandwidth by $3\times$ and competing directly with etcd wal_fsync latency.
Keeping cluster worker nodes stateless and Cattle-disposable dramatically simplifies upgrades and disaster recovery:
- Relational, JSON Document & Vector Workloads: Connect pods over TLS to the SAU Multi-Model Cloud Database (
100K RU/mofree developer tier, SQL + JSON + key-value +pgvector, tunableSTRONG/SESSION/EVENTUALconsistency). - Uploads, Registry Layers & Backups: Offload blobs, Helm artifacts, and
etcdsnapshots to Managed Cloud Storage hosted via Microsoft Azure (5 GiBfree tier,250 GiBat$9/moup to25 TiBat$399/mo,AES-256-GCM+ Azure storage encryption).
With externalized state, worker nodes can be cordoned, drained, and replaced cleanly whenever your Deployment declares anti-affinity topologySpreadConstraints and a PodDisruptionBudget:
# k8s/api-deployment-pdb.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: billing-gateway
namespace: production
spec:
replicas: 4
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
selector:
matchLabels:
app: billing-gateway
template:
metadata:
labels:
app: billing-gateway
spec:
terminationGracePeriodSeconds: 35
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: billing-gateway
containers:
- name: gateway
image: registry.example.com/billing-gateway:2026.11.2
ports:
- containerPort: 8080
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 5"]
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
memory: "768Mi"
readinessProbe:
httpGet:
path: /readyz
port: 8080
periodSeconds: 5
failureThreshold: 2
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: billing-gateway-pdb
namespace: production
spec:
minAvailable: 3
selector:
matchLabels:
app: billing-gateway
Notice the preStop: sleep 5 hook: when a pod terminates, kube-proxy iptables/nftables rules and ingress controllers update endpoint lists asynchronously alongside SIGTERM. Sleeping 5 seconds while readiness flips false allows in-flight connections to drain before the container closes its listening socket, eliminating deploy-time 502 Bad Gateway blips.
Sizing Managed K3s and Pooled Virtual Datacenters on Servers Are Up
Servers Are Up provides two distinct cluster paths: turnkey Managed Kubernetes (v1.36.3+k3s1) with a cluster-admin kubeconfig and zero control-plane management surcharge, or Pooled Virtual Datacenters where you divide raw KVM vCPU, RAM, and persistent storage across custom Linux VMs, load balancers, and database replicas:
| Product Tier | Cluster / Pool Specification | Monthly Price | Consensus & Workload Architecture |
|---|---|---|---|
| K8s Dev Cluster | 1-node K3s (v1.36.3+k3s1), 2 vCPU, 8 GB RAM, 100 GB persistent storage |
$18/mo |
Single-node ($N=1$); Kubernetes manifest parity for CI, Helm testing, and staging |
| K8s Standard Cluster | 3-node K3s (1 control plane + 2 workers), 6 vCPU, 24 GB RAM, 300 GB persistent storage | $42/mo |
Multi-worker pod scheduling, rolling deploys, and worker failover |
| K8s Production Cluster | 5-node HA K3s (3 HA control-plane/etcd nodes + 2 workers), 20 vCPU, 80 GB RAM, 800 GB persistent storage | $140/mo |
Full 3-voter Raft HA quorum ($Q=2, F_{\text{max}}=1$) + redundant worker pool |
| VDC Hobby Lab | Pooled 4 vCPU, 8 GB RAM, 100 GB persistent storage (up to 5 instances) | $12/mo |
Compact multi-VM network, proxy, or WireGuard lab pool |
| VDC Developer Studio | Pooled 8 vCPU, 16 GB RAM, 250 GB persistent storage (up to 15 instances) | $24/mo |
Multi-service staging + internal tooling pool ($1.50/GB RAM) |
| VDC Team Workspace | Pooled 24 vCPU, 64 GB RAM, 1 TB persistent storage (up to 40 instances) | $95/mo |
Custom 3- or 5-node K3s/Nomad/Postgres HA clusters with isolated subnets |
| VDC Production Cluster | Pooled 64 vCPU, 192 GB RAM, 4 TB persistent storage (up to 100 instances) | $285/mo |
High-density multi-environment production fleet |
| VDC Enterprise | Pooled 96 vCPU, 256 GB RAM, 6 TB persistent storage (up to 200 instances) | $380/mo |
Large-scale platform engineering and multi-tenant isolation |
All clusters integrate directly with the browser SSH terminal, SAU Network Builder ($5/mo per dedicated public IPv4 only on ingress gateways), and our /api/mcp/v1 automation plane. If your stack runs fewer than ten containers without needing multi-node pod failover, compare single-host tradeoffs in our Docker container hosting guide.
Frequently Asked Questions
Why is a 2-node cluster less reliable than a single Linux server?
Consensus algorithms like Raft require a strict majority ($\lfloor N/2 \rfloor + 1$) of voting nodes to commit state safely without split-brain. In a 2-node cluster, $\lfloor 2/2 \rfloor + 1 = 2$, meaning both nodes must stay online for the control plane to function. Any failure on Node A or Node B takes down cluster writes, roughly doubling failure frequency compared to a single VM.
What K3s version runs on Servers Are Up Managed Kubernetes plans?
Managed Kubernetes clusters run CNCF-certified K3s v1.36.3+k3s1 and deliver a standard cluster-admin kubeconfig, making kubectl, Helm, Argo CD, Flux, and standard Kubernetes Operators work identically to upstream Kubernetes without bloated control-plane memory overhead.
Should I stretch etcd voting nodes across geographically distant clouds?
Avoid stretching synchronous Raft etcd voters across high-latency WAN links unless your workload tolerates elevated write latency on every Kubernetes API mutation, because every Raft commit incurs at least one cross-node RTT plus wal_fsync. Keep your 3 or 5 etcd voters in one low-latency datacenter fabric, and federate multi-cloud workloads at the ingress or service layer.
Why do pods briefly return 502 Bad Gateway during a RollingUpdate even with ReadinessProbe configured?
When Kubernetes deletes an old pod, SIGTERM is sent to the container in parallel with EndpointSlice updates propagating to kube-proxy and your ingress controller. Adding a preStop lifecycle hook (sleep 5) while your app stops reporting ready—before closing its TCP listener—gives load balancers time to stop routing new requests to the terminating pod.
Explore With AI & Cite This Page
Open a grounded prompt in your AI assistant or copy a clean Markdown, HTML, or APA reference link.
Related Diagnostic Tools & Engineering Guides
Validate protocols and subnets directly in your browser or continue reading related architecture notes.
SSH Keygen Generator & Cloud-Init YAML Builder
Cloud Cost & Infrastructure ROI Calculator
Free DNS Zone File, CAA & DMARC Record Generator
Managed Kubernetes: K3s clusters set up for you
Managed Kubernetes is on sale: three SHA-pinned K3s v1.36.3+k3s1 cluster tiers provisioned on dedicated OpenStack virtual machines with persistent block storage and downloadable kubeconfig.
High-Concurrency CMS Hosting: PHP-FPM Memory Math, Nginx Microcaching, Headless Node, and Object Offload
Scale Ghost, WordPress, Strapi, Directus, and Payload CMS: PHP-FPM worker sizing, Nginx stale-while-revalidate microcaching, Redis, and Azure media offload.
Coolify v4 Architecture & Production Review (2026): Traefik Routing, Build-Server OOM Traps, and VPS Sizing
An engineering review of Coolify v4+ in production: dynamic Traefik/Caddy container labels, why Next.js Nixpacks builds OOM-kill Postgres on 2 GB nodes, and Coolify vs Managed Docker ($6-$18/mo).
Docker Container Hosting in Production: Daemon Hardening, UFW Port Traps, Compose Limits, and Security
Run Docker containers safely in production: compose.yaml cgroup limits, daemon.json live-restore, fixing the UFW 0.0.0.0 port-binding bypass, and non-root builds.