Skip to main content
Servers Are Up
Virtual Datacenter
PricingFree Tools
Sign inBrowse catalog
Servers Are Up
Virtual Datacenter
PricingFree Tools
Sign inBrowse catalog
Servers Are Up

Your apps, automation and data on hardware we own. Build a private stack with predictable capacity and access through your dashboard, API or SSH.

SLA with service creditsOwned hardwareUS-based

Make room for your next idea

Explore the capacity, apps and access options that fit your workload. Check current availability and your total before you buy.

Sign up freeView Pricing

Products

  • Linux VPS
  • Windows VPS
  • Cloud & SaaS Services
  • Cloud Storage (Azure)
  • Virtual Datacenter
  • AI & Agent Hosting
  • Containers
  • Instant Apps
  • Software Platforms
  • Network builder
  • Game Servers
  • Pricing

Resources

  • Free Network Tools
  • 2026 Pricing Benchmark
  • Live SVG Badges
  • Latency & Speed Test
  • Guides & Tutorials
  • Documentation
  • API Reference
  • MCP Server
  • CLI & SDKs
  • OpenAPI & llms.txt
  • Use Cases
  • Free Migration
  • Uptime & Reliability
  • Datacenter
  • System Status
  • Network
  • Provider Comparisons Hub
  • Compare Providers

Company

  • About Us
  • Enterprise
  • Security Architecture
  • Trust Center
  • Blog
  • FAQ
  • Compliance
  • Startup Program
  • Open-Source Sponsorship
  • Press & Media Kit
  • Referral Program
  • Help Center
  • Contact Us
© Servers Are Up
TermsPrivacySLARefund PolicyDPAAcceptable Use
Service information
Back to Engineering BlogContainers & Kubernetes
Servers Are Up Engineering·Published 2026-11-12·10 min read·Technical Review Verified

Cloud Cluster Architecture: Raft Quorum Math, etcd Latency, East-West MTU, and K3s Topology

cloud cluster

Engineering Summary & Direct Answer

Engineer resilient multi-node cloud clusters: Raft (N/2)+1 split-brain math, etcd fsync budgets, WireGuard MTU 1420 clamping, external state, and K3s sizing.

Instant 60s deploy · Browser SSH & WebRDP included
View Container & K3s Plans Browse 105 Instant AppsVirtual Datacenter Pools ($12–$380/mo)
On this page (6 sections)
  1. Raft Quorum Math: Why Two Nodes Have Worse Availability Than One
  2. etcd Storage Physics and the < 10 ms wal_fsync Budget
  3. East-West CNI Overlays and the 1420 Byte MTU Fragmentation Trap
  4. StatefulSets vs. Externalized Managed State
  5. Sizing Managed K3s and Pooled Virtual Datacenters on Servers Are Up
  6. Frequently Asked Questions
  7. Why is a 2-node cluster less reliable than a single Linux server?
  8. What K3s version runs on Servers Are Up Managed Kubernetes plans?
  9. Should I stretch etcd voting nodes across geographically distant clouds?
  10. Why do pods briefly return 502 Bad Gateway during a RollingUpdate even with ReadinessProbe configured?

The most counterintuitive failure in cloud infrastructure happens when a team upgrades from one virtual server to a "two-node cluster" for redundancy—and inadvertently doubles its outage probability. Moving from a single Linux KVM instance to a distributed container cluster trades single-machine hardware failure for consensus quorum math, write-ahead log (WAL) fsync budgets, east-west overlay MTU fragmentation, and network-partition split-brain edge cases. Architected rigorously around odd-numbered Raft quorums, bounded etcd storage latency, clamped 1420-byte WireGuard overlays, and externalized stateful storage, a CNCF-certified K3s cluster delivers automated pod self-healing and zero-downtime rolling deployments. Assembled without those invariants, a minor disk I/O burst triggers leader election thrashing across the entire control plane.

Raft Quorum Math: Why Two Nodes Have Worse Availability Than One

Every Kubernetes control plane relies on a strongly consistent key-value store—etcd in multi-master HA topologies—governed by the Raft consensus protocol. When a network link partitions a cluster into two isolated halves, both halves cannot accept writes simultaneously without corrupting cluster state (split-brain). Raft prevents divergence by requiring a strict majority quorum $Q$ of voting control-plane members to persist each log entry before committing:

$$Q = \left\lfloor \frac{N}{2} \right\rfloor + 1, \qquad F_{\text{max}} = N - Q = \left\lfloor \frac{N - 1}{2} \right\rfloor$$

Evaluating maximum tolerable node failures $F_{\text{max}}$ across voting cluster sizes $N$ reveals why even-numbered control planes are an operational trap:

  • $N = 1$ (Single-Node Cluster): $Q = 1$, $F_{\text{max}} = 0$. If an individual VM experiences independent unscheduled unavailability with probability $p$, cluster failure probability equals $p$. Ideal for development, CI preview namespaces, and staging where simplicity and cost dominate.
  • $N = 2$ (Two-Node Cluster — Anti-Pattern): $Q = \lfloor 2/2 \rfloor + 1 = 2$, $F_{\text{max}} = 0$. Losing either VM drops the survivor to $1$ vote—short of the required $2$-vote quorum—forcing etcd read-only and halting all kube-apiserver mutations, pod evictions, and endpoint slice updates. Unscheduled failure probability rises to $1 - (1 - p)^2 = 2p - p^2 \approx 2p$, cutting reliability in half while paying for twice the compute.
  • $N = 3$ (Three-Node HA Control Plane): $Q = 2$, $F_{\text{max}} = 1$. Any single control-plane VM can reboot for a kernel patch or fail unexpectedly while the remaining 2 voters preserve full API read/write availability ($P_{\text{fail}} = 3p^2 - 2p^3 \ll p$).
  • $N = 5$ (Five-Node HA Control Plane): $Q = 3$, $F_{\text{max}} = 2$. Essential when strict change-management policies require surviving an unexpected hardware fault during a planned rolling OS upgrade: with 1 node intentionally drained for maintenance, 4 active voters remain, still tolerating 1 surprise crash before dropping below $Q = 3$.

Notice that $N = 4$ ($Q = 3, F_{\text{max}} = 1$) tolerates only 1 failure—no better than $N = 3$—while requiring an extra synchronous Raft acknowledgement on every write. Scale stateless worker nodes to any count your CPU/RAM demand requires, but keep voting etcd control-plane members strictly at 1, 3, or 5.

etcd Storage Physics and the < 10 ms wal_fsync Budget

Distributed consensus turns storage latency directly into network availability. Before an etcd voter replies to the Raft leader's AppendEntries RPC, it must flush the log entry to persistent storage via fdatasync(). By default, etcd sends leader heartbeats every 100 ms and triggers a follower election timeout at 1000 ms.

If co-located batch jobs, container image layer unpacks, or heavy log indexing saturate disk queues and push etcd_disk_wal_fsync_duration_seconds p99 above 10 ms (or etcd_disk_backend_commit_duration_seconds p99 above 25 ms), Raft followers begin missing heartbeats, increment the Raft term, strip leadership from the healthy leader, pause kube-apiserver watch caches, and cascade 504 timeouts through ingress controllers.

Protect your control plane with two concrete engineering habits:

  1. Isolate etcd I/O from noisy workloads—keep heavy CI runners, Elasticsearch/Prometheus compaction, and unthrottled log shippers off control-plane nodes via NoSchedule taints or dedicated worker VMs.
  2. Instrument and audit live etcd WAL sync histograms and Raft term stability directly:
# 1. Inspect embedded K3s etcd member health, DB size, Raft index, and leader term
sudo ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
  --cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
  --key=/var/lib/rancher/k3s/server/tls/etcd/server-client.key \
  endpoint status --write-out=table

# 2. Verify p99 wal_fsync_duration_seconds stays below 0.010s (10 ms)
curl -sk https://127.0.0.1:2381/metrics \
  | awk '/^etcd_disk_wal_fsync_duration_seconds_bucket\{le="(0\.008|0\.016|\+Inf)"\}/ || /^etcd_server_leader_changes_seen_total/'

# 3. Trigger an on-demand compacted snapshot before control-plane upgrades
sudo k3s etcd-snapshot save --name pre-upgrade-$(date -u +%Y%m%dT%H%M%SZ)

When deploying self-managed HA K3s (v1.36.3+k3s1) across three or five Linux KVM servers, configure /etc/rancher/k3s/config.yaml with explicit kube-reserved and system-reserved cgroup slices so pod memory bursts on converged nodes can never starve k3s server or etcd, and tighten Kubernetes' default 300s node-failure pod eviction delay down to 30s:

# /etc/rancher/k3s/config.yaml — HA Control-Plane Server Node
cluster-init: true
tls-san:
  - "k8s-api.internal.example.com"
  - "10.20.0.10"
flannel-backend: "wireguard-native"
etcd-expose-metrics: true
etcd-snapshot-schedule-cron: "0 */6 * * *"
etcd-snapshot-retention: 14
kube-apiserver-arg:
  - "default-not-ready-toleration-seconds=30"
  - "default-unreachable-toleration-seconds=30"
kubelet-arg:
  - "max-pods=110"
  - "system-reserved=cpu=300m,memory=768Mi"
  - "kube-reserved=cpu=300m,memory=768Mi"
  - "eviction-hard=memory.available<256Mi,nodefs.available<10%"

East-West CNI Overlays and the 1420 Byte MTU Fragmentation Trap

Multi-node clusters connect pod subnets across hosts using an overlay Container Network Interface (CNI)—commonly VXLAN (UDP/8472, 50-byte header on IPv4) or K3s's wireguard-native backend (UDP/51820, 60-byte header on IPv4 and 80-byte header on IPv6). On standard 1500-byte virtual Ethernet NICs, encapsulating dual-stack pod packets inside WireGuard leaves a maximum inner payload MTU of exactly 1420 bytes (1500 - 80).

MTU mismatches produce the single most maddening symptom in Kubernetes networking: curl http://service/healthz (90 bytes) succeeds instantaneously every time, whereas POST bodies, TLS 1.3 certificate chains, or gRPC responses larger than 1420 bytes hang until ETIMEDOUT. This occurs whenever nested Docker-in-Docker builders, custom sidecar proxies, or cross-cloud VPN bridges assume a 1500 MTU with the IP DF (Don't Fragment) bit set while an intermediate firewall drops ICMP Fragmentation Needed packets.

Always verify inner CNI link MTUs with ip -d link, test unfragmented packet ceilings using ping -M do, and enforce nftables TCP Maximum Segment Size (MSS) clamping on cluster gateway interfaces:

# 1. Confirm WireGuard CNI interface reports mtu 1420
ip -d link show flannel-wg

# 2. Test maximum unfragmented ICMP payload between worker Pod IPs
#    1420 inner MTU - 20 byte IPv4 header - 8 byte ICMP header = 1392 bytes
ping -M do -s 1392 -c 4 10.42.2.18

# 3. Clamp TCP SYN MSS dynamically to route PMTU via nftables
sudo nft add table inet mangle
sudo nft 'add chain inet mangle forward { type filter hook forward priority mangle; policy accept; }'
sudo nft 'add rule inet mangle forward tcp flags syn / syn,rst tcp option maxseg size set rt mtu'

StatefulSets vs. Externalized Managed State

Kubernetes StatefulSet controllers provide stable network identities (db-0, db-1) and sticky PersistentVolumeClaim bindings, tempting teams to run distributed block storage (Rook-Ceph or Longhorn) plus self-managed PostgreSQL failover operators inside a small 3-node cluster. On compact clusters, hyperconverged storage replicates every single 4 KiB database page write three times across east-west WireGuard tunnels, amplifying disk write IOPS and network bandwidth by $3\times$ and competing directly with etcd wal_fsync latency.

Keeping cluster worker nodes stateless and Cattle-disposable dramatically simplifies upgrades and disaster recovery:

  • Relational, JSON Document & Vector Workloads: Connect pods over TLS to the SAU Multi-Model Cloud Database (100K RU/mo free developer tier, SQL + JSON + key-value + pgvector, tunable STRONG / SESSION / EVENTUAL consistency).
  • Uploads, Registry Layers & Backups: Offload blobs, Helm artifacts, and etcd snapshots to Managed Cloud Storage hosted via Microsoft Azure (5 GiB free tier, 250 GiB at $9/mo up to 25 TiB at $399/mo, AES-256-GCM + Azure storage encryption).

With externalized state, worker nodes can be cordoned, drained, and replaced cleanly whenever your Deployment declares anti-affinity topologySpreadConstraints and a PodDisruptionBudget:

# k8s/api-deployment-pdb.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: billing-gateway
  namespace: production
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: billing-gateway
  template:
    metadata:
      labels:
        app: billing-gateway
    spec:
      terminationGracePeriodSeconds: 35
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: billing-gateway
      containers:
        - name: gateway
          image: registry.example.com/billing-gateway:2026.11.2
          ports:
            - containerPort: 8080
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 5"]
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              memory: "768Mi"
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8080
            periodSeconds: 5
            failureThreshold: 2
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: billing-gateway-pdb
  namespace: production
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: billing-gateway

Notice the preStop: sleep 5 hook: when a pod terminates, kube-proxy iptables/nftables rules and ingress controllers update endpoint lists asynchronously alongside SIGTERM. Sleeping 5 seconds while readiness flips false allows in-flight connections to drain before the container closes its listening socket, eliminating deploy-time 502 Bad Gateway blips.

Sizing Managed K3s and Pooled Virtual Datacenters on Servers Are Up

Servers Are Up provides two distinct cluster paths: turnkey Managed Kubernetes (v1.36.3+k3s1) with a cluster-admin kubeconfig and zero control-plane management surcharge, or Pooled Virtual Datacenters where you divide raw KVM vCPU, RAM, and persistent storage across custom Linux VMs, load balancers, and database replicas:

Product Tier Cluster / Pool Specification Monthly Price Consensus & Workload Architecture
K8s Dev Cluster 1-node K3s (v1.36.3+k3s1), 2 vCPU, 8 GB RAM, 100 GB persistent storage $18/mo Single-node ($N=1$); Kubernetes manifest parity for CI, Helm testing, and staging
K8s Standard Cluster 3-node K3s (1 control plane + 2 workers), 6 vCPU, 24 GB RAM, 300 GB persistent storage $42/mo Multi-worker pod scheduling, rolling deploys, and worker failover
K8s Production Cluster 5-node HA K3s (3 HA control-plane/etcd nodes + 2 workers), 20 vCPU, 80 GB RAM, 800 GB persistent storage $140/mo Full 3-voter Raft HA quorum ($Q=2, F_{\text{max}}=1$) + redundant worker pool
VDC Hobby Lab Pooled 4 vCPU, 8 GB RAM, 100 GB persistent storage (up to 5 instances) $12/mo Compact multi-VM network, proxy, or WireGuard lab pool
VDC Developer Studio Pooled 8 vCPU, 16 GB RAM, 250 GB persistent storage (up to 15 instances) $24/mo Multi-service staging + internal tooling pool ($1.50/GB RAM)
VDC Team Workspace Pooled 24 vCPU, 64 GB RAM, 1 TB persistent storage (up to 40 instances) $95/mo Custom 3- or 5-node K3s/Nomad/Postgres HA clusters with isolated subnets
VDC Production Cluster Pooled 64 vCPU, 192 GB RAM, 4 TB persistent storage (up to 100 instances) $285/mo High-density multi-environment production fleet
VDC Enterprise Pooled 96 vCPU, 256 GB RAM, 6 TB persistent storage (up to 200 instances) $380/mo Large-scale platform engineering and multi-tenant isolation

All clusters integrate directly with the browser SSH terminal, SAU Network Builder ($5/mo per dedicated public IPv4 only on ingress gateways), and our /api/mcp/v1 automation plane. If your stack runs fewer than ten containers without needing multi-node pod failover, compare single-host tradeoffs in our Docker container hosting guide.

Frequently Asked Questions

Why is a 2-node cluster less reliable than a single Linux server?

Consensus algorithms like Raft require a strict majority ($\lfloor N/2 \rfloor + 1$) of voting nodes to commit state safely without split-brain. In a 2-node cluster, $\lfloor 2/2 \rfloor + 1 = 2$, meaning both nodes must stay online for the control plane to function. Any failure on Node A or Node B takes down cluster writes, roughly doubling failure frequency compared to a single VM.

What K3s version runs on Servers Are Up Managed Kubernetes plans?

Managed Kubernetes clusters run CNCF-certified K3s v1.36.3+k3s1 and deliver a standard cluster-admin kubeconfig, making kubectl, Helm, Argo CD, Flux, and standard Kubernetes Operators work identically to upstream Kubernetes without bloated control-plane memory overhead.

Should I stretch etcd voting nodes across geographically distant clouds?

Avoid stretching synchronous Raft etcd voters across high-latency WAN links unless your workload tolerates elevated write latency on every Kubernetes API mutation, because every Raft commit incurs at least one cross-node RTT plus wal_fsync. Keep your 3 or 5 etcd voters in one low-latency datacenter fabric, and federate multi-cloud workloads at the ingress or service layer.

Why do pods briefly return 502 Bad Gateway during a RollingUpdate even with ReadinessProbe configured?

When Kubernetes deletes an old pod, SIGTERM is sent to the container in parallel with EndpointSlice updates propagating to kube-proxy and your ingress controller. Adding a preStop lifecycle hook (sleep 5) while your app stops reporting ready—before closing its TCP listener—gives load balancers time to stop routing new requests to the terminating pod.

Isolated OpenStack KVM Containers & SHA-Pinned K3s

Ship Containers Without Per-Seat or Per-Minute Billing Surprises

Managed Docker hosts from $6/mo, 105 single-tenant Instant Apps, and managed K3s clusters from $18/mo ($42/mo for 3-node HA) — plus 69 free 1-click installers on every Linux VPS.

View Container & K3s Plans Browse 105 Instant Apps

Explore With AI & Cite This Page

Open a grounded prompt in your AI assistant or copy a clean Markdown, HTML, or APA reference link.

Ask ChatGPT Ask Perplexity Ask Claude
Share:HN·Reddit·X·LinkedIn
Newer Article

AI Infrastructure Management: Deterministic Control Planes, MCP Tooling, and Stateful Agent Fleets

Older Article

Automated Server Provisioning: Declarative cloud-init, OpenTofu Convergence, and Zero-Downtime Cutovers

Related Diagnostic Tools & Engineering Guides

Validate protocols and subnets directly in your browser or continue reading related architecture notes.

Free Browser Tool

SSH Keygen Generator & Cloud-Init YAML Builder

Launch utility
Free Browser Tool

Cloud Cost & Infrastructure ROI Calculator

Launch utility
Free Browser Tool

Free DNS Zone File, CAA & DMARC Record Generator

Launch utility

Managed Kubernetes: K3s clusters set up for you

Managed Kubernetes is on sale: three SHA-pinned K3s v1.36.3+k3s1 cluster tiers provisioned on dedicated OpenStack virtual machines with persistent block storage and downloadable kubeconfig.

High-Concurrency CMS Hosting: PHP-FPM Memory Math, Nginx Microcaching, Headless Node, and Object Offload

Scale Ghost, WordPress, Strapi, Directus, and Payload CMS: PHP-FPM worker sizing, Nginx stale-while-revalidate microcaching, Redis, and Azure media offload.

Coolify v4 Architecture & Production Review (2026): Traefik Routing, Build-Server OOM Traps, and VPS Sizing

An engineering review of Coolify v4+ in production: dynamic Traefik/Caddy container labels, why Next.js Nixpacks builds OOM-kill Postgres on 2 GB nodes, and Coolify vs Managed Docker ($6-$18/mo).

Docker Container Hosting in Production: Daemon Hardening, UFW Port Traps, Compose Limits, and Security

Run Docker containers safely in production: compose.yaml cgroup limits, daemon.json live-restore, fixing the UFW 0.0.0.0 port-binding bypass, and non-root builds.

On this page

  1. Raft Quorum Math: Why Two Nodes Have Worse Availability Than One
  2. etcd Storage Physics and the < 10 ms wal_fsync Budget
  3. East-West CNI Overlays and the 1420 Byte MTU Fragmentation Trap
  4. StatefulSets vs. Externalized Managed State
  5. Sizing Managed K3s and Pooled Virtual Datacenters on Servers Are Up
  6. Frequently Asked Questions
  7. Why is a 2-node cluster less reliable than a single Linux server?
  8. What K3s version runs on Servers Are Up Managed Kubernetes plans?
  9. Should I stretch etcd voting nodes across geographically distant clouds?
  10. Why do pods briefly return 502 Bad Gateway during a RollingUpdate even with ReadinessProbe configured?
Self-Serve Infrastructure

Ship Containers Without Per-Seat or Per-Minute Billing Surprises

Managed Docker hosts from $6/mo, 105 single-tenant Instant Apps, and managed K3s clusters from $18/mo ($42/mo for 3-node HA) — plus 69 free 1-click installers on every Linux VPS.

View Container & K3s Plans Browse 105 Instant Apps