Clicking through web forms and pasting ad-hoc bash snippets into SSH works for your first prototype VM and becomes an operational liability by your fifth. Manual server setup guarantees configuration drift: subtle differences in sysctl parameters, OpenSSH drop-in files, apt package versions, and nftables rules accumulate across staging and production until an routine reboot refuses to bring up networking at 02:00 UTC. Engineering-grade automated server provisioning replaces human folklore with a deterministic, idempotent state machine—covering hypervisor API allocation, first-boot cloud-init convergence, KMS secret hydration, synthetic health gating, and atomic blue/green traffic cutover.
Brittle provisioning scripts treat VM creation as a fire-and-forget HTTP POST followed by sleep 45 && ssh root@.... Under real datacenter load, package mirror latency or cloud-init key generation varies across runs; a fixed sleep either races an unconfigured sshd daemon or leaves half-provisioned zombie VMs billing silently when step four fails.
A production provisioning pipeline enforces five explicit state transitions, each guarded by machine-verifiable assertions and an automatic teardown path:
Requested -> Allocated): Call the compute API with a unique Idempotency-Key header, pinning compute flavor, OS image, VPC network segment, and cloud-init user-data digest so network retries never create duplicate servers.Allocated -> CloudInitConverged): Before external users log in, cloud-init creates unprivileged accounts, injects Ed25519 keys, disables password and root SSH authentication, loads default-deny nftables rules, and writes /var/lib/cloud/instance/boot-finished.CloudInitConverged -> Configured): Ansible playbooks, container pulls, or guest-API file writes place immutable release artifacts and hydrate runtime secrets from KMS into volatile RAM (tmpfs).Configured -> Verified): Local and upstream health probes confirm systemd units are active, sockets bind expected loopback/private addresses, and schema migrations match. Any failure before Verified automatically triggers DELETE /api/v1/instances/{id}.Verified -> Active): Reverse proxies, virtual IPs, or load balancers shift live connections to the verified green node, drain the older blue generation, and retire it cleanly.cloud-initThe golden rule of cloud-init engineering is to push static OS invariants into native declarative modules (users, write_files, apt, packages) rather than stuffing a 300-line imperative shell script into runcmd. Declarative modules run early in the cloud-init boot graph—prior to ssh.service accepting public connections on Ubuntu 24.04 LTS and Debian 12—eliminating the dangerous window where a freshly booted VM exposes default sshd settings to internet scanners.
#cloud-config
# user-data.yaml — Production Ubuntu 24.04 LTS immutable base layer
package_update: true
package_upgrade: true
packages:
- ca-certificates
- curl
- jq
- nftables
- fail2ban
- unattended-upgrades
users:
- default
- name: opsadmin
groups: [sudo, systemd-journal]
shell: /bin/bash
sudo: ["ALL=(ALL) NOPASSWD:ALL"]
lock_passwd: true
ssh_authorized_keys:
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM8p2ProdAutomationDeployKey2026 ci@example.com
write_files:
- path: /etc/ssh/sshd_config.d/90-zero-password.conf
permissions: "0600"
owner: root:root
content: |
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AllowAgentForwarding no
X11Forwarding no
MaxAuthTries 3
AllowUsers opsadmin
- path: /etc/sysctl.d/90-network-hardening.conf
permissions: "0644"
owner: root:root
content: |
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.core.somaxconn = 4096
vm.swappiness = 10
- path: /etc/nftables.conf
permissions: "0644"
owner: root:root
content: |
#!/usr/sbin/nft -f
flush ruleset
table inet filter {
chain input {
type filter hook input priority 0; policy drop;
iif "lo" accept
ct state established,related accept
ct state invalid drop
ip protocol icmp accept
ip6 nexthdr icmpv6 accept
tcp dport { 22, 80, 443 } accept
}
chain forward { type filter hook forward priority 0; policy drop; }
chain output { type filter hook output priority 0; policy accept; }
}
runcmd:
- [ sysctl, --system ]
- [ systemctl, enable, --now, nftables ]
- [ sshd, -t ]
- [ systemctl, reload, ssh ]
- [ systemctl, enable, --now, unattended-upgrades ]
Always validate sshd -t before reloading ssh: if a syntax typo sneaks into a template variable, sshd -t aborts the reload cleanly and logs the exact line error to /var/log/cloud-init-output.log—which you can inspect immediately in the Servers Are Up browser console.
As fleets grow beyond a handful of hosts, engineering teams must balance cold-boot speed against image-pipeline maintenance overhead:
| Provisioning Architecture | Boot-to-Traffic Time | CI Maintenance Cost | Drift Exposure | Ideal Production Use Case |
|---|---|---|---|---|
| Manual Dashboard + SSH | Slow, error-prone | Zero CI setup, high ops toil | Extreme (snowflake drift) | Throwaway debugging sandboxes only |
Stock OS + cloud-init + OCI Containers |
Fast (base OS + docker pull) |
Low (versioned compose.yaml) |
Low when host layer stays minimal | Stateless web apps, APIs, and AI workers on Linux VPS |
| OpenTofu + Ansible Convergence | Moderate (apt + role tasks) |
Low (Git PR review + state lock) | Very low with scheduled --check audits |
Databases, reverse proxies, WireGuard hubs, and hybrid fleets |
| Immutable Golden Images (Blue/Green) | Fastest (pre-baked root disk) | Higher (image rebuild on every CVE) | Near zero (SSH disabled in prod) | Strict compliance workloads and rapid horizontal replacement |
With OpenTofu, encoding create_before_destroy = true on compute definitions ensures that changing user-data.yaml or upgrading Ubuntu LTS versions provisions and verifies the replacement instance before detaching and destroying the prior node:
# infra/main.tf — Blue/Green immutable replacement lifecycle
terraform {
required_version = ">= 1.8.0"
}
variable "release_sha" {
type = string
}
locals {
cloud_init_raw = templatefile("${path.module}/user-data.yaml.tftpl", { release_sha = var.release_sha })
cloud_init_sha256 = sha256(local.cloud_init_raw)
}
resource "terraform_data" "edge_proxy_generation" {
triggers_replace = [
local.cloud_init_sha256,
var.release_sha,
]
input = {
tier = "linux-standard" # 4 vCPU, 8 GB RAM, 160 GB persistent storage ($18/mo)
os_image = "ubuntu-24.04"
userdata_sha = local.cloud_init_sha256
}
lifecycle {
create_before_destroy = true
}
}
When converging stateful configurations on top of provisioned hosts with Ansible, design every task around three rules: never use ansible.builtin.shell without creates: or changed_when:, validate every rendered daemon config via validate: before overwriting /etc, and run nightly ansible-playbook site.yml --check --diff drift audits from CI:
# ansible/roles/caddy_edge/tasks/main.yml
- name: Install Caddy system group and unprivileged user
ansible.builtin.user:
name: caddy
system: true
shell: /usr/sbin/nologin
create_home: true
home: /var/lib/caddy
- name: Render validated Caddyfile atomically
ansible.builtin.template:
src: Caddyfile.j2
dest: /etc/caddy/Caddyfile
owner: root
group: caddy
mode: "0640"
validate: "/usr/bin/caddy validate --config %s --adapter caddyfile"
notify: Reload caddy service
- name: Ensure Caddy systemd unit is enabled and active
ansible.builtin.systemd_service:
name: caddy
enabled: true
state: started
/api/v1 + /api/mcp/v1)For SaaS platforms provisioning isolated per-customer servers, CI pipelines spinning up branch preview environments, or AI SRE agents executing runbooks, Servers Are Up exposes full programmatic control over REST (/api/v1/instances) and Streamable HTTP MCP (/api/mcp/v1: sau_write_instance_file, sau_exec_instance, sau_ssh_authorize).
Notice a major security advantage of guest-channel automation (PUT /api/v1/instances/{id}/files and POST /api/v1/instances/{id}/exec, mirrored by sau_write_instance_file and sau_exec_instance): your orchestrator can push config files, wait for cloud-init status --wait, and run local HTTP readiness checks on internal private-subnet VMs with zero public IPv4 address ($0 IPv4 cost) and zero inbound SSH ports exposed:
// scripts/blue-green-provision.mjs
import crypto from "node:crypto";
const SAU_API = "https://serversareup.com/api/v1";
const SAU_TOKEN = process.env.SAU_API_TOKEN;
if (!SAU_TOKEN) throw new Error("SAU_API_TOKEN environment variable is required");
async function sauFetch(path, options = {}) {
const res = await fetch(`${SAU_API}${path}`, {
...options,
headers: {
Authorization: `Bearer ${SAU_TOKEN}`,
"Content-Type": "application/json",
...(options.headers || {}),
},
});
if (!res.ok) {
const body = await res.text();
throw new Error(`${options.method || "GET"} ${path} failed (${res.status}): ${body}`);
}
return res.status === 204 ? null : res.json();
}
export async function deployVerifiedGreenServer({ gitSha, productSlug = "linux-basic" }) {
const idempotencyKey = `bg-${gitSha.slice(0, 12)}-${crypto.randomUUID()}`;
let candidateId = null;
try {
// 1. Allocate green candidate VM deterministically
const vm = await sauFetch("/instances", {
method: "POST",
headers: { "Idempotency-Key": idempotencyKey },
body: JSON.stringify({
name: `app-green-${gitSha.slice(0, 8)}`,
productSlug, // e.g., linux-basic ($10/mo: 2 vCPU, 4 GB RAM, 80 GB persistent storage)
os: "ubuntu-24.04",
}),
});
candidateId = vm.id;
// 2. Poll until hypervisor & guest control channel report RUNNING
let ready = false;
for (let i = 0; i < 45; i++) {
const detail = await sauFetch(`/instances/${candidateId}`);
if (detail.status === "RUNNING") {
ready = true;
break;
}
if (detail.status === "ERROR") throw new Error("Hypervisor reported ERROR state");
await new Promise((r) => setTimeout(r, 4000));
}
if (!ready) throw new Error("Timed out waiting for RUNNING guest channel");
// 3. Write release manifest atomically via guest file API (MCP: sau_write_instance_file)
await sauFetch(`/instances/${candidateId}/files`, {
method: "PUT",
body: JSON.stringify({
path: "/etc/app-release.json",
mode: "0644",
content: JSON.stringify({ gitSha, provisionedAt: new Date().toISOString() }, null, 2),
}),
});
// 4. Block on cloud-init completion & local readiness endpoint (MCP: sau_exec_instance)
const check = await sauFetch(`/instances/${candidateId}/exec`, {
method: "POST",
body: JSON.stringify({
command: "cloud-init status --wait && systemctl is-active nftables && curl -fsS http://127.0.0.1:8080/readyz",
timeoutSeconds: 180,
}),
});
if (check.exitCode !== 0) {
throw new Error(`Synthetic verification failed (exit ${check.exitCode}): ${check.stderr}`);
}
return { instanceId: candidateId, status: "VERIFIED_GREEN" };
} catch (err) {
// 5. Mandatory rollback: delete failed candidate so broken builds never leak spend
if (candidateId) {
await sauFetch(`/instances/${candidateId}`, { method: "DELETE" }).catch(() => {});
}
throw err;
}
}
Once deployVerifiedGreenServer returns VERIFIED_GREEN, cutting traffic over without dropping active HTTP or WebSocket sessions requires updating your upstream reverse proxy (Caddy dynamic upstreams or Nginx reload) to point at the green node's private WireGuard or VPC IP, waiting 30–60 seconds for in-flight requests on the blue node to drain to zero, and only then terminating the blue server.
Choosing the right billing unit on Servers Are Up keeps automated blue/green pipelines cost-efficient:
Linux Starter ($6/mo, 2 vCPU / 2 GB RAM / 50 GB persistent storage), Linux Basic ($10/mo, 2 vCPU / 4 GB RAM / 80 GB persistent storage), Linux Standard ($18/mo, 4 vCPU / 8 GB RAM / 160 GB persistent storage), and Linux Pro ($28/mo, 8 vCPU / 16 GB RAM / 320 GB persistent storage).$12/mo, 4 vCPU / 8 GB RAM / 100 GB / 5 instances), Developer Studio ($24/mo, 8 vCPU / 16 GB RAM / 250 GB / 15 instances), or Team Workspace ($95/mo, 24 vCPU / 64 GB RAM / 1 TB / 40 instances)—leaving 25–30% spare pool headroom so green nodes boot alongside blue nodes inside your existing flat monthly pool with zero extra subscription churn.$3/machine/mo (/dashboard/hybrid) and pull runtime secrets dynamically from SAU Key Vault KMS (/dashboard/key-vault). For autonomous LLM-driven deployment workflows, pair this pipeline with our AI infrastructure management architecture.For hybrid teams pairing cloud-init automation with on-premises colocation cabinets, software convergence only solves half of the availability equation—standardized physical server rack organization, structured cable labeling, and cold-aisle airflow management prevent thermal throttling and accidental patch-cord disconnects before sau-connector or PXE boot scripts ever execute.
RUNNING state?Guest OS boot happens in stages: the Linux kernel and network interface come online several seconds before cloud-init finishes writing ~/.ssh/authorized_keys and reloading sshd. Never poll raw TCP port 22 as your sole readiness signal; wait for cloud-init status --wait (or /var/lib/cloud/instance/boot-finished) before running post-boot SSH tasks.
cloud-init changes locally without creating twenty VMs?Validate YAML syntax and schema structure first using cloud-init schema --config-file user-data.yaml (included in the cloud-init package on Ubuntu/Debian or inside an Ubuntu container). That catches indentation errors, deprecated module keys, and malformed write_files permissions before making a single hypervisor API call.
Apply the expand-and-contract pattern: every database migration deployed before a blue/green cutover must be backward-compatible with the currently running blue code (additive nullable columns, new indexes created CONCURRENTLY, dual-read tolerances). Drop legacy columns only in a subsequent release after the old blue server has been permanently retired.
Never commit terraform.tfstate to Git, because state files frequently capture computed attributes and sensitive tokens in cleartext. Store encrypted remote state inside an off-host object container such as Managed Cloud Storage hosted via Microsoft Azure (5 GiB free tier, AES-256-GCM + Azure storage encryption) and inject API credentials at runtime from CI secret stores or SAU Key Vault KMS.
Explore With AI & Cite This Page
Open a grounded prompt in your AI assistant or copy a clean Markdown, HTML, or APA reference link.
Validate protocols and subnets directly in your browser or continue reading related architecture notes.
SSH Keygen Generator & Cloud-Init YAML Builder
Free WHOIS & RDAP Domain & ASN Lookup Tool
Linux, Windows & Game Server Hardening Checklist
Per-Server vs. Resource-Based Cloud Pricing: Bin-Packing Math and Pooled Virtual Datacenter Economics
Why fixed T-shirt instance billing strands 35-55% of cloud RAM and vCPU, and how pooled Virtual Datacenters turn multi-node infrastructure sizing into an elastic bin-packing problem.
Production Incident Escalation, Self-Service Linux Diagnostics, and the ServersAreUp Support Runbook
A deterministic incident triage runbook for ServersAreUp engineers: isolating SDN vs guest OS faults with ss, journalctl, and mtr, plus escalating via /dashboard/support.
Production Bash Scripting on Linux Servers: Strict Mode, Signal Traps, Atomic Tempfiles, and flock Concurrency
Write defensive, production-grade Bash scripts on Linux servers: set -Eeuo pipefail subtleties, IFS safety, trap cleanup handlers, atomic mktemp writes, flock concurrency locks, and shellcheck.
Secure Remote Access to Your Servers From a Browser
How to get secure remote access to a private Linux or Windows server from a browser: keys, isolation, logging and a setup you can run in one tab.