feat: add scalable multi-node kubeadm quickstart

Makefile

@@ -1,7 +1,7 @@
 SHELL := /usr/bin/env bash
 CONTROL_IMAGE ?= site-control:local
 
-.PHONY: help install test test-db test-db-down test-supabase test-oss console homepage-check image chart-lint chart-render benchmark quickstart-doctor quickstart quickstart-status quickstart-access quickstart-token quickstart-clean standalone-install standalone-smoke standalone-uninstall
+.PHONY: help install test test-db test-db-down test-supabase test-oss console homepage-check image chart-lint chart-render benchmark quickstart-doctor quickstart quickstart-scale quickstart-status quickstart-access quickstart-token quickstart-clean standalone-install standalone-smoke standalone-uninstall
 
 # Generated from the targets themselves rather than kept as a second list. The
 # hand-written listing this replaces had already lost `help`, and nothing could
@@ -82,9 +82,12 @@ benchmark: ## — run the fail-closed deterministic benchmark profile
 quickstart-doctor: ## — check every host prerequisite and local port before the trial
 	scripts/quickstart-kubeadm.sh doctor
 
-quickstart: ## — build and prove Site on a disposable kubeadm VM
+quickstart: ## — build and prove Site on a disposable multi-node kubeadm cluster
 	scripts/quickstart-kubeadm.sh up
 
+quickstart-scale: ## — reconcile the trial to SITES_QUICKSTART_WORKERS=1-4
+	scripts/quickstart-kubeadm.sh scale
+
 quickstart-status: ## — inspect the disposable kubeadm trial
 	scripts/quickstart-kubeadm.sh status
 
@@ -94,7 +97,7 @@ quickstart-access: ## — open and serve the console from the disposable kubeadm
 quickstart-token: ## — print the disposable trial's local admin token
 	scripts/quickstart-kubeadm.sh token
 
-quickstart-clean: ## — delete only the disposable Site kubeadm VM
+quickstart-clean: ## — delete only the disposable Site kubeadm VMs and network
 	scripts/quickstart-kubeadm.sh clean
 
 standalone-install: ## — bootstrap local Secrets and install the Helm Chart

README.md

@@ -24,8 +24,8 @@ environments, not a public hosting platform — [Known limitations](#known-limit
 ### Host prerequisites
 
 The supported source-checkout trial runs on macOS or Linux with hardware virtualization,
-outbound HTTPS, and enough capacity for a Lima VM configured with **6 CPUs, 8 GiB RAM,
-and a 40 GiB sparse disk**. Install Git, Lima, Docker, `kubectl`, Helm, `curl`, `uv`,
+outbound HTTPS, and enough capacity for three Lima VMs configured with **8 CPUs, 10 GiB
+RAM, and 70 GiB of sparse disk in total**. Install Git, Lima, Docker, `kubectl`, Helm, `curl`, `uv`,
 `lsof`, and Python 3.12+. Docker must be running, not merely installed.
 
 On macOS with Homebrew, the command-line dependencies are:
@@ -42,11 +42,12 @@ Linux users should install [Docker Engine](https://docs.docker.com/engine/instal
 [Helm](https://helm.sh/docs/intro/install/) using their distribution's supported path.
 Run `make quickstart-doctor` at any time for an explicit dependency, Docker-daemon, Python,
 and local-port check. The first uncached run downloads a VM and several container images;
-depending on the network it can take 5–25 minutes and needs ports 18090–18098 and 18447.
+depending on the network it can take 8–30 minutes and needs ports 18090–18098 and 18447.
 
 The fastest path is deliberately end to end. With the prerequisites above, one command
-creates a disposable single-node Kubernetes
-cluster with `kubeadm`, builds the image from this checkout, installs Site and Prometheus,
+creates a disposable Kubernetes cluster with one control-plane and two worker nodes using
+`kubeadm`, keeps all Site components off the tainted control-plane, builds the image from
+this checkout, installs Site and Prometheus,
 deploys the included HTML example, and refuses to pass until both server-side HTTP
 verification and the administrative observability APIs return real evidence:
 
@@ -61,6 +62,9 @@ A successful run ends with facts rather than only a Helm exit code:
 ```text
 Site kubeadm quickstart passed
   elapsed: <seconds>
+  topology: 1 control-plane + 2 workers (3 Ready)
+  example node: site-quickstart-w<1-or-2>
+  control-plane scheduling: protected by NoSchedule taint
   phase: Running
   HTTP verification: 200
   body SHA-256: <digest of the served response>
@@ -93,13 +97,31 @@ application from an existing container image; choose **Public** to receive a bro
 or **Internal** when no public URL should exist. The token remains in
 a Kubernetes Secret and is never written into this checkout or browser storage. Inspect
 the running objects with `make quickstart-status`. Pressing Ctrl-C in terminal 1 only
-closes console access; the application and cluster keep running. `make quickstart-clean`
-permanently deletes the disposable `site-quickstart` VM, all applications inside it, and
-the repository-local kubeconfig; it does not touch another cluster or repository.
+closes console access; the application and cluster keep running.
 
-This trial harness is entirely owned by this repository: it uses Lima's default network,
-bootstraps Kubernetes directly with kubeadm, and builds the control image locally. No
-other checkout, shared cluster configuration, or pre-published Site image is required.
+Worker count can be changed in place from one through four. Expansion creates and joins
+new kubeadm worker VMs; contraction drains the highest-numbered workers before deleting
+them. Quickstart binds local persistent volumes to `site-quickstart-w1`, which is always
+retained, and refuses to delete a worker if a local persistent volume is unexpectedly
+pinned there:
+
+```bash
+SITES_QUICKSTART_WORKERS=3 make quickstart-scale  # add w3
+SITES_QUICKSTART_WORKERS=1 make quickstart-scale  # drain/delete w3 and w2
+```
+
+This changes worker capacity only. The single control-plane makes the trial a real
+multi-node functional environment, not a highly available production topology.
+`make quickstart-clean` permanently deletes the three default `site-quickstart` VMs, all
+applications inside them, the repository-owned Lima network, and the repository-local
+kubeconfig; it does not touch another cluster, network, or repository.
+
+This trial harness is entirely owned by this repository: it creates its own Lima user-v2
+network, bootstraps Kubernetes directly with kubeadm, and builds the control image locally.
+Fixed host ports enter through the control-plane VM; a repository-managed, reboot-persistent
+relay sends only those ports to retained worker `w1`, and Kubernetes routes them to whichever
+worker owns the application Pod. No other checkout, shared cluster configuration, or
+pre-published Site image is required.
 
 ```text
  ┌─ an agent asks for a deployment ────────────────────────────────────────┐

charts/site/templates/01-storage.yaml

@@ -134,6 +134,13 @@ metadata:
 provisioner: rancher.io/local-path
 volumeBindingMode: WaitForFirstConsumer
 reclaimPolicy: Delete
+{{- with .Values.localPathProvisioner.allowedNodeNames }}
+allowedTopologies:
+  - matchLabelExpressions:
+      - key: kubernetes.io/hostname
+        values:
+{{ toYaml . | indent 10 }}
+{{- end }}
 ---
 kind: ConfigMap
 apiVersion: v1

charts/site/templates/11-monitoring.yaml

@@ -260,12 +260,6 @@ spec:
     spec:
       serviceAccountName: sites-prometheus
       enableServiceLinks: false
-      nodeSelector:
-        node-role.kubernetes.io/control-plane: ""
-      tolerations:
-        - key: node-role.kubernetes.io/control-plane
-          operator: Exists
-          effect: NoSchedule
       securityContext:
         runAsNonRoot: true
         runAsUser: 65534

charts/site/values.schema.json

@@ -90,7 +90,25 @@
       }
     },
     "localPathProvisioner": {
-      "$ref": "#/$defs/feature"
+      "type": "object",
+      "additionalProperties": false,
+      "required": [
+        "enabled",
+        "allowedNodeNames"
+      ],
+      "properties": {
+        "enabled": {
+          "type": "boolean"
+        },
+        "allowedNodeNames": {
+          "type": "array",
+          "items": {
+            "type": "string",
+            "minLength": 1
+          },
+          "uniqueItems": true
+        }
+      }
     },
     "postgresql": {
       "type": "object",

charts/site/values.yaml

@@ -58,6 +58,9 @@ storage:
 
 localPathProvisioner:
   enabled: false
+  # Optional safety boundary for disposable multi-node clusters. All local
+  # volumes bind to one retained worker so other workers can be drained safely.
+  allowedNodeNames: []
 
 postgresql:
   embedded:

dev/kubeadm/lima.yaml

@@ -4,21 +4,24 @@ images:
   - location: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-arm64.img
     arch: aarch64
 
-cpus: 6
-memory: 8GiB
-disk: 40GiB
+# Control-plane defaults. The quickstart overrides these for worker VMs.
+cpus: 4
+memory: 4GiB
+disk: 30GiB
 
-# The host talks to the API only through this repository-owned forward. The VM
-# uses Lima's default network, so it does not require a pre-created network or
-# configuration from another project.
+# The host talks to the API only through this repository-owned forward. The
+# quickstart attaches every VM to a user-v2 network that it creates and owns;
+# no pre-created network or configuration from another project is required.
 portForwards:
   - guestPort: 6443
     hostPort: 18447
     static: true
-  # NodePort exposure is the quickstart's public application path. Keep this
-  # mapping aligned with src/sites/exposure.py: 30080, 30082-30088 become
-  # 18090, 18092-18098. 30081 is deliberately excluded and 18091 serves the
-  # admin console through kubectl port-forward instead.
+  # NodePort exposure is the quickstart's public application path. Lima sends
+  # these fixed forwards to guest loopback; quickstart-kubeadm.sh owns a narrow
+  # systemd/iptables relay from loopback to retained worker w1 so the target Pod
+  # may live on any worker. Keep this mapping aligned with src/sites/exposure.py:
+  # 30080, 30082-30088 become 18090, 18092-18098. 30081 is deliberately excluded
+  # and 18091 serves the admin console through kubectl port-forward instead.
   - guestPort: 30080
     hostPort: 18090
     static: true

docs/DEPLOYMENT.md

@@ -66,26 +66,34 @@ workflow, plus a separate `site-values-VERSION.yaml` digest-pinned override and
 
 Prerequisites for the complete local trial: Lima, a running Docker daemon, kubectl, Helm,
 curl, uv, lsof, and Python 3.12+. Run `make quickstart-doctor` for the executable preflight.
-The VM allocates 6 CPUs, 8 GiB RAM, and a 40 GiB sparse disk; outbound HTTPS and host
-ports 18090–18098 plus 18447 are required. The repository creates one disposable VM and
-bootstraps Kubernetes with kubeadm; it does not use a shared cluster or another checkout.
+The default three VMs allocate 8 CPUs, 10 GiB RAM, and 70 GiB of sparse disk in total;
+outbound HTTPS and host ports 18090–18098 plus 18447 are required. The repository creates
+one control-plane and two worker VMs on its own Lima network and bootstraps Kubernetes with
+kubeadm; it does not use a shared cluster or another checkout.
 
 ```bash
 make quickstart-doctor
 make quickstart
+SITES_QUICKSTART_WORKERS=3 make quickstart-scale  # supported range: 1-4 workers
 make quickstart-status
 make quickstart-access
 make quickstart-token     # run in a second terminal, then log in
 make quickstart-clean
 ```
 
 `make quickstart` exits after verification. `make quickstart-access` must remain running
-while the console is in use; Ctrl-C stops only that tunnel. `make quickstart-clean`
-permanently removes the disposable VM and all applications stored inside it.
-
-The quickstart uses a single node to minimize first-run time. Production cluster
-provisioning, upgrades, backup, and disaster recovery are deliberately not
-implemented in this repository.
+while the console is in use; Ctrl-C stops only that tunnel. Worker scale-down drains nodes
+and retains `w1`, where quickstart constrains local persistent volumes. It refuses to
+remove a worker carrying an unexpected local volume. `make quickstart-clean` permanently
+removes the disposable VMs, their repository-owned network, and all applications inside.
+
+The quickstart is a multi-node functional topology with one control-plane, not a highly
+available production cluster. Production provisioning, control-plane HA, upgrades, backup,
+and disaster recovery are deliberately not implemented in this repository.
+The host-visible application ports terminate on control-plane loopback and pass through a
+repository-managed systemd/iptables relay to retained worker `w1`; kube-proxy and Cilium
+route onward to the selected workload Pod. The relay covers only the documented NodePort
+pool, survives VM reboot, and disappears when the disposable control-plane VM is deleted.
 
 The chart defaults deploy the core API, operator, PostgreSQL, registry, build
 plane, and console on the NodePort backend. Gateway routing and scale-to-zero

docs/STANDALONE.md

@@ -6,20 +6,22 @@ root:
 ```bash
 make quickstart-doctor  # checks commands, Docker, Python, and fixed host ports
 make quickstart
+SITES_QUICKSTART_WORKERS=3 make quickstart-scale  # optional: resize to 1-4 workers
 make quickstart-access   # terminal 1: opens and serves the console
 make quickstart-token    # terminal 2: paste into 管理员 token / Admin token
-make quickstart-clean    # deletes only the site-quickstart Lima VM and local state
+make quickstart-clean    # deletes only the Site trial VMs, network, and local state
 ```
 
-The trial VM is configured for 6 CPUs, 8 GiB RAM, and a 40 GiB sparse disk. It needs
+The default three VMs allocate 8 CPUs, 10 GiB RAM, and 70 GiB of sparse disk in total. They need
 outbound HTTPS while the Ubuntu, Kubernetes, Cilium, and workload images download, plus
-host ports 18090–18098 and 18447. A first uncached install may take 5–25 minutes.
+host ports 18090–18098 and 18447. A first uncached install may take 8–30 minutes.
 `make quickstart` exits after printing the proof. Keep `make quickstart-access` running;
 pressing Ctrl-C there closes only the console tunnel, not the cluster. In the default
 Chinese UI choose **进入控制台**; in English choose **Enter the console**.
 
-That path creates a single-node Lima VM, installs Kubernetes 1.36 with kubeadm, installs
-a pinned Cilium release, and builds the control image locally. The Helm release supplies
+That path creates one control-plane and two worker Lima VMs, installs Kubernetes 1.36 with
+kubeadm, joins the workers, installs a pinned Cilium release, and builds the control image
+locally. The Helm release supplies
 its own local storage provisioner, enables Prometheus, deploys `examples/hello-site`, and
 gates success on the deployment's HTTP status/body digest, a host-openable public URL with
 the same body digest, plus the admin metrics and deployment-snapshot APIs. It needs no
@@ -30,8 +32,22 @@ for its resource samples. **Deploy application** creates or updates from an exis
 container image: public applications receive a browser URL, while internal applications
 are explicitly labeled as having no public URL.
 
+`SITES_QUICKSTART_WORKERS=N make quickstart-scale` reconciles the live trial to one through
+four workers. Scale-down drains the highest-numbered workers first. All local persistent
+volumes are constrained to worker `w1`, which is retained; the command refuses removal if
+it detects a local volume on a candidate node. This only scales workers: one control-plane
+is not an HA topology.
+
+The control-plane keeps its kubeadm `NoSchedule` taint. Site, Prometheus, and tenant Pods
+run on workers; the quickstart fails if a Site workload crosses that boundary.
+Lima's fixed host ports terminate on control-plane loopback, so the repository installs a
+narrow systemd/iptables relay for ports 30080 and 30082–30088 to retained worker `w1`.
+Kube-proxy and Cilium then route to the actual application worker. The relay is recreated
+on control-plane reboot and removed with the disposable VM.
+
 `make quickstart-clean` is intentionally destructive only to the disposable trial: it
-deletes that VM, every application inside it, and this checkout's local kubeconfig.
+deletes its VMs, repository-owned Lima network, every application inside it, and this
+checkout's local kubeconfig.
 
 The lower-level path installs Site into the Kubernetes context selected by `kubectl`.
 Review the current context before continuing: