feat: add scalable kubeadm runtime worker pool

Makefile

@@ -1,5 +1,6 @@
 # Standalone deployment surface for this repository.
-# The only cluster profile is single-VM Lima + kubeadm + Cilium + gVisor.
+# The local profile is one Lima + kubeadm cluster: trusted system services on
+# the control-plane node, gVisor Runtime workloads on a scalable tainted worker pool.
 # Image build contexts are deliberately rooted in this repository.
 SHELL := /usr/bin/env bash
 SANDBOX_KUBE_CONTEXT ?= sandbox-local
@@ -20,7 +21,7 @@ SANDBOX_BUILD_LOG_DIR ?= $(SANDBOX_STATE_DIR)/logs/images
 DOCKER_BUILD_PROGRESS ?= plain
 SANDBOX_BOOTSTRAP_STAMP ?= .venv/.sandbox-bootstrap
 
-.PHONY: help bootstrap quickstart acceptance smoke-local e2e-local verify verify-manifests doctor images image-runtime image-file-service image-control-plane image-console test chart-lint chart-render dev-token control-plane-forward console-forward up-local status-local down-local destroy-local
+.PHONY: help bootstrap quickstart acceptance smoke-local e2e-local verify verify-manifests doctor images image-runtime image-file-service image-control-plane image-console test chart-lint chart-render dev-token control-plane-forward console-forward up-local status-local scale-workers down-local destroy-local
 
 help:
 	@grep -E '^[a-z-]+:.* — ' Makefile | sed 's/:.* — / — /'
@@ -49,6 +50,7 @@ smoke-local: ## — prove gVisor isolation, persistence, fail-closed behavior, a
 e2e-local: ## — run all network, Runtime, storage, restart, and adversarial E2E checks
 	SANDBOX_KUBECONFIG=$(KUBECONFIG) \
 	SANDBOX_KUBE_CONTEXT=$(SANDBOX_KUBE_CONTEXT) \
+	SANDBOX_EXPECT_DEDICATED_RUNTIME_NODE=1 \
 	PYTHON=$(SANDBOX_PYTHON) bash scripts/run-all-e2e.sh
 
 verify: ## — run the complete source, package, Console, and manifest gate with logs
@@ -104,8 +106,12 @@ up-local: ## — deploy the standalone Lima/kubeadm environment
 status-local: ## — show the standalone Lima/kubeadm environment
 	bash scripts/local-cluster.sh status
 
-down-local: ## — stop the Lima VM and retain its disk
+scale-workers: ## — resize the Runtime worker pool with WORKERS=N
+	@test -n "$(WORKERS)" || { echo 'usage: make scale-workers WORKERS=N' >&2; exit 2; }
+	bash scripts/local-cluster.sh scale-workers "$(WORKERS)"
+
+down-local: ## — stop the control plane and all Runtime workers; retain disks
 	bash scripts/local-cluster.sh down
 
-destroy-local: ## — delete the Lima VM, .sandbox state, and local images
+destroy-local: ## — delete all profile VMs, .sandbox state, and local images
 	bash scripts/local-cluster.sh destroy

README.md

@@ -19,7 +19,7 @@ operation fails; it never falls back to running on the host.
                                 │  the tenant is decided by the credential,
                                 │  never by anything in the request body
 ════════════════════════════════▼═════════════════════════════════════════════
- namespace: sandbox-system                              trusted control plane
+ node: sandbox-local · namespace: sandbox-system        trusted control plane
  ┌─────────────────────────────────────────────────────────────────────────┐
  │  Control Plane  (control_plane/)                                        │
  │    admission · quotas · tokens · workspace and runtime lifecycle        │
@@ -33,7 +33,7 @@ operation fails; it never falls back to running on the host.
  └──────────────────────────────┬──────────────────────────────────────────┘
                                 │  in-cluster, NetworkPolicy-scoped
 ════════════════════════════════▼═════════════════════════════════════════════
- namespace: sandbox-workloads                default-deny, untrusted workload
+ nodes: sandbox-local-w1…wN · namespace: sandbox-workloads  Runtime worker pool
  ┌─────────────────────────────────────────────────────────────────────────┐
  │  Runtime Pod   RuntimeClass: gvisor                                     │
  │    runtime/       shell, PTY sessions, SSE streaming                    │
@@ -50,7 +50,7 @@ operation fails; it never falls back to running on the host.
 
 ```bash
 make bootstrap                 # create .venv and install SDK + test dependencies
-make test                      # 839 unit and contract tests, no network, no cluster
+make test                      # 844 unit and contract tests, no network, no cluster
 make verify                    # complete Python, Console, manifest, Helm, wheel gate
 make help                      # every Make target with its one-line description
 ```
@@ -187,8 +187,11 @@ that order with one command.
 
 ## Run the full local cluster
 
-`make up-local` builds a single-node kubeadm Kubernetes cluster inside a Lima VM with
-Cilium and gVisor, then deploys the whole platform into it. This is the only local
+`make up-local` builds one multi-node kubeadm Kubernetes cluster in Lima. The
+`sandbox-local` node hosts trusted system services; the tainted
+`sandbox-local-w1…wN` pool hosts gVisor Runtime workloads. Cilium connects the
+nodes and enforces the workload policies; CephFS keeps Workspaces mountable from
+any active worker. The default is one worker, not a fixed maximum. This is the only local
 cluster profile the project ships.
 
 ### Prerequisites
@@ -202,17 +205,29 @@ it first rather than discovering a gap halfway through the VM build.
 | Python | 3.11 or newer |
 | Host OS | macOS or Linux |
 | Host architecture | amd64 or arm64 (`scripts/local-cluster.yaml` pins Ubuntu images for both; gVisor is installed for `x86_64` and `aarch64`) |
-| **Available memory** | **6.5 GiB free** for a new profile — the default VM reserves 6 GiB; this is a hard check, not a warning |
-| **Free disk** | **25 GiB free** under `$LIMA_HOME` (default `~/.lima`) for a new profile — the 60 GiB VM disk is sparse; this is also a hard check |
+| **Available memory** | **10.5 GiB free** for the default new profile — 6 GiB for the control plane plus 4 GiB per worker; this is a hard check, not a warning |
+| **Free disk** | **35 GiB free** under `$LIMA_HOME` for the default new profile — the 60 GiB control-plane and 30 GiB-per-worker disks are sparse |
 | Virtualization | On Linux, a readable and writable `/dev/kvm`. Without it Lima falls back to QEMU TCG software emulation, which boots kubeadm many times slower and is not usable in practice. `make doctor` warns rather than fails on this one. |
 | Network | Egress to pull the Ubuntu cloud image, Kubernetes apt packages, Cilium, gVisor, Metrics Server, and Rook/Ceph images |
 
 Set `SANDBOX_DOCTOR_SKIP_RESOURCES=1` to bypass only the memory and disk checks. The
-VM itself uses 4 CPUs, 6 GiB memory, and a 60 GiB disk by default, adjustable through
-`SANDBOX_LOCAL_CPUS`, `SANDBOX_LOCAL_MEMORY_GIB`, and `SANDBOX_LOCAL_DISK_GIB`.
-When `sandbox-local` already exists, doctor switches to a 2 GiB memory / 5 GiB disk
-reuse gate because it is not allocating a second VM; explicit
-`SANDBOX_DOCTOR_MIN_*` overrides still win.
+control-plane VM uses 4 CPUs, 6 GiB memory, and a 60 GiB disk by default,
+adjustable through `SANDBOX_LOCAL_CPUS`, `SANDBOX_LOCAL_MEMORY_GIB`, and
+`SANDBOX_LOCAL_DISK_GIB`. The Runtime worker uses 4 CPUs, 4 GiB memory, and a
+30 GiB disk. `SANDBOX_LOCAL_WORKER_COUNT` selects the initial pool size;
+`SANDBOX_LOCAL_WORKER_CPUS`, `SANDBOX_LOCAL_WORKER_MEMORY_GIB`, and
+`SANDBOX_LOCAL_WORKER_DISK_GIB` size each worker. When all requested VMs already
+exist, doctor switches to a 2 GiB memory / 5 GiB disk reuse gate; expanding the
+pool adds capacity checks per missing worker. Explicit `SANDBOX_DOCTOR_MIN_*`
+overrides still win. When VM memory or disk sizes are customized, the resource
+gate derives its reservation from those configured sizes rather than retaining
+the default-profile threshold.
+
+Every profile owns a Lima `user-v2` network named `<control-plane>-net`; this is
+what gives each VM a distinct, mutually reachable address. Override it with
+`SANDBOX_LOCAL_NETWORK`; override its automatically selected unused `/24` with
+`SANDBOX_LOCAL_NETWORK_GATEWAY`. An older VM created on Lima's isolated default usernet
+is rejected rather than silently reused as a broken multi-node cluster.
 
 ### Bring it up
 
@@ -223,7 +238,7 @@ make up-local
 ```
 
 The first `make up-local` downloads and builds everything listed above, so its
-duration is dominated by your network throughput; subsequent runs reuse the VM disk.
+duration is dominated by your network throughput; subsequent runs reuse the existing VM disks.
 It ends by printing:
 
 ```text
@@ -236,6 +251,20 @@ The kubeconfig is written to `.sandbox/kubeconfig` inside the checkout, not to
 The Makefile exports `KUBECONFIG` for its own targets, so `make dev-token`,
 `make status-local`, and the port-forward targets need no manual export.
 
+Resize the Runtime pool without rebuilding the control plane:
+
+```bash
+make scale-workers WORKERS=3  # add/start w2 and w3, install gVisor, load Runtime image
+make scale-workers WORKERS=1  # drain and stop w2 and w3; disks are retained
+make scale-workers WORKERS=0  # allowed only when no Runtime Pods remain
+```
+
+Scale-down fails closed if a target node still has an active Runtime Pod. Release
+or checkpoint that Runtime first, then retry. Scaling back up reuses stopped worker
+disks; the stale Kubernetes Node registration is removed while stopped and kubeadm
+registers it again on scale-up. At zero workers, Runtime admission
+is paused and fails fast; the trusted storage and Console services remain available.
+
 ### First command
 
 ```bash
@@ -256,11 +285,11 @@ Workspace `demo`.
 ### Tear it down
 
 ```bash
-make down-local     # stop the Lima VM; its 60 GiB disk and .sandbox/ remain
-make destroy-local  # delete the VM with its disk and the generated .sandbox/ state
+make down-local     # stop the control plane and every worker; VM disks and .sandbox/ remain
+make destroy-local  # delete the control plane, every worker, and generated state
 ```
 
-`down-local` is the right choice between sessions — `up-local` reuses the stopped VM.
+`down-local` is the right choice between sessions — `up-local` reuses both stopped VMs.
 `destroy-local` is irreversible: Workspace files, checkpoints, and the SQLite state
 inside the VM are gone. It also removes the four fixed-tag project images from the
 local Docker daemon; shared base layers remain available to Docker's cache.
@@ -325,8 +354,8 @@ rather than a deployable overlay; CI checks the others on every push.
 
 | Profile | Intended for | Maturity |
 | --- | --- | --- |
-| `overlays/local` | The `make up-local` VM. SQLite state, local-path PVCs, single pinned volume agent. | Reference. Exercises isolation and recovery behavior; its single-node storage and SQLite database are **not** a production durability claim. |
-| `overlays/rwo-single-node` | Clusters without RWX storage. Trades volume-agent availability for portability: one RWO claim, one pinned replica. | Reference. |
+| `overlays/local` | The `1+N` `make up-local` cluster. SQLite state, CephFS RWX Workspaces, and a scalable Runtime pool. | Reference. Exercises placement, isolation, scaling, and recovery behavior; its single-OSD Ceph and SQLite database are **not** production durability claims. |
+| `overlays/rwo-single-node` | Clusters where all Workspace consumers share one RWO-capable Runtime node. Trades volume-agent availability for portability: one RWO claim, one pinned replica. | Reference. |
 | `overlays/eks` | Amazon EKS. Swaps in an EFS-backed StorageClass, which the operator must install first. | Adapter example — see [Known limitations](#known-limitations) before using it. |
 | `overlays/external-deps` | Control-plane state and object storage outside the cluster (managed database, S3-compatible store). | Example with placeholders. Nothing references it by default; copy it and fill in your own Secrets. |
 | `charts/sandbox` | Helm-based installs. `make chart-lint` and `make chart-render` validate it without a cluster. | Independently deployable package. |
@@ -373,7 +402,7 @@ the image references before treating this overlay as deployable.
 **The `sandbox-system` namespace has no NetworkPolicy.** All four shipped
 NetworkPolicies target `sandbox-workloads`. The Control Plane and Console are also
 exposed as NodePort services (30080 and 30081) in the base manifests. Both are
-acceptable for a single-node local VM and are not acceptable on a shared cluster
+acceptable for the dedicated local cluster and are not acceptable on a shared cluster
 without an ingress and namespace policies in front of them.
 
 **The workspace admission gate is a process-local lock.**

control_plane/core.py

@@ -638,10 +638,11 @@ def object_store_env(
     ("CHECKPOINT_RETENTION_SECONDS", CHECKPOINT_RETENTION_SECONDS),
     ("CHECKPOINT_GC_INTERVAL_SECONDS", CHECKPOINT_GC_INTERVAL_SECONDS),
     ("SANDBOX_MAX_WORKSPACES", MAX_WORKSPACES),
-    ("SANDBOX_MAX_RUNTIMES", MAX_RUNTIMES),
 ):
     if _value <= 0:
         raise ValueError(f"{_name} must be greater than zero")
+if MAX_RUNTIMES < 0:
+    raise ValueError("SANDBOX_MAX_RUNTIMES must be zero or greater")
 MAX_CONCURRENT_OBJECT_OPS = int(
     os.getenv("SANDBOX_MAX_CONCURRENT_OBJECT_OPS", "1")
 )

docs/ARCHITECTURE.md

@@ -112,7 +112,7 @@ or memory snapshots. See [Lifecycle and data](LIFECYCLE_AND_DATA.md).
 
 | Profile | Isolation and dependencies | Intended use |
 | --- | --- | --- |
-| Local integration | gVisor, Cilium, SQLite, local-path Workspace PVC, and cluster-local Ceph RGW | Reproducible isolation and E2E validation |
+| Local integration | gVisor, Cilium, SQLite, CephFS RWX Workspace PVC, and cluster-local Ceph RGW | Reproducible isolation, scaling, and E2E validation |
 | Operator production | gVisor nodes, enforcing CNI, PostgreSQL, external RWX and S3-compatible storage | ACK, EKS, or any conforming Kubernetes platform |
 
 The same Control Plane/Runtime contract applies across profiles. The local profile proves
@@ -150,6 +150,18 @@ provider is selected independently by `SANDBOX_RUNTIME_DRIVER`.
 The repository is self-contained: build, test, and local-cluster entrypoints
 resolve paths from this checkout.
 
+The repository-managed local profile uses one kubeadm cluster with a trusted
+control-plane node and a scalable pool of tainted gVisor Runtime workers. This is
+node isolation inside one Kubernetes trust domain, not a dual-cluster claim.
+Workers become schedulable only after node readiness, gVisor installation, and
+Runtime image loading. Scale-down first validates the entire target set and stops
+no node when any target still owns an active Runtime; it cordons and rechecks all
+targets to close the scheduler race, rolling those cordons back on refusal. CephFS RWX makes Workspace
+state independent of worker identity; scaling the pool to zero pauses Runtime
+admission without removing the trusted system and storage node. Stopped worker
+disks are retained, while their ephemeral Kubernetes Node registrations are removed
+and recreated on the next scale-up so node-wide DaemonSet rollouts remain healthy.
+
 Several control-plane modules remain larger than their domain boundaries. Future
 refactors should extract authentication, Workspace, Runtime, object, and admin route
 groups without changing the OpenAPI contract or authorization behavior. Route,

docs/COMPATIBILITY.md

@@ -13,14 +13,21 @@ not supported until a versioned compatibility suite proves otherwise.
 | Console | Node.js 22 build environment | Locked npm install, i18n, ESLint, TypeScript, and production build |
 | Kubernetes manifests | Provider-neutral base, local integration, EKS | Kustomize render and contract tests |
 
+The currently validated local integration stack is kubeadm/Kubernetes 1.36,
+Cilium 1.19.6, Rook 1.20.6, Ceph 20.2.4, and Ceph-CSI 3.17.1 on Ubuntu Noble.
+Rook and Ceph-CSI are separate checksum-pinned Helm charts; the Ceph and Ceph-CSI
+images are digest-pinned. Noble's 6.8 kernel uses the CephFS FUSE mounter in this
+profile. Treat any change to that tuple as a compatibility change that requires
+the full local E2E suite.
+
 ## Compatibility rules
 
 - Kubernetes must support RuntimeClass, NetworkPolicy, batch/v1 Jobs/CronJobs,
   leases, and the storage access mode selected by the overlay.
 - The `gvisor` RuntimeClass must resolve to a working `runsc` handler on every node
   eligible for Runtime Pods. An object alone is not proof of isolation.
-- Production deployments must replace development SQLite, the single-node Ceph
-  profile, and local-path storage with the dependencies and backup policies in
+- Production deployments must replace development SQLite and the single-OSD Ceph
+  profile with the dependencies and backup policies in
   [Production](PRODUCTION.md).
 - The local profile provides Metrics Server only through its overlay. Production must
   provide its own Metrics API and trusted kubelet serving certificate path.

docs/CONFIGURATION.md

@@ -144,7 +144,7 @@ attributed to a person, so:
 | `SANDBOX_TEMPLATES` | built-in default only | JSON object mapping approved template IDs to image references |
 | `SANDBOX_IMAGE_REGISTRIES` | empty | Allowed literal image prefixes; empty fails closed for API-managed templates |
 | `SANDBOX_MAX_WORKSPACES` | `64` | Global workspace admission cap |
-| `SANDBOX_MAX_RUNTIMES` | `4` | Global Runtime admission cap |
+| `SANDBOX_MAX_RUNTIMES` | `4` | Global Runtime admission cap; `0` pauses new Runtime admission |
 | `SANDBOX_TTL_SECONDS` | `1800` | Idle Runtime TTL |
 | `SANDBOX_RUNTIME_HARD_TTL_SECONDS` | `43200` | Absolute Runtime lifetime |
 | `WORKSPACE_IDLE_TTL_SECONDS` | `21600` | Workspace data idle TTL |