review: whole-repo audit after the volume merges, with the #403 controller docs

README.md

@@ -34,8 +34,9 @@ performance) — source in
   baked into the base image
 - `sandboxd/` — per-node control plane (Go): warm pools refilled from golden
   snapshot exports (online-retunable), claim/release/renew/hibernate/fork/
-  promote/checkpoint HTTP API, operator-catalog dataset volumes (read-only or
-  writable, mounted or attach-only, with live add and remove), signed preview URLs,
+  promote/checkpoint HTTP API, operator-catalog dataset volumes (static or
+  registered at runtime, read-only or writable, mounted or attach-only, with
+  live add and remove), signed preview URLs,
   the HTTP-upgrade byte relays to silkd and to any guest port, read-only file
   access to hibernated or archived claims from a warm VM of their pool
   (opt-in per pool `reader`), claim env with

docs/deploy.md

@@ -310,7 +310,8 @@ A volume claim may consume an ordinary warm Cloud Hypervisor VM. sandboxd
 attaches after the warm pop or provision, waits for the device settle — a
 serial match under `/sys/block`, then the `/dev/<name>` node itself, since the
 kernel publishes sysfs before devtmpfs creates it — up to 2 seconds, then
-mounts the device before finalizing the claim: `mode: "ro"` (the default)
+mounts the device, retrying `not a valid block device` within the 10-second
+setup budget, before finalizing the claim: `mode: "ro"` (the default)
 attaches and mounts read-only; `mode: "rw"` requires the catalog entry's
 `writable: true` and attaches and mounts read-write. Setup failure destroys
 the VM without quiescing — the claim was never handed out, so no workload

docs/index.md

@@ -29,6 +29,8 @@ vsock-only I/O (hardened default); `net=egress` attaches a bridge/CNI NIC.
   online pool reconfigure, metrics, the usage journal
 - [Replace a dataset in a running sandbox](live-volumes.md) — swapping a
   read-only catalog volume on a running claim with the volume API
+- [Hibernate with an external volume controller](volume-controller.md) — detach,
+  hibernate, wake, and reattach while retaining the reservation and access gate
 - [Go SDK](sdk.md) — connecting (single node and clusters), every option,
   the full sandbox surface, error handling
 - [Python SDK](sdk-python.md) — the guest and data-plane surface for the

docs/live-volumes.md

@@ -7,7 +7,8 @@ Register both read-only catalog entries on the owning node and grant the
 sandbox's tenant access to them. Put `v1` and `v2` in each image's `/version`
 file. Keep each name bound to the same immutable image while held. Use Cocoon
 v0.7.2 or newer and set `SANDBOX_ADDR` and `SANDBOX_API_TOKEN` for the client.
-A sandbox token alone cannot manage volumes.
+A sandbox token alone cannot manage volumes. For a complete sleep and wake sequence,
+see [the external controller example](volume-controller.md).
 
 Entries [registered at runtime](sandboxd-api.md#put-v1volumesname) work the
 same way. The API also supports

docs/sandboxd-api.md

@@ -129,8 +129,10 @@ entries, matching the request shape. sandboxd attaches each disk, waits for
 the device settle — a serial match under `/sys/block`, then the `/dev/<name>`
 node itself, since the kernel publishes sysfs before devtmpfs creates it — up
 to 2 seconds total, and mounts the filesystem — read-only, unless the entry
-requested and was granted `rw` — before returning. A custom mount may shadow
-an existing populated guest directory for the claim's life.
+requested and was granted `rw` — before returning. A mount that reports
+`not a valid block device` is retried within the 10-second setup budget,
+since the new node can exist before the kernel opens the disk. A custom mount
+may shadow an existing populated guest directory for the claim's life.
 
 ### Attach-only volumes
 
@@ -168,7 +170,8 @@ done
 printf '%s\n' "$device"
 ```
 
-Then mount it however the workload needs. A `ro` entry is attached
+Then mount it however the workload needs, retrying a mount that reports
+`not a valid block device` for a short time. A `ro` entry is attached
 `--readonly` and stays read-only at the guest block layer no matter who
 mounts it, so the guarantee does not depend on your mount flags.
 
@@ -887,7 +890,7 @@ an SDK caller should set.
 Auth: node API token. Root sees every live claim; a tenant sees only its own.
 The index is `{"sandboxes": [{id, key, tenant?, deadline, claimed_at?, hibernated,
 archived?, from_checkpoint?, claim_ref?, metadata?, on_expire?, cpu_count, mem_total_bytes,
-volumes?: [{name, mount?, mode?}], volumes_want?: [{name, mount?, mode?}], restarts?, restarted_at?, failed?}]}` — `cpu_count` and `mem_total_bytes` are the
+volumes?: [{name, mount?, mode?, attach_only?}], volumes_want?: [{name, mount?, mode?, attach_only?}], restarts?, restarted_at?, failed?}]}` — `cpu_count` and `mem_total_bytes` are the
 size tier's allocation; `restarts` counts the cold boots that replaced an
 exited VMM and `restarted_at` stamps the last one (each loses guest memory and
 the instance-metadata document, so a caller re-establishes guest state when

docs/sdk-python.md

@@ -106,8 +106,8 @@ sb = client.new("ghcr.io/cocoonstack/sandbox/rt:24.04",
 |---|---|---|---|
 | `net` | `"none"`, `"egress"` | `"none"` | Cloud Hypervisor network shape: `none` disables the NIC and uses vsock-only I/O; `egress` attaches a bridge/CNI NIC |
 | `size` | `"small"`, `"medium"`, `"large"`, `"xlarge"`, `"2xlarge"` | `"small"` | resource tier: 1cpu/512M, 2cpu/1G, 4cpu/4G, 4cpu/8G, 8cpu/16G |
-| `volumes` | bare names or `{name, mount?, mode?}` mappings | `None` | attach and mount up to eight unique catalog dataset disks; an omitted mount defaults to `/volumes/<name>`; `mode` is `"ro"` (default) or `"rw"` — `"rw"` requires the catalog entry's `writable: true`; accepted by `Client.new` and `Template.new` |
-| `mount` | bool | `True` | mount every requested volume. `False` attaches the devices and leaves the mounting to the workload; a mapping carrying `mount` is then a `TypeError` |
+| `volumes` | bare names or `{name, mount?, mode?, attach_only?}` mappings | `None` | attach and mount up to eight unique catalog dataset disks; an omitted mount defaults to `/volumes/<name>`; `mode` is `"ro"` (default) or `"rw"` — `"rw"` requires the catalog entry's `writable: true`; accepted by `Client.new` and `Template.new` |
+| `mount` | bool | `True` | mount every requested volume. `False` attaches the devices and leaves the mounting to the workload; sandboxd then rejects a mapping carrying `mount` with 400 |
 | `ttl_seconds` | int | server default 5m | sandbox TTL, server-capped at 24h. The node reaps the sandbox after the TTL even if the client vanishes |
 | `deadline` | `time.monotonic()` value | `None` | keyword-only wall clock over the whole claim, redirect targets included: every request is bounded by whatever is left and a spent deadline raises `TimeoutError`. Without it each redirect candidate gets a fresh `timeout`, so a cluster with unreachable peers can cost a multiple of it. Also on `Template.new` and `Checkpoint.new` |
 
@@ -137,6 +137,9 @@ entirely yours: sandboxd writes and clears no dirty marker for an attach-only
 `rw` claim, because it cannot verify your unmount, so releasing without
 unmounting cleanly leaves the image as a crash would — see
 [sandboxd-api](sandboxd-api.md#attach-only-volumes) for the full contract.
+For a root client managing a tenant claim, see the
+[external volume controller example](volume-controller.md). It keeps the
+controller reservation and access gate across detach, hibernate, and wake.
 
 The caller-visible constraints are deliberate: volume claims may consume a
 warm VM, remain non-capturable, mount read-only by default, and require Cloud

docs/sdk.md

@@ -210,8 +210,8 @@ defer sb.Close()
 |---|---|---|---|
 | `WithNetwork(n)` | `NetNone`, `NetEgress` | `NetNone` | Cloud Hypervisor network shape: `NetNone` disables the NIC and uses vsock-only I/O; `NetEgress` attaches a bridge/CNI NIC |
 | `WithSize(s)` | `Small`, `Medium`, `Large`, `XLarge`, `XXLarge` | `Small` | resource tier: 1cpu/512M, 2cpu/1G, 4cpu/4G, 4cpu/8G, 8cpu/16G |
-| `WithVolumes(volumes...)` | `Volume{Name, Mount?, Mode?}` entries | none | attach and mount up to eight unique catalog dataset disks; `Mount` defaults to `/volumes/<name>`; `Mode` is `"ro"` (default) or `"rw"` — `"rw"` requires the catalog entry's `writable: true`; supported by `Client.New` and `Template.New` |
-| `WithVolumesAttachOnly()` | — | mount | attach the requested volumes without mounting them; the workload finds each device and owns the mount. Rejects a `Volume.Mount` locally |
+| `WithVolumes(volumes...)` | `Volume{Name, Mount?, Mode?, AttachOnly?}` entries | none | attach and mount up to eight unique catalog dataset disks; `Mount` defaults to `/volumes/<name>`; `Mode` is `"ro"` (default) or `"rw"` — `"rw"` requires the catalog entry's `writable: true`; supported by `Client.New` and `Template.New` |
+| `WithVolumesAttachOnly()` | — | mount | legacy claim-wide form of `Volume.AttachOnly`: attach every requested volume without mounting it; the workload finds each device and owns the mount. sandboxd rejects a `Volume.Mount` with 400 |
 | `WithTimeout(d)` | duration | server default 5m | sandbox TTL, rounded up to seconds, server-capped at 24h. The node reaps the sandbox after the TTL even if the client vanishes |
 
 `New` returns when the sandbox's silkd answers: a warm hit is milliseconds,

docs/volume-controller.md

@@ -0,0 +1,90 @@
+# Hibernate a sandbox with an external volume controller
+
+Remove a data disk before hibernation and restore it after wake, keeping the same
+claim and catalog registration. The controller owns reservation and access policy.
+
+## Prepare the claim and controller
+
+Use a `net: none` claim on its owning node, since egress-lane claims cannot
+hibernate, and keep its TTL valid across sleep. Its pool sets
+`replicate_seconds: 0` and `archive_after_seconds: 0`, and the claim uses
+`on_expire: "destroy"`: a detached sleeping claim holds no volume, so a replica
+or an archive of it could wake on another node. Keep the pool configured, because
+an unpooled key falls back to the node-wide `archive_after_seconds`. Replication
+changes only at a restart; see
+[disabling archive failover](deploy.md#disabling-archive-failover) before
+changing it on a node with replicated claims.
+
+Use Cloud Hypervisor, Cocoon v0.7.2 or newer, and the current Python SDK with
+[per-entry attach-only support](sdk-python.md#changing-live-volumes).
+Prepare and format the backing outside sandboxd. Follow the
+[registration and backing requirements](sandboxd-api.md#put-v1volumesname),
+including stable paths outside daemon cleanup directories. Register on the
+claim's node with `writable: true`, the configured root token, and an explicit
+[tenant access list](sandboxd-api.md#volume-access-lists).
+
+Hold a durable external reservation throughout use, detached sleep, and recovery.
+Prevent other root or tenant callers from attaching the backing while detached.
+A tenant access list is not a reservation. On owner change, keep access closed
+and fence the old node before attaching elsewhere. Node-local admission is not
+a distributed single-writer lock.
+
+## Supply the application hooks
+
+Copy the [tested example](https://github.com/cocoonstack/sandbox/blob/main/sdk/python/examples/volume_controller.py) and implement
+its `VolumeHooks`. Each hook raises on failure, and the controller keeps the
+reservation and the access gate through every step.
+
+- `assert_reserved` checks the external reservation and the owning node. It is
+  not a lock or a lease renewal; serialize controller operations on one claim.
+- `close_access` blocks and drains business traffic on every node, including the
+  guest API and direct ports. Issued preview URLs can wake the claim, and
+  stopping new URL issuance does not close access.
+- `quiesce_and_unmount` stops writers, flushes state, and unmounts normally,
+  never lazily or by force. It raises on a busy mount, accounts for raw-device
+  users, partitions, and other mount namespaces, and tolerates a disk that is
+  already unmounted.
+- `mount_and_verify` finds the device by its serial and mounts it as described
+  in [attach-only volumes](sandboxd-api.md#attach-only-volumes), then checks the
+  filesystem identity and data.
+- `open_access` admits traffic only after verification succeeds.
+
+Use one attach-only writable volume. The controller owns filesystem consistency
+and crash recovery. Attach-only RW writes no `.dirty` marker. An automatically
+mounted RW volume that the controller unmounts externally retains its marker,
+because sandboxd cannot confirm the clean unmount. See the
+[dirty-marker rules](sandboxd-api.md#patch-v1sandboxesidvolumes).
+
+Bind a root client to the existing tenant claim. The root token authorizes
+volume changes. The sandbox token authorizes guest commands and hibernation.
+
+```python
+from cocoonsandbox import Client
+
+root = Client(owner_addr, api_token=root_node_token)
+operator = root.attach(owner_addr, sandbox_id, sandbox_token)
+```
+
+Call `resume_volume` for initial attachment to an empty running claim or after
+sleep. It wakes silkd, attaches, mounts, and verifies before opening access.
+Call `hibernate_volume` after use. Do not use a context manager that releases
+the claim when the sleep function returns. Neither function releases the reservation.
+
+## Recover with access closed
+
+For attach or detach failures, follow [partial failure and recovery](sandboxd-api.md#partial-failure-and-recovery).
+Require a successful remove response before hibernation. An empty GET view does
+not acknowledge durable retirement. After a failed, timed-out, or lost response,
+retry the same remove target until acknowledged. Any remaining desired, applied,
+or per-volume state, or ongoing convergence, prevents sleep.
+
+On any failure or cancellation, retain the reservation and keep access closed.
+Inspect the confirmed claim state and resume from that stage. Do not blindly
+repeat writes or formatting. `exec("true")` proves silkd readiness, not disk
+attachment or mount readiness. Recover an uncertain gate operation externally,
+including a partial reopen.
+
+After a cycle, verify the same claim, backing, serial, filesystem, and payload.
+The backing is external to the memory snapshot. The repository tests mock the
+HTTP API, so qualify guest unmount, disk integrity, external gates, and fencing
+on the real deployment.

e2e/volume_catalog_test.go

@@ -2,6 +2,7 @@ package e2e
 
 import (
 	"encoding/json/v2"
+	"errors"
 	"net/http"
 	"slices"
 	"testing"
@@ -72,3 +73,85 @@ func TestRuntimeVolumeRegistrationThroughHTTP(t *testing.T) {
 		t.Fatalf("forbidden and unknown responses differ: %s versus %s", responses[0], responses[1])
 	}
 }
+
+func TestRuntimeVolumeDetachHibernateWakeThroughHTTP(t *testing.T) {
+	st := startTenantStack(t, "node-token", []tenants.TenantSpec{{Name: "acme", Token: "acme-token"}}, nil)
+	request, err := json.Marshal(map[string]any{"path": writeVolumeImage(t, "home.img", "external bytes"), "writable": true, "tenants": []string{"acme"}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if status, body := rawJSON(t, st, http.MethodPut, "/v1/volumes/home", string(request)); status != http.StatusNoContent {
+		t.Fatalf("registration: %d %s", status, body)
+	}
+	acme, err := sandbox.Connect(st.addr, sandbox.WithAPIToken("acme-token"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	claim, err := acme.New(t.Context(), "rt:24.04")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer claim.Close()
+	other, err := acme.New(t.Context(), "rt:24.04")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer other.Close()
+	operator := st.client.Attach(st.addr, claim.ID, "")
+	otherOperator := st.client.Attach(st.addr, other.ID, "")
+	volume := sandbox.Volume{Name: "home", Mode: "rw", AttachOnly: true}
+	if _, err = operator.PatchVolumes(t.Context(), sandbox.VolumePatch{Add: []sandbox.Volume{volume}}); err != nil {
+		t.Fatal(err)
+	}
+	if err = claim.Hibernate(t.Context()); err == nil {
+		t.Fatal("captured an attached disk")
+	} else if refusal, ok := errors.AsType[*sandbox.APIError](err); !ok || refusal.Status != http.StatusConflict {
+		t.Fatalf("attached capture refusal: %v", err)
+	}
+	if _, err = otherOperator.PatchVolumes(t.Context(), sandbox.VolumePatch{Add: []sandbox.Volume{volume}}); err == nil {
+		t.Fatal("root admitted a second writer before detach")
+	} else if refusal, ok := errors.AsType[*sandbox.APIError](err); !ok || refusal.Status != http.StatusConflict {
+		t.Fatalf("second writer refusal: %v", err)
+	}
+	view, err := operator.PatchVolumes(t.Context(), sandbox.VolumePatch{Remove: []string{"home"}})
+	if err != nil || len(view.Volumes) != 0 || len(view.VolumesWant) != 0 || len(view.States) != 0 || view.Converging {
+		t.Fatalf("detach acknowledgment: %+v %v", view, err)
+	}
+	if err = claim.Hibernate(t.Context()); err != nil {
+		t.Fatal(err)
+	}
+	if !hibernated(t, st, claim.ID) {
+		t.Fatal("detach followed by hibernate did not sleep the claim")
+	}
+	registered, listErr := st.client.Volumes(t.Context())
+	if listErr != nil || len(registered) != 1 || registered[0].Name != "home" || !registered[0].Writable {
+		t.Fatalf("registration lost across sleep: %+v %v", registered, listErr)
+	}
+	if _, err = other.PatchVolumes(t.Context(), sandbox.VolumePatch{Add: []sandbox.Volume{volume}}); err != nil {
+		t.Fatalf("detached volume stayed reserved during sleep: %v", err)
+	}
+	if _, err = otherOperator.PatchVolumes(t.Context(), sandbox.VolumePatch{Remove: []string{"home"}}); err != nil {
+		t.Fatal(err)
+	}
+	beforeWake := st.eng.volumeOpsLog()
+	if out, execErr := claim.Exec(t.Context(), "echo", "awake"); execErr != nil || out != "awake\n" {
+		t.Fatalf("implicit wake: %q %v", out, execErr)
+	}
+	if hibernated(t, st, claim.ID) {
+		t.Fatal("guest request did not wake the original claim")
+	}
+	if ops := st.eng.volumeOpsLog(); !slices.Equal(ops, beforeWake) {
+		t.Fatalf("wake unexpectedly operated on external disks: %v -> %v", beforeWake, ops)
+	}
+	view, err = claim.GetVolumes(t.Context())
+	if err != nil || len(view.Volumes) != 0 || len(view.VolumesWant) != 0 || len(view.States) != 0 || view.Converging {
+		t.Fatalf("wake unexpectedly restored volume references: %+v %v", view, err)
+	}
+	view, err = operator.PatchVolumes(t.Context(), sandbox.VolumePatch{Add: []sandbox.Volume{volume}})
+	if err != nil || !slices.Equal(view.Volumes, []sandbox.Volume{volume}) || view.Converging {
+		t.Fatalf("controller reattachment: %+v %v", view, err)
+	}
+	if status, body := rawJSON(t, st, http.MethodDelete, "/v1/volumes/home", ""); status != http.StatusConflict {
+		t.Fatalf("unregister attached restored disk: %d %s", status, body)
+	}
+}

sandboxd/pool/archive.go

@@ -14,6 +14,7 @@ import (
 
 	"github.com/cocoonstack/sandbox/sandboxd/store"
 	"github.com/cocoonstack/sandbox/sandboxd/types"
+	"github.com/cocoonstack/sandbox/sandboxd/utils"
 )
 
 const archiveDeleteDir = "archive-deletes"
@@ -315,7 +316,7 @@ func (m *Manager) markArchiveCks(ckIDs []string) (failed map[string]error) {
 	if len(written) == 0 {
 		return failed
 	}
-	if err := syncDir(dir); err != nil {
+	if err := utils.SyncDir(dir); err != nil {
 		for _, id := range written {
 			failed[id] = err
 		}