Quiesce device I/O across suspend and snapshots

CHANGELOG.md

@@ -62,6 +62,15 @@ versions may contain breaking changes.
 
 ### Fixed
 
+- Pause, suspend, and snapshot now require block, network, and vsock workers to
+  acknowledge quiescence before capturing state or releasing guest RAM. Resume
+  waits for every worker to leave its parked state before restarting vCPUs, so
+  rapid resume/suspend cycles cannot reuse a stale acknowledgement. Worker
+  startup, unexpected exit, and five-second quiescence failures are surfaced
+  instead of publishing a partially functional VM or snapshot. If a partial
+  vCPU or device-worker transition cannot be rolled back and acknowledged,
+  the VM is fenced paused and the orchestrator durably records that state
+  before returning the failure.
 - OCI image conversion now verifies manifest, config, and layer descriptors and
   streams layers through disk-size-derived expansion, entry-count, file-size,
   path-length, and layer-count limits before unpack. Rejected images publish no

PRODUCTION_READINESS.md

@@ -39,8 +39,9 @@ its scope.
   and PTY idle time are bounded. Invalid credentials pass through the outer
   admission limit.
 - Suspend is distinct from pause: it retains ownership and scheduler quota,
-  releases resident guest memory, and requires successful rehydration before
-  resume returns.
+  parks vCPUs plus block, network, and vsock workers at one boundary, releases
+  resident guest memory, and requires every worker to leave its parked state
+  before vCPUs restart and resume returns.
 - Hibernation releases the VMM and scheduler allocation. HTTP, PTY, SSH, and
   share ingress activate a hibernated VM through a single-flight restore gate;
   failed activation leaves a retryable hibernated record instead of a second

README.md

@@ -111,7 +111,9 @@ guest RAM; resume brings it back. Restore boots a fresh VMM from a snapshot.
 Snapshots include the architectural and host-advertised KVM paravirtual MSR
 state used by the guest. Restore validates that state against the destination
 before changing a vCPU and rejects incompatible hosts without a partial
-restore.
+restore. Suspend parks the block, network, and vsock workers after stopping the
+vCPUs; resume waits for those workers to leave the parked state before vCPUs
+run again.
 
 ```sh
 sudo vmm --socket /tmp/vm.sock snapshot              # full snapshot, prints the .snap path

orch/crates/taritd/src/api.rs

@@ -1889,9 +1889,8 @@ fn is_network_pool_exhausted(message: &str) -> bool {
     message.contains("network address pool exhausted")
 }
 
-/// Restore a snapshot into a running VM. Routes to the node that holds the
-/// snapshot file (`host_id`, as returned by the snapshot call) so no cross-node
-/// file transfer is needed; `None`/self restores locally.
+/// Restore a snapshot from its opaque public handle. The control plane resolves
+/// the private host and artifact locator, then routes the restore to that host.
 async fn restore_vm(
     State(state): State<AppState>,
     Extension(identity): Extension<ApiIdentity>,

orch/crates/taritd/src/ops.rs

@@ -152,7 +152,7 @@ async fn observe_and_compensate_vm_status(
     compensate_vm_status(state, prior, control_status(observed.state)?).await
 }
 
-async fn reconcile_snapshot_pause_failure(
+async fn reconcile_failed_live_operation(
     state: &AppState,
     prior: &VmRecord,
     primary: OrchError,
@@ -167,24 +167,24 @@ async fn reconcile_snapshot_pause_failure(
         Ok(Ok(status)) => match control_status(status.state) {
             Ok(status) => status,
             Err(error) => {
-                return retain_snapshot_reconciliation(
+                return retain_live_operation_reconciliation(
                     state,
                     prior,
                     primary,
-                    format!("snapshot pause reconciliation rejected VMM state: {error}"),
+                    format!("live-operation reconciliation rejected VMM state: {error}"),
                 );
             }
         },
         Ok(Err(error)) => {
-            return retain_snapshot_reconciliation(
+            return retain_live_operation_reconciliation(
                 state,
                 prior,
                 primary,
                 format!("VMM state could not be observed: {error}"),
             );
         }
         Err(error) => {
-            return retain_snapshot_reconciliation(
+            return retain_live_operation_reconciliation(
                 state,
                 prior,
                 primary,
@@ -211,11 +211,11 @@ async fn reconcile_snapshot_pause_failure(
                 ));
             }
             OrchError::Internal(format!(
-                "{primary}; VM was fenced {} after snapshot compensation",
+                "{primary}; VM was fenced {} after live-operation compensation",
                 observed.as_str()
             ))
         }
-        Err(compensation) => retain_snapshot_reconciliation(
+        Err(compensation) => retain_live_operation_reconciliation(
             state,
             prior,
             primary,
@@ -227,7 +227,7 @@ async fn reconcile_snapshot_pause_failure(
     }
 }
 
-fn retain_snapshot_reconciliation(
+fn retain_live_operation_reconciliation(
     state: &AppState,
     prior: &VmRecord,
     primary: OrchError,
@@ -2717,10 +2717,10 @@ async fn snapshot_local_locked(
     let mut bundle = match bundle {
         Ok(Ok(bundle)) => bundle,
         Ok(Err(error)) => {
-            return Err(reconcile_snapshot_pause_failure(state, &vm, error).await);
+            return Err(reconcile_failed_live_operation(state, &vm, error).await);
         }
         Err(error) => {
-            return Err(reconcile_snapshot_pause_failure(
+            return Err(reconcile_failed_live_operation(
                 state,
                 &vm,
                 OrchError::Internal(format!("snapshot task failed: {error}")),
@@ -3899,19 +3899,19 @@ where
         TransitionDecision::Apply => {}
     }
     let operation_supervisor = Arc::clone(&state.supervisor);
-    tokio::task::spawn_blocking(move || op(&operation_supervisor, id))
+    let operation = tokio::task::spawn_blocking(move || op(&operation_supervisor, id))
         .await
-        .map_err(|e| OrchError::Internal(format!("join: {e}")))?
-        .map_err(|error| {
-            tracing::warn!(
-                vm = %id,
-                from = current.status.as_str(),
-                to = new_status.as_str(),
-                %error,
-                "VMM lifecycle operation failed"
-            );
-            error
-        })?;
+        .map_err(|e| OrchError::Internal(format!("join: {e}")))?;
+    if let Err(error) = operation {
+        tracing::warn!(
+            vm = %id,
+            from = current.status.as_str(),
+            to = new_status.as_str(),
+            %error,
+            "VMM lifecycle operation failed"
+        );
+        return Err(reconcile_failed_live_operation(state, &current, error).await);
+    }
     match vm_set_status(state, id, new_status).await {
         Ok(record) => Ok(record),
         Err(persist_error) => {
@@ -4505,6 +4505,91 @@ mod tests {
         assert!(!socket.exists());
     }
 
+    #[cfg(target_os = "linux")]
+    #[test]
+    fn failed_pause_is_observed_and_fenced_to_the_actual_vmm_state() {
+        let (state, _) = test_state_with_durable_writer();
+        let id = insert_running_vm(&state);
+        let initial = vm_get(&state, id).unwrap();
+        state.store.lock().unwrap().insert_vm(&initial).unwrap();
+
+        let socket = PathBuf::from(format!(
+            "/tmp/taritd-pause-reconcile-{}-{id}.sock",
+            std::process::id()
+        ));
+        let _ = std::fs::remove_file(&socket);
+        let listener = UnixListener::bind(&socket).unwrap();
+        let (requests_tx, requests_rx) = std::sync::mpsc::channel();
+        let server = std::thread::spawn(move || loop {
+            let (mut stream, _) = listener.accept().unwrap();
+            let mut length = [0_u8; 4];
+            stream.read_exact(&mut length).unwrap();
+            let mut body = vec![0; u32::from_be_bytes(length) as usize];
+            stream.read_exact(&mut body).unwrap();
+            let request: tarit_vmm_client::ApiRequest = serde_json::from_slice(&body).unwrap();
+            let response = match &request {
+                tarit_vmm_client::ApiRequest::Pause => tarit_vmm_client::ApiResponse::Err {
+                    msg: "injected I/O quiescence failure".into(),
+                },
+                tarit_vmm_client::ApiRequest::Status => {
+                    tarit_vmm_client::ApiResponse::Status(tarit_vmm_client::VmStatus {
+                        state: tarit_vmm_client::VmState::Paused,
+                        uptime_ms: 1,
+                        vcpus: 1,
+                        mem_mib: 256,
+                        volumes: 0,
+                        nets: 0,
+                        kernel: "kernel".into(),
+                        vcpu_alive: true,
+                    })
+                }
+                _ => tarit_vmm_client::ApiResponse::Ok,
+            };
+            let encoded = serde_json::to_vec(&response).unwrap();
+            stream
+                .write_all(&(encoded.len() as u32).to_be_bytes())
+                .unwrap();
+            stream.write_all(&encoded).unwrap();
+            stream.flush().unwrap();
+            let stopped = matches!(request, tarit_vmm_client::ApiRequest::Stop);
+            requests_tx.send(request).unwrap();
+            if stopped {
+                break;
+            }
+        });
+        state
+            .supervisor
+            .install_test_control_runtime(id, socket.clone());
+
+        let error = test_runtime()
+            .block_on(pause_local(&state, id))
+            .expect_err("a failed pause must reconcile an actually paused VMM");
+        assert!(error.to_string().contains("fenced paused"));
+
+        let cached = vm_get(&state, id).unwrap();
+        let durable = state.store.lock().unwrap().get_vm(id).unwrap();
+        assert_eq!(cached.status, VmStatus::Paused);
+        assert_eq!(durable.status, VmStatus::Paused);
+        assert_eq!(cached.revision, initial.revision + 2);
+        assert_eq!(durable.revision, initial.revision + 2);
+
+        state.supervisor.stop_vm(id).unwrap();
+        server.join().unwrap();
+        let requests = requests_rx.into_iter().collect::<Vec<_>>();
+        assert!(
+            matches!(
+                requests.as_slice(),
+                [
+                    tarit_vmm_client::ApiRequest::Pause,
+                    tarit_vmm_client::ApiRequest::Status,
+                    tarit_vmm_client::ApiRequest::Stop
+                ]
+            ),
+            "unexpected VMM request sequence: {requests:?}"
+        );
+        assert!(!socket.exists());
+    }
+
     #[cfg(target_os = "linux")]
     #[test]
     fn failed_live_snapshot_is_observed_and_fenced_paused() {

orch/docs/API.md

@@ -40,7 +40,7 @@ admin keys can call admin-only routes such as `/v1/cluster`.
 ```json
 {
   "id": "uuid",
-  "status": "creating|running|paused|suspended|error",
+  "status": "creating|running|paused|suspended|hibernated|stopped|error",
   "revision": 3,
   "startup_path": "cold|warm|snapshot_restore",
   "memory_mib": 256,
@@ -408,26 +408,65 @@ Note: the local SQLite VM row is not deleted. On the owner, a later local `GET`
 
 ### `POST /v1/vms/{id}/pause`
 
-Resolve owner and pause the VM. The public handler does not require a JSON body.
+Resolve the owner, stop every vCPU, drain and park the block, network, and vsock
+workers, then publish the paused state. The public handler does not require a
+JSON body.
+
+If a vCPU or device-worker transition fails and the VMM cannot confirm a safe
+rollback, it fences the VM paused. The orchestrator observes and durably
+records that state before returning the operation failure, allowing a later
+explicit resume or delete instead of leaving the control plane marked running.
 
 Response `200`: updated `VmRecord` with `status: "paused"`.
 
-Status codes: `200`, `401`, `403`, `404`, `409` (VM is stopped), `500`.
+Status codes: `200`, `401`, `403`, `404`, `409` (invalid lifecycle
+transition), `500`.
+
+### `POST /v1/vms/{id}/suspend`
+
+Resolve the owner and capture a coherent in-process suspend image after every
+vCPU and guest-memory-writing device worker acknowledges quiescence. Resident
+guest RAM is released, but VM ownership, the VMM process, scheduler capacity,
+and tenant quota remain reserved.
+
+Response `200`: updated `VmRecord` with `status: "suspended"`.
+
+Status codes: `200`, `401`, `403`, `404`, `409` (invalid lifecycle
+transition), `500`.
+
+### `POST /v1/vms/{id}/hibernate`
+
+Capture and authenticate a live RAM, device, and private-disk artifact, satisfy
+the configured replication policy, stop the resident VMM, and release CPU,
+memory, cgroup, network, and scheduler capacity. The logical VM and its tenant
+ownership remain durable for later activation.
+
+Response `200`: updated `VmRecord` with `status: "hibernated"`.
+
+Status codes: `200`, `401`, `403`, `404`, `409` (VM is not running), `500`,
+`503` (durable artifact or peer lifecycle requirements are unavailable).
 
 ### `POST /v1/vms/{id}/resume`
 
-Resolve owner and resume a paused VM. The public handler does not require a JSON body.
+Resolve the owner and resume a paused or suspended VM. A hibernated VM is
+activated through the fenced single-flight restore path, including normal
+placement, artifact verification, network repair, and policy restoration.
+Device workers must leave their parked state before vCPUs restart, and the
+operation returns only after guest readiness succeeds. The public handler does
+not require a JSON body.
 
 Response `200`: updated `VmRecord` with `status: "running"`.
 
-Status codes: `200`, `401`, `403`, `404`, `409` (VM is stopped), `500`.
+Status codes: `200`, `401`, `403`, `404`, `409` (invalid lifecycle state),
+`429` (no placement capacity), `500`, `503` (owner or restore prerequisites are
+unavailable).
 
 ### `POST /v1/vms/{id}/snapshot`
 
-Resolve owner and ask the VMM to write a snapshot. Snapshot files are node-local.
-Only full snapshots are accepted. `diff: true` returns `422` before contacting
-the VMM because incremental snapshot headers contain parent paths; durable
-parent-chain relocation is not implemented yet.
+Resolve the owner and publish an authenticated snapshot behind an opaque UUID.
+Host paths, physical host identity, and artifact locators remain private. Only
+full snapshots are accepted. `diff: true` returns `422` before contacting the
+VMM because durable parent-chain relocation is not implemented yet.
 
 Request:
 
@@ -441,25 +480,25 @@ Response `200`:
 
 ```json
 {
-  "path": "/path/on/owner/snapshot",
-  "host_id": "node-a"
+  "snapshot_id": "uuid"
 }
 ```
 
-Always preserve `host_id`; pass it to `POST /v1/restore` so the restore routes to the node that has the file.
-
-Status codes: `200`, `401`, `403`, `404`, `409` (VM is stopped), `500`.
+Status codes: `200`, `401`, `403`, `404`, `409` (the lifecycle state does not
+support snapshots), `422` (`diff` is true), `500`.
 
 ### `POST /v1/restore`
 
-Restore a VM from a snapshot file. If `host_id` is present and is not the receiving node, the request is routed to that host. No snapshot bytes are copied between nodes.
+Restore a VM from an opaque tenant-owned snapshot handle. The control plane
+resolves its private locator, verifies the authenticated artifact, and routes
+the restore to the node that holds it. Clients cannot provide a host, path, or
+storage locator.
 
 Request:
 
 ```json
 {
-  "snapshot_path": "/path/on/snapshot-owner/snapshot",
-  "host_id": "node-a",
+  "snapshot_id": "uuid",
   "id": "optional new vm uuid"
 }
 ```
@@ -470,12 +509,14 @@ Status codes:
 
 | Status | Meaning |
 | --- | --- |
-| `201` | VM restored on the selected node. |
+| `201` | VM restored and ready. |
 | `401` | Missing or wrong `X-API-Key`. |
-| `403` | Tenant VM quota reached. |
-| `404` | `host_id` not found in the fleet. |
-| `429` | Selected node is at local capacity; includes `Retry-After` in seconds. Restore does not exhaustively try other nodes because the snapshot file is node-local. |
-| `500` | VMM restore, peer, or fleet failure. |
+| `403` | Tenant VM quota is reached. |
+| `404` | Snapshot handle not found or belongs to another tenant. |
+| `409` | Requested VM id already exists. |
+| `429` | The snapshot-owning node has no capacity; includes `Retry-After` in seconds. |
+| `500` | Internal restore failure. |
+| `503` | The snapshot-owning node is unhealthy, stale, or unavailable. |
 
 ### `POST /v1/execute`