Quiesce device I/O across suspend and snapshots

[abhishek-anand]

Summary

• quiesce block, network, and vsock workers at suspend and snapshot boundaries
• make worker pause/resume rollback explicit and fail closed when restoration is incomplete
• preserve healthy in-flight block I/O when a snapshot quiescence deadline expires
• keep suspend state transitions observable and fenced to the actual VMM state
• update the public API, generated SDKs, and operational documentation

Verification

• c8i real-KVM block I/O isolation and timeout-recovery integration tests
• Ubuntu OCI guest with Linux 6.6: API and CLI suspend/resume, network lifecycle, memory reclamation, preserved state, idempotence, and 20 rapid cycles
• Ubuntu OCI guest with Linux 5.10: the same API and CLI matrix
• guest security assertions verify that /dev/kvm and VMX/SVM are not exposed
• lifecycle model gates: 689 operations on Linux 6.6 and 693 operations on Linux 5.10
• local orchestrator suite: 383 tests
• local VMM core/device suites, formatting, clippy, shell syntax, and shellcheck

The rebased PR tree is ba0cdeba1daf04a117fbd28bd4453eff5c86170a, identical to the c8i-qualified source tree.

[Copilot]

Pull request overview

This PR tightens VM lifecycle boundaries (pause/suspend/snapshot/resume) by making block, network, and vsock worker quiescence explicit, bounded, and fail-closed. It also updates the orchestrator reconciliation path plus public API/SDK/docs to reflect the new semantics and error surfaces.

Changes:

• Add worker pause/resume acknowledgement + 5s quiescence timeouts, and mark devices failed/unsnapshotable on worker startup/exit/quiesce failures (net/vsock; blk timeout preserves in-flight I/O).
• Make snapshot/suspend/pause/resume state transitions rollback-aware and “fence paused” when rollback/resume can’t be confirmed.
• Update OpenAPI + generated TypeScript/Python SDKs and operational documentation; expand e2e coverage for suspend/resume and network lifecycle.

Reviewed changes

Copilot reviewed 25 out of 26 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
vmm/crates/vmm-integration/blk_io_isolation.rs	Integration coverage for blk quiescence and snapshot timeout behavior.
vmm/crates/vmm-devices/src/virtio/vsock.rs	Expose worker-failure hook; add unit test ensuring failed vsock is unsnapshotable.
vmm/crates/vmm-devices/src/virtio/vsock_io_loop.rs	Add quiesce/resume acks + bounded timeouts; validate kick fd; startup readiness handshake; tests.
vmm/crates/vmm-devices/src/virtio/net_transport.rs	Expose worker-failure hook; add unit test ensuring failed net is unsnapshotable.
vmm/crates/vmm-devices/src/virtio/net_io_loop.rs	Add quiesce/resume acks + bounded timeouts; startup readiness handshake; improved startup error propagation; tests.
vmm/crates/vmm-devices/src/virtio/blk_transport.rs	Track per-device delayed-service counters for failpoint-based tests.
vmm/crates/vmm-devices/src/virtio/blk_io_loop.rs	Add quiesce/resume acks + bounded timeouts; validate kick fd; startup readiness handshake; tests.
vmm/crates/vmm-devices/src/virtio/blk_backend.rs	Replace global delayed-service counter with per-backend/device counter (test-failpoints).
vmm/crates/vmm-core/src/live_snapshot.rs	Make I/O quiesce fallible; enforce fail-closed release ordering (I/O first, then vCPUs) with explicit rollback semantics.
vmm/crates/vmm-core/src/error.rs	Add structured IoQuiescence error with “vCPUs may resume” flag.
vmm/crates/vmm-core/src/controller.rs	Wire new quiesce semantics through pause/suspend/snapshot/live-snapshot; change resume ordering; fence paused when rollback can’t be confirmed; treat net/vsock worker setup as creation-fatal.
sdk/typescript/src/generated/schema.ts	Update generated schema docs/status codes to match new lifecycle semantics (resume/restore/snapshot/etc).
sdk/python/tarit_sdk/api/default/suspend_vm.py	Update generated docstring to reflect new suspend semantics.
sdk/python/tarit_sdk/api/default/snapshot_vm.py	Update generated docstring to reflect paused-vs-running snapshot semantics.
sdk/python/tarit_sdk/api/default/resume_vm.py	Add parsing for new 429/503 responses; update generated docstring.
sdk/python/tarit_sdk/api/default/restore_vm.py	Add parsing for new 409/503 responses; update generated docstring.
sdk/python/tarit_sdk/api/default/pause_vm.py	Update generated docstring (“Pause a VM”).
sdk/python/tarit_sdk/api/default/hibernate_vm.py	Update generated docstring (“Hibernate a VM…”).
README.md	Document worker quiesce ordering across suspend/resume.
PRODUCTION_READINESS.md	Update operational semantics for suspend vs pause and resume ordering.
orch/tests/e2e_suspend_resume.sh	Expand e2e suspend/resume gating: optional net checks, stronger diagnostics/cleanup, rapid-cycle coverage, client selection.
orch/openapi.yaml	Update endpoint summaries/descriptions and response codes (resume/restore/snapshot/etc).
orch/docs/API.md	Update API docs: expanded lifecycle states, quiesce/fencing semantics, opaque snapshot handles, new status codes.
orch/crates/taritd/src/ops.rs	Generalize reconciliation for failed live operations; fence durable state to actual VMM state on failures.
orch/crates/taritd/src/api.rs	Update restore doc comment to match opaque-handle semantics.
CHANGELOG.md	Document new quiesce + fencing semantics and surfaced failure modes.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.