[Enrique Saurez (esaurez)]

Part of #1111 (SL3.9: one exec runs at a time per sandbox, across CLI processes).

What and why

Concurrent nvx-rs processes could run two sandbox exec workloads in one sandbox at once (#819). This PR does SL3.9's Scope and no more:

• host::cli::exec_lock (new) owns one OS file lock per sandbox, exec.lock in its state directory. Under the lifecycle lock, after the configuration check, an exec opens it as the store opens its own lock file (no link followed, a private regular file), tries the lock without waiting, and checks that the directory entry still names the locked file. The exec holds it through connect and Session::exec; the lifecycle lock is released before connecting, as before.
• sandbox exec --wait SECONDS (60 by default): an exec that finds the lock held releases the lifecycle lock, pauses for the store's 25 ms retry interval and tries again — once at least, and only until --wait has passed, even when a retry waited past it for the lifecycle lock (--timeout still bounds that wait) — then fails with sandbox-busy.
• sandbox stop takes no exec lock, and proceeds while a workload runs.
• The lock file stays, as sandbox.lock does, so that no release unlinks a file that another exec has open; deprovision removes it, also when a Windows file system without POSIX deletion keeps it delete-pending after the VM exited. It is the exec lock module's own file, not a StateFile, so the store's temporary-name rules and snapshot capture's owned files do not change.
• Docs: doc/usage.md (--wait, sandbox-busy), doc/run.md (lifecycle rules, sandbox-lifecycle acceptance), nvx-rs/README.md (state files, exec rules, error codes), and ErrorCode::SandboxBusy's rustdoc. The sandbox acceptance (scripts/nvx_tools/sandbox_lifecycle_tests.py) expects exec.lock from a sandbox's first exec until deprovision.

Not in this PR (per #1111): a queue, daemon or fairness; more than one exec per sandbox or a global lock; stop taking the lock; other flags.

Takeover. Enrique Saurez (@esaurez)'s Draft (9 commits on an earlier head of #1460) was replayed onto feature/nvx-refactor, now that #1460 (SL3.5b) has merged, and squashed into one commit that keeps his authorship, with: the deadline recheck and its regression test (review thread), the --wait docs (review thread), the acceptance update, which the earlier commits lacked (the public sandbox scenarios check exact state listings), and exec.lock as the module's own file instead of StateFile::ExecLock, which had made snapshot capture's recovery treat a foreign exec.lock as its own.

Rebase. Rebased onto feature/nvx-refactor at a53a2626, after #1476, #1477 and #1466 merged, as 1090c512. Two conflicts were resolved without changing what this PR does. In sandbox/options.rs, OPTIONS becomes [Spec; 37], since #1466 added --layer-format and --ram-upper-percent. In sandbox/lifecycle/start.rs's imports, #1466 dropped the activation, configuration and plan imports that moved with vm_config into sandbox/state.rs, next to the Deadline import that this PR extends with RETRY_INTERVAL and ExecLock. git range-diff against the previous head 120f2b0c shows only these hunks and changed context, and the diff stays 21 files, 911 insertions and 37 deletions.

Size. 82 core lines (estimate 60, stop 120). 6 counted files: host/cli/exec_lock.rs (new, 39), sandbox/lifecycle/start.rs (25), sandbox/lock.rs (7), sandbox/options.rs (7), cli/mod.rs (1), sandbox/mod.rs (1). Not counted: store/locked.rs, which only makes open_lock crate-private (1 line), store/mod.rs, with the re-export that this forces and one doc line, and error.rs, rustdoc only. BOOT_CONTRACT_VERSION, CONTROL_CONTRACT_REVISION and the public API are unchanged.

Validation

Check	Result
cargo fmt --check; Clippy -D warnings: --all-targets --all-features, --lib, --lib --no-default-features, --lib --bins --features cli, --lib --bins --no-default-features --features testing	Passed on Windows and on Linux (nvx-kvm-test)
cargo test --all-features --test cli	Linux: 100 passed. Windows: the 6 exec_lock cases pass; see the note below on load-sensitive timing tests
cargo test --all-features / cargo test / --lib --no-default-features	Linux: every integration target passes; the lib's 28 guest:: failures are the same set at the base on that host (isolated child processes of that environment). Windows: see note
cargo doc with -D warnings, MSRV 1.89 check, macOS check, python3 scripts/public_api.py, runtime adapter protocol-only and host-only fmt, Clippy, tests and MSRV	Passed (Linux; doc and MSRV also on Windows)
New regression exec_lock::a_retry_that_waits_for_the_lifecycle_lock_past_the_deadline_is_busy	Fails without the deadline recheck (the exec ran and exited 0), passes with it; passed in 6 repeated runs on Windows and 5 on Linux
check_test_layout.py, check_markdown_links.py, Ruff check and format, Pyright (Linux and Windows)	Passed
python -m unittest scripts/test_microvm_tests.py -k PublicSandbox -k PublicManagedExec; scripts.test_nvx_tools test_usage_documents_every_option_of_nvx_rs with NVX_RS set	Passed
nvx.py test-microvm --scenario managed-exec-config / sandbox-lifecycle / sandbox-proxy-only / sandbox-ram-upper, KVM, nvx-kvm-test (Azure Linux 3, virtual machine)	All passed
Same four scenarios, WHP, Windows 11 developer machine	All passed

The scenarios used the guest artifacts that #1460's CI built (run 42979403236), OpenVMM 14dc4116 from CI, and nvx-rs built from this commit.

At the rebased head 1090c512, on the Windows developer machine unless noted:

• cargo fmt --check, Clippy -D warnings in the five configurations above and for x86_64-unknown-linux-musl (--all-targets --all-features; --lib --bins --features cli), cargo +1.89 check --all-targets --all-features, cargo doc with -D warnings, public_api.py, check_test_layout.py and check_markdown_links.py: passed.
• cargo test --all-features: in the full run, one case of the lib and one of cli failed under load. The cli case was stop_forces_the_verified_vm_after_its_grace_period_even_after_an_acknowledgement (still-running), a stop path that this PR does not change; the truncated log lost the lib case's name. Rerun, --lib passed (1460 passed, 17 ignored), --test cli passed (93 passed, 2 ignored), and the stop case passed 3 times of 3 alone.
• scripts.test_nvx_tools.NvxRsTests with NVX_RS set to a release nvx-rs built from this head: 22 run, 1 skipped (it needs a Linux pseudo-terminal), including test_usage_documents_every_option_of_nvx_rs and test_usage_documents_every_error_code_of_nvx_rs.
• WSL Ubuntu: python3 -m unittest scripts/test_microvm_tests.py -k PublicSandbox -k PublicManagedExec: 29 passed.
• Ruff check and format, and Pyright for Linux and for Windows: passed.
• check_test_changes.py --base origin/feature/nvx-refactor: 16 changes and 33 added tests, as below.

The microVM scenarios above were not repeated locally after the rebase; CI runs them.

How the deadline regression synchronizes. The race that it checks needs an exec whose retry waits for the lifecycle lock until after its --wait deadline, while the exec lock becomes free. The test holds the exec lock itself and starts an exec with --wait 4. That exec holds the lifecycle lock only while it tries the exec lock, so the test polls the lifecycle lock until it finds one of these tries, and then takes the lock. When it took it within the first half of --wait, the exec's deadline, which lies at least --wait after the spawn, is still at least 2 s away, so the exec sleeps one 25 ms retry interval and tries again, and that try waits for the lifecycle lock: this is the precondition. Such a round then has no way out but its assertions: it holds the lifecycle lock until the deadline has surely passed, fails unless the exec is still waiting, frees the exec lock, releases the lifecycle lock, and requires sandbox-busy with the workload not run. Rounds repeat only to reach the precondition: a round that misses every try, or finds one too late, releases the lifecycle lock and requires the sandbox-busy of an exec that waited out --wait before another starts, and the test fails if 5 rounds never reach it. No outcome is retried.

Note: on the loaded Windows developer machine, a few timing-sensitive tests in modules that this PR does not touch failed under load and passed alone (one_shot::a_ready_console_stream_is_not_throttled_by_idle_sleeps, host::template::clone::tests::restore::the_restore_to_ready_deadline_runs_from_the_launch_through_every_step, host::platform::windows::tests::children_inherit_only_their_standard_handles); host::cli::snapshots::capture::tests::a_capture_ack_and_successful_output_without_timely_exit_are_stopped_but_never_finished fails the same way at the base in the same batch. The new exec_processes_contend_but_other_sandboxes_and_stop_proceed bounds its refused exec and its stop by 30 s, which still rules out the 60-second default and the minute-long workload, since a refused exec took up to 7 s there under load.

Left to CI: MSHV, the guest-binary checks (no guest code changes), the adapter's guest-only configuration, and the remaining scripts/test_*.py modules.

Changes to existing tests (tests-modified)

check_test_changes.py --base origin/feature/nvx-refactor: 16 changes, 33 added tests, none relocated. Every other test of the area is kept.

• cli/lifecycle: check_lifecycle, lifecycle_of_a_sandbox_whose_upper_is_in_ram, stops_keep_an_outcome_report_that_the_caller_reads_whatever_its_umask, managed_execs_exit_with_the_workload_status — their exact state listings include exec.lock after the sandbox's first exec (with_exec). refusals_of_a_running_sandbox — a refused exec of a stopped sandbox leaves exec.lock, since admission creates it before it finds no VM; the listing after stop is unchanged.
• cli: a_restarted_cli_drives_only_a_vm_of_its_runtime_identity, a_sandbox_whose_openvmm_exited_reports_it_once_and_starts_again — their exact listings include the exec.lock of their earlier execs.
• cli/usage: HELP — sandbox exec --help lists --wait WAIT.
• host::cli::sandbox::lock::tests: removals_refuse_live_records_and_keep_foreign_files — deprovision also removes the owned exec.lock and keeps exec.lock.foreign, .exec.lock.42.tmp, .exec.lock.x.tmp and .exec.lock.42.tmp.more as foreign; removal_cleans_owned_temporaries_without_touching_a_neighbour — exec.lock is among the owned files that removal deletes.
• scripts/nvx_tools/sandbox_lifecycle_tests.py: LifecycleAcceptance.__init__, invoke, require_entries — the acceptance expects exec.lock in a state directory from an exec of the provisioned sandbox until deprovision removes the configuration, in the lifecycle, proxy-only and RAM-upper scenarios.
• scripts/test_microvm_tests.py: _FakeSandboxCli._dispatch, _deprovision — the fake CLI creates exec.lock for an exec of a provisioned sandbox and deprovision removes it; test_acceptance_detects_lifecycle_regressions — the new exec-skips-lock fault must fail the acceptance.

[github-actions]

This pull request changes existing tests

It modifies or removes 16 existing test asset(s), listed below. Review each with care: a change to a test can hide a regression that the test caught. See doc/contribute.md#changes-to-existing-tests.

Test code

Change	What	Before	After
modified helper	fn check_lifecycle in cli/lifecycle (tests)	nvx-rs/tests/cli/lifecycle.rs:548	nvx-rs/tests/cli/lifecycle.rs:554
modified helper	fn refusals_of_a_running_sandbox in cli/lifecycle (tests)	nvx-rs/tests/cli/lifecycle.rs:1117	nvx-rs/tests/cli/lifecycle.rs:1133
modified helper	const HELP in cli/usage (tests)	nvx-rs/tests/cli/usage.rs:675	nvx-rs/tests/cli/usage.rs:675
modified helper	def _FakeSandboxCli._dispatch in scripts/test_microvm_tests.py	scripts/test_microvm_tests.py:1006	scripts/test_microvm_tests.py:1006
modified helper	def _FakeSandboxCli._deprovision in scripts/test_microvm_tests.py	scripts/test_microvm_tests.py:1393	scripts/test_microvm_tests.py:1401
modified test	fn removals_refuse_live_records_and_keep_foreign_files in host::cli::sandbox::lock::tests	nvx-rs/src/host/cli/sandbox/lock/tests.rs:100	nvx-rs/src/host/cli/sandbox/lock/tests.rs:100
modified test	fn removal_cleans_owned_temporaries_without_touching_a_neighbour in host::cli::sandbox::lock::tests	nvx-rs/src/host/cli/sandbox/lock/tests.rs:131	nvx-rs/src/host/cli/sandbox/lock/tests.rs:145
modified test	fn a_restarted_cli_drives_only_a_vm_of_its_runtime_identity in cli (tests)	nvx-rs/tests/cli.rs:830	nvx-rs/tests/cli.rs:833
modified test	fn a_sandbox_whose_openvmm_exited_reports_it_once_and_starts_again in cli (tests)	nvx-rs/tests/cli.rs:1189	nvx-rs/tests/cli.rs:1192
modified test	fn lifecycle_of_a_sandbox_whose_upper_is_in_ram in cli/lifecycle (tests)	nvx-rs/tests/cli/lifecycle.rs:633	nvx-rs/tests/cli/lifecycle.rs:644
modified test	fn stops_keep_an_outcome_report_that_the_caller_reads_whatever_its_umask in cli/lifecycle (tests)	nvx-rs/tests/cli/lifecycle.rs:715	nvx-rs/tests/cli/lifecycle.rs:731
modified test	fn managed_execs_exit_with_the_workload_status in cli/lifecycle (tests)	nvx-rs/tests/cli/lifecycle.rs:1023	nvx-rs/tests/cli/lifecycle.rs:1039
modified test	def PublicSandboxLifecycleTests.test_acceptance_detects_lifecycle_regressions in scripts/test_microvm_tests.py	scripts/test_microvm_tests.py:1680	scripts/test_microvm_tests.py:1689

End-to-end tests

Change	What	Before	After
modified helper	def LifecycleAcceptance.__init__ in scripts/nvx_tools/sandbox_lifecycle_tests.py	scripts/nvx_tools/sandbox_lifecycle_tests.py:411	scripts/nvx_tools/sandbox_lifecycle_tests.py:414
modified helper	def LifecycleAcceptance.invoke in scripts/nvx_tools/sandbox_lifecycle_tests.py	scripts/nvx_tools/sandbox_lifecycle_tests.py:486	scripts/nvx_tools/sandbox_lifecycle_tests.py:491
modified helper	def LifecycleAcceptance.require_entries in scripts/nvx_tools/sandbox_lifecycle_tests.py	scripts/nvx_tools/sandbox_lifecycle_tests.py:531	scripts/nvx_tools/sandbox_lifecycle_tests.py:543

Relocated unchanged: 0; added: 33. Compared the merge base a53a2626d6c2 with 1090c51251b2.

[Copilot]

🟡 Changes recommended

A retry can launch a workload after the --wait admission deadline expires, and missing --wait documentation causes the existing usage validation to fail when NVX_RS is set.

2 open issues

• Recheck admission deadline after reacquiring the lifecycle lock · New
• Document the --wait option in the sandbox usage section · New

What changed in this PR

Implements SL3.9’s cross-process serialization of sandbox exec using a persistent per-sandbox file lock, while allowing Stop to proceed independently.

Changes:

• Adds exec admission locking and a --wait option with a 60-second default.
• Integrates the persistent lock into state ownership and deprovision cleanup, including Windows delete-pending handling.
• Adds admission and cleanup tests and updates existing file-inventory expectations.

File	Description
nvx-rs/​tests/​cli/​usage.rs	Checks that sandbox help includes --wait.
nvx-rs/​tests/​cli/​lifecycle.rs	Updates persistent-lock inventories and exposes a Windows test helper.
nvx-rs/​tests/​cli/​exec_lock.rs	Adds cross-process admission, release, Stop, and cleanup tests.
nvx-rs/​tests/​cli.rs	Registers admission tests and updates state inventories.
nvx-rs/​src/​host/​cli/​store/​names.rs	Registers exec.lock without accepting temporary lookalikes.
nvx-rs/​src/​host/​cli/​store/​mod.rs	Documents the lock and re-exports its opening helper.
nvx-rs/​src/​host/​cli/​store/​locked.rs	Makes the existing lock-opening helper reusable.
nvx-rs/​src/​host/​cli/​sandbox/​tests.rs	Tests --wait defaults and numeric validation.
nvx-rs/​src/​host/​cli/​sandbox/​options.rs	Defines, parses, and describes --wait.
nvx-rs/​src/​host/​cli/​sandbox/​mod.rs	Passes the admission timeout to managed exec.
nvx-rs/​src/​host/​cli/​sandbox/​lock/​tests.rs	Tests owned-lock cleanup and foreign-file preservation.
nvx-rs/​src/​host/​cli/​sandbox/​lock.rs	Handles confirmed delete-pending exec locks during cleanup.
nvx-rs/​src/​host/​cli/​sandbox/​lifecycle/​start.rs	Adds admission retries and holds the guard through execution.
nvx-rs/​src/​host/​cli/​mod.rs	Registers the exec-lock module.
nvx-rs/​src/​host/​cli/​exec_lock/​tests.rs	Tests lock identity, privacy, contention, and persistence.
nvx-rs/​src/​host/​cli/​exec_lock.rs	Implements the nonblocking OS lock and release guard.

_{Review effort: Balanced}

[Pedro Henrique Penna (ppenna)]

CI on 1090c512 (run 43050413053): every job that ran passed except Build Ubuntu and Azure Linux artifacts, which failed in attempts 1 to 3. Each time, the runner could not resolve cdimage.ubuntu.com while downloading Ubuntu Base 26.04.1 (urlopen error [Errno -3] Temporary failure in name resolution). The guest scenarios that need those artifacts were skipped.

• This PR changes nvx-rs sources that the guest input digest covers, so the Ubuntu and Azure Linux artifacts miss the cache (…-30b50ecc…) and have to be rebuilt.
• The Dockerfile, the guest-artifacts action and the download code are unchanged since this morning's runs, which downloaded the same archive (for example run 42985903207, at 08:19 UTC).
• Canonical reports a DNS major outage since 14:26 UTC (https://status.canonical.com).

The failed jobs will be rerun once DNS recovers, and this PR merges only on a green run.

[Pedro Henrique Penna (ppenna)]

Merged by rebase as ab7c9b5 on feature/nvx-refactor, directly on a53a2626. It is one commit with no merge commit, and its tree (6e536ee5) is the tree that CI tested at 1090c512.

• CI run 43050413053 passed in attempt 5. Attempts 1 to 4 failed only in Build Ubuntu and Azure Linux artifacts, because the runners could not resolve cdimage.ubuntu.com during Canonical's DNS outage. Attempt 5 downloaded Ubuntu Base, built and cached the artifacts, and passed the KVM, MSHV and WHP guest and sandbox scenarios, the runtime adapter, the packages and the required status check.
• Copilot Setup Steps (run 43050390992) failed on the same outage (Temporary failure resolving 'azure.archive.ubuntu.com') and passed when rerun.
• The three cancelled checks belong to the runs that the ready event superseded for the same head, and their replacements passed.