Parallel execs on one VM: completion blocks the control loop, no concurrency test; command pool as a possible later design (low priority)

[ebursztein]

Priority: low. This is a corner case: it matters only when clients send many commands at once to one VM, or hammer several VMs at the same time. It is not blocking anything. It came up while working on #230.

What is wrong today

A VM accepts any number of exec commands at once, and they really do run in parallel in the guest: each command gets its own connection, host reader thread and guest process group. Two concrete problems:

1. Completion blocks the control loop. A buffered command's completion runs inline in that VM's control loop (crates/capsem-process/src/vsock.rs, handle_guest_msg / ExecDone). It can wait up to EXEC_OUTPUT_DEPOSIT_TIMEOUT (5 s) plus a ledger write, and meanwhile acks, keep-alives, file operations and other commands' completions wait behind it. Streamed commands already complete in their own task (tokio::spawn). Buffered completion should do the same.
2. Untested. No test runs commands in parallel on one VM. tests/capsem-stress/test_rapid_exec.py runs its execs one after another, and the parallel tests use different VMs. Add one that runs N commands in parallel on one VM and checks each command's exact output, exit code and ledger body.

Context: what the system bounds, and what it does not

Nothing limits how many commands are in flight per VM, or across VMs. JobStore.active_execs (crates/capsem-process/src/job_store.rs) has no size limit, and the service has no per-VM exec semaphore; its only limit is max_concurrent_vms.

Each command's cost is capped, though. Since #230 a command in flight retains at most 10 MiB per output lane (EXEC_LEDGER_BODY_BYTES, asserted at compile time). A buffered command briefly adds its result copy (at most 10 MiB, MAX_EXEC_OUTPUT_BYTES), plus up to about 4 MiB of queued stdin. Host memory therefore grows with the number of concurrent noisy commands, but with no command growing without bound.

That is the same model docker exec uses: no queue and no pool in the daemon, a fixed cap on what is kept per exec (log drivers rotate), and cgroup limits on the container for runaway workloads. By that yardstick, the fixes above may be enough.

Potential solutions for concurrency itself, if it ever proves necessary

From simplest to most involved. Pick the simplest one that holds up under a real workload:

1. Nothing beyond the fixes above (the Docker model): rely on the per-command caps.
2. Max commands in flight per VM: one configurable number, and requests over it are rejected immediately with a clear "VM busy, retry" error (for example 429). A counter and an error, not a queue.
3. A command pool with a queue, designed as a component:
   • Where it runs: in-process (capsem-process already owns every command in flight) or a dedicated subprocess. Weigh isolation, crash behavior and confinement (see P1: Enforce cross-VM host isolation, extract the sandboxed proxy, add capsem proxy, and document the architecture #206) against simplicity.
   • N slots per VM, with up to N commands running truly in parallel.
   • A bounded wait queue, possibly spilled to disk, so waiting commands and stdin already sent do not hold host RAM.
   • A variable rate / host-wide admission, so several VMs hammered at once share capacity fairly, perhaps adapting to memory pressure.
   • Fail closed when saturated, with a busy error rather than unbounded growth. Keep Stabilize security correctness: policy updates, credential redaction, audit admission and preview revocation #229's bounded queue, cancellation and fail-closed admission.
   • Cancellation frees capacity immediately, whether the command is running or still queued.

Options 2 and 3 need a saturation test: over-capacity requests are refused or queued as designed, host memory stays under the stated bound, and cancellation frees the slot. Option 3 also needs a multi-VM test.

Related: a possible exec-id reuse hazard (unverified)

The guest agent caches the exit code of every exec id it has seen and replays it without running the command (crates/capsem-agent/src/control_reader.rs, exec_done). Service exec ids restart at 1 when the service process restarts. If a VM ever outlived a service restart, a new command could reuse an old id and get a stale exit code with no output. No code was found that re-adopts running VMs after a restart, so this may be unreachable. Confirm or rule it out as part of this work.