fix(sandbox): end runs that stay paused beyond a pause budget

[ttbombadil]

Root cause (verified)

• A paused run keeps its runner, its admission reservation and its sandbox container. SimulationTimeoutManager stops the runtime clock during a pause, and nothing bounded the paused time.
• docker pause freezes every process of the container via the cgroup freezer, including PID 1, the timeout --signal=KILL hard lifetime. The documented "7200 s wall clock, including paused time" therefore never fired for a paused container.
• The browser answers WebSocket pings automatically, and the client never stops a paused run by itself. A forgotten paused tab, or 5–8 subjects with 5 paused tabs each, held capacity indefinitely.

Proof on the old code:

• tests/integration/paused-run-budget.test.ts (real Docker): the container stayed Paused and the run never ended; the test timed out.
• tests/server/services/sandbox-runner-pause-budget.test.ts: 2 of 3 cases red.

Fix

• SIMULATION_MAX_PAUSED_SECONDS (default 600 s, min 10) bounds the total paused time of a run. A run that pauses and resumes repeatedly cannot extend its hold that way.
• Startup rejects a value that is not lower than SANDBOX_MAX_LIFETIME_SECONDS.
• SandboxRunner starts a budget timer on a committed pause and clears it on resume, stop and reset.
• When the budget runs out while paused, the runner ends the run through the same path as the runtime timeout:
  • output --- Simulation stopped: paused longer than N s ---;
  • terminationRequested and SIGKILL of the run process;
  • the process exit reaches the session as onExit, which releases runner and admission;
  • the container (also when frozen) is removed by docker rm -f;
  • the client receives simulation_status: stopped.
• Compose passes the value through.
• Docs adjusted to the actual semantics:
  • Pause/Resume SSOT: PAUSED → STOPPED is now also possible server-side, and the new section 7.2 describes the pause budget;
  • SECURITY.md, INSTALL_SERVER.md (incl. the variable table) and ARCHITECTURE.md: the hard lifetime applies to running sandboxes; paused ones are bounded by the backend.

Tests

• tests/integration/paused-run-budget.test.ts (integration-docker, real frozen container): after the pause budget the run ends with the notice and onExit, and the container is removed.
• tests/server/services/sandbox-runner-pause-budget.test.ts:
  • fake timers: termination exactly at the budget;
  • paused time adds up across pauses;
  • a stop while paused does not trigger the budget;
  • configuration validation.
• tests/server/routes/simulation-paused-run-end.test.ts: a paused session whose run ends releases admission and runner over the real WS route and reports stopped.

Existing tests are unchanged. npm run check, npm run test:unit, npm run check:docs, Knip, git diff --check, docker compose config and the full ./run-tests.sh (incl. toolchain, Docker, E2E, build, audit, Sonar) are green, also after rebasing onto #233. The pre-push gate is green as well.

Product decision documented in the SSOT: a default of 10 minutes total pause per run. The maximum run time is 5 minutes, so this is generous for debugging and still frees capacity reliably.

🤖 Generated with Claude Code