P1: Enforce cross-VM host isolation, extract the sandboxed proxy, add capsem proxy, and document the architecture

[ebursztein]

Status and scheduling

P1 — unified sandboxing work for the 0.7 stabilization milestone. Ledger confinement, VM proxy/DNS extraction, standalone capsem proxy, and architecture documentation remain together in this issue. The steps below are internal implementation milestones, not separate product projects.

• PR feat(sdk): add typed gateway SDKs and npm MCP #199 merged on 2026-09-21 (914c72ede83c860656063b5a5e2424268a854379): typed SDKs and authenticated OCI/preview control are landed.
• Unify internal IPC and VSOCK codecs with bounded framing and serde_bytes #209's service-to-VM-owner IPC migration landed with feat(sdk): add typed gateway SDKs and npm MCP #199: named-field MessagePack (rmp_serde::to_vec_named), explicit big-endian length prefix, 16 MiB limit checked before receive allocation, and existing version negotiation. Reuse capsem-foundation::ipc_channel; do not build another codec.
• PR feat(networking): private networks for VM workloads, and OCI container run #200 landed (2026-09-15).
• PR Session ledgers: v3 archive publication, bounded memory, sparse events #227 lands and qualifies the archive protocol, one ledger writer and disk-backed readers. Build on its final merged format, not a parallel archive redesign.
• Snapshot removal Remove all automatic and manual snapshot features, including SDK surfaces #228 and security correctness Stabilize security correctness: policy updates, credential redaction, audit admission and preview revocation #229 land; do not restore snapshot product features here.
• Preserve the bounded readiness and exec-evidence contracts tracked in Finish bounded session ledgers: cheap readiness and complete capped exec evidence #230 when moving ledger ownership.
• Engine-owned session counters and one telemetry crate with OpenTelemetry #223 supplies committed session counters. Carry its counter snapshot into the ledger process rather than reimplementing statistics.
• Re-audit the resulting main branch and the relevant OCI/network lifecycle proof before moving ownership.

Roadmap: landed #199 → finish #227 → snapshot removal and correctness/counter stabilization → this unified #206 effort. #207/#209 describe control and codec work already carried by #199, not new migrations to schedule here.

#201 supplies the original MITM privilege finding; its proposed owner-side evaluator is superseded by the proxy ownership specified below. Preserve the authenticated control, browser isolation, policy and lifecycle behavior already landed with #199.

Internal implementation order

1. Define and enforce worker authority, trusted session/generation grants and broker authorization (section 1).
2. Confine all session-ledger reads and writes; settle stopped-session access, worker supervision and recovery (section 5).
3. Extract the existing VM proxy/DNS into the confined per-session engine (section 2).
4. Expose that same engine through standalone capsem proxy, without a VM (section 3).
5. Complete integration/platform proof and the architecture documentation (section 4); update affected contracts and security claims as each boundary changes.

Keep these as focused, reviewable commits and Sprinty items within one effort. The standalone endpoint depends on the confined engine and ledger; it is not a second proxy implementation. Keep the #227 implementation sprint separate.

Goal

Complete Capsem's host-side isolation so a compromised VM, or a host worker processing its traffic, cannot access another VM's files, credentials, control sockets, or unrelated networks.

Extract the existing MITM, security enforcement, logging, and telemetry pipeline into sandboxed proxy workers. Reuse that engine for both VM interception and a new capsem proxy command exposing an OpenAI-compatible API endpoint.

Deliver a clean architecture page for the rebuilt documentation site, describing the implemented system rather than the historical design. The host MCP package is already an SDK client following #199; preserve that ownership.

Problem and security contract

The source audit found explicit OS confinement around capsem-router switches and published-port relays, but incomplete host boundaries elsewhere:

• HTTP gateway: authenticated separate process, without application-installed OS confinement.
• VM owners: cleared inherited environments, but still the launching user's filesystem/socket authority.
• MITM and DNS: inside the process holding VM control and hypervisor resources.
• Documentation: 0700 directories and 0600 sockets incorrectly described as preventing sibling same-UID access.

Library separation, parent watching, authentication, and environment clearing are useful controls, not substitutes for an OS sandbox.

Assume malicious guest traffic and potential code execution inside a guest-facing host worker. Its resulting authority must be limited to its assigned VM, proxy session, or network. The coordinator and host kernel remain trusted. Intentional communication between members of the same network follows that network's contract. Standalone proxy clients are not themselves sandboxed.

A compromised worker must also be unable to trick the coordinator into acting for another session: broker IPC authorization is part of the containment boundary, not merely input validation.

Required implementation

1. Enforce process authority

Define and implement each process's permitted files, descriptors, listeners, upstream connections, IPC operations, credentials, and lifetime owner.

Process	Required authority
capsem-service	Trusted lifecycle coordinator and capability broker
HTTP gateway	API listener, necessary control/terminal channels, narrow runtime/log access
VM owner	One VM's virtualization resources, session, read-only assets, required IPC
Proxy worker	One VM or standalone proxy session's traffic, policy, credentials, telemetry
Session ledger process	One session's directory (session.db, its WAL/SHM sidecars, archive generation files and locks defined by the landed #227 protocol); its socket to granted producers and readers. No network, no exec, no entitlement, no other path
Network switch	One network's attachments and forwarding control
Published-port relay	Explicitly granted connections for its owning VM

• Apply appropriate confinement on macOS and Linux, preserving required VZ/KVM operations without unrelated session access.
• Clear worker environments and close unrelated inherited descriptors.
• Prefer scoped descriptor grants and narrow broker operations over access to entire runtime directories.
• Install confinement before untrusted input or readiness; fail startup, terminate, and reap on confinement failure.
• Bind capabilities to trusted session identity and generation. Never authorize from a worker-supplied VM ID, path, or network ID alone.
• Make cancellation close resources and join/reap owned tasks and children.
• Retain and verify existing confined switches/relays rather than replace them unnecessarily.

Gateway confinement needs its own policy: it must accept connections and use authorized service/terminal channels. Do not copy the descriptor-only router policy blindly. Preserve SDK gateway authentication; the coordinator must also enforce the authority of its control connection.

2. Extract one reusable proxy implementation

• Move MITM/TLS/HTTP processing, traffic rules/plugins, provider routing, credential integration, logging, and telemetry out of the VM owner.
• Run one confined proxy worker per VM; each standalone invocation gets an independent session/worker. Share implementation, not unrestricted authority across sessions.
• Keep the coordinator on configuration, scoped grants, and lifecycle. Do not relay every HTTP body, packet, or audit record through a central coordinator.
• Keep database execution/storage in capsem-logger, run inside the confined per-session ledger process (section 5). A worker's logging capability is a granted connection to its own session's ledger process, never a path. There is no global logging daemon: one ledger process per session. No route-owned SQLite access or projection caches.
• Relocate guest-facing DNS into the confined networking/proxy responsibility, preserving private-name resolution and upstream DNS semantics.
• Preserve rules/plugins, credential handling/redaction, streaming/cancellation/backpressure/timeouts, upstream errors, usage and session attribution, audit visibility/flushing, and VM boot/restart/stop/recovery.
• Remove superseded in-process handlers and all unconfined fallbacks.

Provision only the keys/certificates and credential capabilities required by that worker. The threat model must state that a compromised proxy holds its own session's granted authority, never another session's authority. Scoped upstream connection capability must not become arbitrary dialing.

3. Add capsem proxy

Expose the shared processing engine as an OpenAI-compatible API endpoint: configure an existing OpenAI SDK client's base_url to point at it.

• Use the existing profile/provider model to select upstream and security configuration.
• Configurable bind address and port; default loopback, OS-assigned port. Report usable base URL and proxy-session identity.
• Start without booting a VM.
• Preserve supported OpenAI-compatible surfaces, including Chat Completions and Responses where supported by the landed engine, streaming, and cancellation.
• Route only to configured provider authority; client headers/payloads cannot create an arbitrary-destination relay or forge trusted VM identities.
• Use existing scoped credentials and redaction.
• Keep the data listener separate from the authenticated SDK/control gateway.
• Stopping the command stops its listener/worker, cancels active requests, and completes bounded audit shutdown.
• Register the new command in the public-surface contract; update help, examples, and relevant schemas without unrelated API changes.

The proxy listener is unauthenticated by design. Remote binding is supported; deployment/network controls own access restriction. Do not introduce proxy authentication. SDK gateway authentication remains independent and mandatory.

No standalone forward-proxy mode, CONNECT, transparent host interception, or client CA installation. Transparent interception belongs inside Capsem VMs. This supersedes the previous forward-proxy/client-trust wording of #206.

Standalone mode controls and audits only model traffic sent through its endpoint. It cannot prevent an unrestricted agent from bypassing it or observe all host files, tool execution, or other network connections. State that limit accurately.

4. New documentation architecture page

• Write a canonical page for the rebuilt site based on the landed implementation, not the historical manual.
• Show actual processes separately from libraries/threads, and distinguish control, private-network, and proxied-traffic paths.
• Explain per-VM/per-network/global ownership and each sandbox's permitted and denied capabilities.
• Cover gateway, coordinator, VM owner, switch, relay, proxy, session ledger process, and MCP above the SDK.
• Show VM interception and standalone API access as adapters to the same processing engine.
• Explain policy, credentials, logging/telemetry, revocation, failure/recovery, OS differences, and verified limitations.
• Include a working SDK base_url example.
• Correct misleading same-UID isolation claims in developer references and replace/redirect contradictory pages through the new site's publication workflow.

Do not restore the old documentation site merely to publish this page. If the rebuild is not ready, retain reviewed content/diagrams as an explicitly pending deliverable and do not claim publication complete.

5. Confine the session ledger: one sandboxed process per session

Why this is needed. The ledger turns guest-controlled bytes (HTTP bodies, model traffic, tool payloads, file events) into SQLite writes. SQLite is C, and it is the only C library in the runtime today. Right now it runs inside the VM owner, which holds the virtualization entitlement and the user's full file authority. A memory-safety bug reachable from guest bytes there is a host compromise. Moving the proxy out (section 2) also makes a second producer of ledger records, while the ledger must keep exactly one writer (enforced by the writer lock since the ledger-archive work for #213). Both problems have the same answer: the ledger gets its own confined process, and the VM owner and proxy worker become clients of it.

Why C belongs only inside it. Pure-Rust compression tops out below what the ledger needs: measured on real ledgers, ruzstd 0.9 (and its master) implements only its fastest level with a fixed 128 KiB window and no dictionary encoder, and loses to our current pure-Rust deflate (7.0x vs 10.8x on an 80 MB ledger). The C zstd library at 1-2 MiB blocks reaches roughly 20-50x on the same data, more with a trained dictionary. We accept C code only where a compromise of it is contained: inside this process's Seatbelt profile on macOS and Landlock + seccomp on Linux, never in an unconfined host process. SQLite moves into the same boundary.

Required:

• At most one authoritative ledger process per session, with an explicit supervisor and parent-death behavior. Never one process for all sessions: a compromise must stay inside one session. Stopped-session reads must also use a confined ledger process, without booting a VM or falling back to service-side SQLite.
• It is the only process that executes SQLite or zstd against that session's data. Every read and write of that session's ledger, including retention and WARC export, goes through it. This is a session-ledger boundary, not an implicit migration of unrelated service databases.
• Readers query it over its socket and decode typed results in Rust. No host process outside it opens session.db with SQLite: a compromised ledger process could otherwise hand the service a crafted database file, moving the C attack surface to the reader side.
• macOS: deny-by-default Seatbelt, read/write only under that session's directory, no network, no process-exec, no Mach lookups beyond what SQLite needs (ideally none). Linux: Landlock restricted to the session directory plus a seccomp allowlist. Cleared environment, inherited descriptors closed, libraries loaded before confinement, temp storage in memory. Confinement failure fails startup.
• The session identity is stamped by the ledger process from its grant, never taken from a producer's message.
• Typed write/query messages use the landed named-field MessagePack framing and limits from feat(sdk): add typed gateway SDKs and npm MCP #199/Unify internal IPC and VSOCK codecs with bounded framing and serde_bytes #209. Keep operation types independent of transport; a future micro-VM is not an implementation requirement here.
• Refused audit admission still refuses the action: if the ledger process is gone or refuses a required record, the producer fails closed. Preserve A stalled ledger writer can freeze a capsem-process runtime worker; DNS ignores its admission result #225's admission/runtime-liveness fixes and Security ledger event_json.decision says allow on blocked HTTP and DNS requests #203's accurate recorded decisions; those are distinct contracts.

Archive and compression:

• Reuse the final Session ledgers: v3 archive publication, bounded memory, sparse events #227 generation/publication/recovery protocol, body indexes, checksums, open-block behavior and cross-block deduplication. Those are existing foundations, not features to implement again or call a new "format v2".
• Add zstd only inside the confined ledger process, with pinned/reviewed C dependencies and explicit codec identification using the landed format's extension contract. Preserve committed-data durability, bounded reads, retention and export semantics.
• Retain compressor history across durability flushes without weakening the archive-before-index ordering. Measure the zstd change against the landed archive baseline.
• Trained dictionaries remain optional. If used, define their durable identity and lifecycle with archive generations; they must not become a prerequisite for proving confinement.

Lifecycle contract — settle before implementation:

• Name the supervisor and granted access for live VMs, standalone proxy sessions and stopped persistent-session reads. Define acquisition/idle shutdown and transitions so two ledger owners cannot coexist.
• Specify the response to ledger/proxy death: active operations fail explicitly, required audit failure refuses the protected action, queues/resources stay bounded, and unrelated sessions continue.
• Define restart ownership, fresh grants/generations, committed versus merely accepted records, and replay/idempotency behavior. Do not blindly retry mutations or claim an accepted-but-uncommitted record survived.
• Preserve named-session stop/resume and historical reads without restarting the VM, bypassing confinement or opening its SQLite database in the coordinator.

Integrity contract:

• Keep producer-side hashing outside the ledger process, but specify exactly what it proves before implementing the chain: record identity/order, multiple producers, durable checkpoints outside the ledger's authority, acknowledgment boundaries, and restart/replay/retention rules.
• Verify alteration, substitution and omission against those trusted checkpoints. State the limits: a bare hash chain does not prove completeness, an unanchored tail may be lost, and a compromised producer can send false records. Do not claim detection beyond the implemented and tested contract.

Verification and acceptance

Effective confinement and cross-VM denial

Use test-owned fixtures to demonstrate behavior, not merely inspect configuration:

• Worker A cannot read/write B's session, obtain B's credentials, open B's control endpoints, or forge access by changing identifiers.
• A switch cannot access another network's attachments or VM control.
• Gateway/proxy workers cannot execute arbitrary processes, signal unrelated processes, open arbitrary host files, or establish ungranted outbound connections.
• Necessary operations still work under confinement on macOS and Linux.
• Deliberately failed confinement prevents readiness and traffic acceptance.
• Compromised-worker/broker requests cannot expand authority beyond trusted session/generation grants.
• Ledger process A cannot open, read or write session B's directory, archive or lock, and cannot open a socket or exec a process; a producer granted A's ledger cannot write rows attributed to B.
• No host process other than a session's ledger process opens that session's session.db with SQLite (citadel guard).

Functional integration

• Real VM/OCI workloads retain interception, policy, credentials, private networking, and telemetry behavior.
• Python and TypeScript OpenAI SDK clients work with hermetic upstreams through the standalone base URL.
• Verify supported streaming/non-streaming APIs, cancellation, tool-call payloads, usage, and upstream failures.
• Denied requests reach no upstream; exact audit facts identify session/provider/decision/outcome. Preserve existing fail-closed enforcement/audit-admission semantics.
• Standalone mode starts no VM and implements no CONNECT/forward-proxy route.
• Remote binding is unauthenticated as specified; SDK gateway authentication remains effective.
• Exercise the new CLI and documentation examples through real entrypoints.

Chaos and lifecycle

Use Greyjoy-style fault injection and retain earlier review regressions:

• Kill a proxy while its VM and another VM are active; effects remain scoped.
• Kill a ledger process mid-flush: its producers fail closed, the other sessions are unaffected, a restart reopens the ledger without losing committed rows or reusing its old grant, and the hash chain still verifies.
• Ledger process memory stays bounded by configured queues, the open block and unflushed rows, never by total ledger size. Use bounded fixtures for routine regression proof; any long soak is an explicit manual measurement, not a new per-change gate.
• Restart workers without reusing stale credentials/grants/session identity.
• Exercise disconnect/reconnect races and network retirement.
• Stall clients/upstreams, interrupt streams, and exhaust bounded queues.
• Repeat creation/destruction without accumulating processes, descriptors, tasks, or buffers.
• Verify bounded audit shutdown and failure handling without bypassing policy.

Verification cost

Use focused deterministic checks for each milestone, existing platform/VM fixtures for crossed boundaries, and the required final integration gates. Reuse fixtures and evidence rather than repeating the full gate per commit. No new multi-hour stress suite on every change. Keep the accepted abstract archive model under one second; do not translate its enumerated schedules into real VM runs.

Completion gates

• Required focused tests and platform gates pass on the final candidate.
• No security behavior regression, unconfined fallback, or duplicate proxy implementation.
• Architecture diagram/capability table match the actual implementation.
• Documentation/examples verified against the final SDK; publication status explicit.
• User-visible changes have a changelog entry.
• Remaining trust assumptions and standalone proxy limitations are explicit.

This issue is complete only when boundaries are enforced and tested—not when code is merely moved into subprocesses.

Related work

• feat(sdk): add typed gateway SDKs and npm MCP #199 (merged): SDK/MCP integration, authenticated OCI/preview control and bounded MessagePack IPC. Reuse the landed contracts.
• feat(networking): private networks for VM workloads, and OCI container run #200 (merged): OCI/private networks; preserve the confined switch/relay and verify lifecycle behavior.
• MITM TLS/HTTP inspection runs inside the privileged VM owner process #201: original MITM privilege finding, covered by this issue's accepted proxy ownership.
• Move OCI container control and port exposure behind one authenticated HTTP path; retire the CLI's direct VM-owner channel #207/Browser preview for exposures through the confined router #208/Unify internal IPC and VSOCK codecs with bounded framing and serde_bytes #209: OCI control, preview and IPC implementation carried by feat(sdk): add typed gateway SDKs and npm MCP #199; no duplicate migration here.
• Stabilize security correctness: policy updates, credential redaction, audit admission and preview revocation #229: consolidated security correctness, replacing Observed credentials are redacted in body previews but not in stored headers #218/Profile mutation routes race each other and write policy files non-atomically #202/Security ledger event_json.decision says allow on blocked HTTP and DNS requests #203/A stalled ledger writer can freeze a capsem-process runtime worker; DNS ignores its admission result #225 and residual security(preview): session expiry does not end admitted connections; no per-session revoke #222. Preserve redaction, atomic policy updates, exact decisions, bounded admission and selective revocation.
• Finish bounded session ledgers: cheap readiness and complete capped exec evidence #230: bounded ledger readiness and complete capped exec evidence, replacing ready() runs PRAGMA integrity_check on every route request #215/Guest exec output is cut to 1 KiB before the session ledger sees it #220.
• Session ledgers: v3 archive publication, bounded memory, sparse events #227/High RAM usage due to in-memory SQLite database #213: storage foundation and memory fix; reuse the final merged archive protocol.
• Engine-owned session counters and one telemetry crate with OpenTelemetry #223: committed counters and telemetry ownership.
• Remove all automatic and manual snapshot features, including SDK surfaces #228: complete product snapshot removal, including SDK surfaces. Normal persistence, forks and coherent ledger copies remain.

This description replaces the older forward-proxy/client-CA proposal. The only standalone mode in scope is the OpenAI-compatible API endpoint above; the unified sandboxing scope remains intact.

[ebursztein]

Handoff from #229 (PR #235). When #206 moves these owners into confined workers, the contracts below must move with them, and the items left open are assigned here.

Contracts #206 must preserve when it moves the owners

1. Policy reload acknowledgement is exact.
   • ServiceToProcess::ReloadConfig { id } is answered by ConfigReloadResult { id, active_profile_digest, error }.
   • The digest is capsem_core::net::policy_config::active_profile_digest of the exact bytes the worker applied.
   • The service counts a worker as updated only when that digest matches the bytes it wrote.
   • A load failure keeps the previous policy and reports the error. It must not close the connection.
   • A proxy worker that takes over policy must answer the same way.
2. Policy edits are serialized.
   • The service's PolicyMutation boundary spans load → save → cache refresh → materialize → worker acknowledgement.
   • Pushes to workers require it (push_profile_to_running_instances(state, &mutation, …)).
   • A new worker type must be reloaded inside the same boundary, not beside it.
3. Policy files are published atomically, through atomic_write_private (guard: tests/citadel/test_policy_publication_is_atomic.py). Grants of the active profile to a confined worker must keep read-whole-file semantics.
4. Stored headers and bodies are redacted in the telemetry hook before build_net_event. Every observed credential is removed from both header sets and both bodies. When MITM moves into the proxy worker, that redaction moves with it, and it must still come before ledger admission (guard: test_stored_headers_are_redacted.py).
5. A rule row's decision.effective is the outcome enforced.
   • Enforcing boundaries use the _with_decision emitters.
   • Record-only rows keep the event's decision as it arrived.
   • HTTP and DNS events are seeded from what the proxy enforced (Denied → block).
   • The extracted proxy must keep passing the enforced outcome into its ledger rows.
6. Fail-closed audit admission.
   • A DNS answer, or a file write into the guest, is released only when its primary audit record is accepted.
   • Admission waits at most SECURITY_ADMISSION_DEADLINE (via admit_within). A timeout counts as refusal.
   • Async code never records audit by blocking a runtime worker (guard: tests/citadel/test_process_audit_admission.py).
   • Once audit becomes a granted connection to the per-session ledger process, "ledger unreachable" must also count as refusal on these paths.

Fold into the proxy extraction

• Apply policy as one snapshot. capsem-process swaps its four policy handles one at a time (net_state.policy, security_rules, plugin_policy, model_endpoints), so during a reload a request can briefly see a mix of two revisions. The proxy worker rewrites these readers anyway: hold one immutable policy snapshot per applied digest.

Related

• Clippy output shared across worktrees, which gave false greens and reds while verifying this work: fixed on worktree-clippy-salt (84b4f0e), details on Harden typed gate composition identity and graph guard coverage #186.

[ebursztein]

Add: confine capsem-process to its own session

The #228 lane found that the guest's writable overlay (rootfs.img) sat inside the read-write VirtioFS share. The host attaches it by path, so a root guest could swap it for a symlink to any host file and get that file as a writable disk. This is fixed on the #228 branch: the image moved to a host-only system/ directory, legacy sessions are migrated with no-follow and a planted link is refused, and a citadel guard (test_system_overlay_outside_share.py) holds it.

That fix is the first line of defense. The second belongs here: capsem-process handles all guest I/O (VirtioFS, vsock, disk attachments) and runs unconfined, with the service user's full filesystem access. It should get the same treatment as capsem-router:

• What it may reach: read-write its session dir; read-only its hash-checked assets (kernel, initrd, base rootfs); its UDS sockets. Nothing else.
• macOS: Seatbelt, reusing the router_sandbox pattern. Enumerate the Mach services and entitlement Virtualization.framework needs, and prove them under sandbox-exec.
• Linux: Landlock for the path rules, plus a seccomp allowlist for the syscall surface. Seccomp can't restrict by path.
• When: applied after the session dir is opened and before the first guest I/O.
• Proof:
  • an adversarial open of a host file outside the session gets EPERM;
  • boot, suspend/resume and fork pass with the sandbox on, on both hypervisors;
  • a citadel guard checks the ordering.

This folds in #237, which I'm closing as a duplicate.

[ebursztein]

Requirement from #223 (metric export), for the process-authority work in section 1:

• capsem-process now exports its own facade metrics (ledger writer, MITM, DNS, security engine, virtio-blk) over OTLP/HTTP, tagged session.id. The code is in crates/capsem-process/src/metric_export.rs.
• It resolves the destination only from the corp config's open_telemetry endpoint, a file it already trusts. It never reads OTEL_EXPORTER_OTLP_*, because that environment can carry collector credentials and the spawn allowlist keeps it out. Authenticated collectors are out of scope for now.
• Once this process, or the proxy worker that takes over MITM/DNS, is confined, that endpoint should be an upstream granted at launch: the one telemetry destination the worker may dial, bound to its session grant, not something it resolves or dials for itself. A worker holding a telemetry grant must not be able to turn it into arbitrary dialing.
• Per this issue, session totals must not be relayed through the coordinator. The service exports them itself from the ledgers' counter snapshots, which the session ledger process should carry as-is.