Skip to content

GA next-gen: sandbox pause/resume/fork, spawn contract + manager authz, cost + security hardening - #254

Closed
dshakes wants to merge 48 commits into
masterfrom
feat/ga-nextgen
Closed

dshakes wants to merge 48 commits into
masterfrom
feat/ga-nextgen

Conversation

@dshakes

@dshakes dshakes commented Oct 23, 2026

Copy link
Copy Markdown
Owner

GA next-gen: audit-driven hardening plus sandbox lifecycle. The full plan and findings table are in docs/roadmap/2026-10-nextgen-sandbox-and-harness.md.

Wave 1: correctness and security

  • Helm. JWT_SECRET is now wired from a secretKeyRef, and LANTERN_ENV defaults to production, so the prod guards are on. Added values-prod.yaml.
  • Spawn contract. AgentSpec limits, egress and secrets now reach the guest through spawn.json on the cert drive (ADR 0025). Workloads drop privileges to LANTERN_WORKLOAD_UID.
  • runtime-manager authz. Each call is checked as either a VM cert or the service token. ExecTool is tenant-scoped (proto field 8).
  • model-router. Service-token auth, fail-closed in prod.
  • API keys. A key can no longer grant itself scopes it doesn't have, and more mutating routes carry WithScope.
  • Startup guards. Slack, Twilio and claude-code settings are now checked at startup.
  • Scheduler. Quota slots are released, idempotency is tenant-scoped, and the run-path idempotency key now includes the attempt.
  • Harness. Tool and exec output is capped. The exec bearer token is compared in constant time.

Cost

  • Workflow ai-step calls, confidence votes and web_search are now metered and budget-checked.
  • Prompt caching now covers the streaming and no-tools paths.
  • LLM idempotency keys include a body hash.
  • New tenant-wide budget cap at /v1/budgets/tenant.

Wave 2: pause, resume, fork

  • New endpoints:
    • POST /v1/runtime/vms/{id}/pause
    • POST /v1/runtime/vms/{id}/resume
    • POST /v1/runtime/vms/{id}/fork with body {"count":N}, N from 1 to 10
  • Exposed through the TS, Python and Go SDKs, the CLI, and the dashboard VM detail page.
  • Identity on restore.
    • A restored VM gets its identity from the wire vm_id and re-identifies itself.
    • Each incarnation gets its own exec token.
    • Each fork child gets a fresh agent identity.
    • A bare fork of a live source is refused (fail-closed).
  • Resume and snapshots. A resumed VM gets a grace window before heartbeat reconcile can terminate it. A snapshot no VM references any more is logged as orphaned; a DeleteSnapshot RPC is listed as a follow-up in ADR 0025.

Bridges

  • New LLM-judged commitments ledger. An entry closes only on verbatim evidence.
  • New encrypted sent-log and an alarm when the introspect audit goes stale.
  • New parity gate test between the two bridges.
  • Fixed the WhatsApp tests that broke when group replies became default-off.

Deps

  • Next 15.5.28 plus postcss and sharp overrides in web, docs and landing. npm audit --omit=dev reports 0.

Verification (run locally on the integrated branch)

  • Go: gofmt, vet and test pass in every module that uses gen/go (control-plane with DATABASE_URL). runtime-scheduler also passes with -race.
  • Rust: fmt, clippy -D warnings and test pass for runtime-manager (398), harness (186) and model-router.
  • TypeScript: tsc and tests pass for bridge-core, the WhatsApp bridge, the iMessage bridge and sdk-ts.
  • Python: sdk-python pytest passes.
  • Web and docs: apps/web tsc, eslint and build pass. The docs build with PAGES_BASE_PATH=/lantern passes.
  • Helm: helm template renders with the default, dev, staging and prod values.
  • Browser: checked against an isolated control-plane (HTTP_ADDR=:8090, its own DB, stub scheduler). On the dashboard VM detail page, Pause → Resume → Fork ×3 all completed, with no console errors.

Needs a human

  • Firecracker/KVM end-to-end on a Linux host. Re-identify and the guest ioctls are compiled and unit-tested only.
  • Live bridge validation in real logs after deploy.
  • helm install on a real cluster.
  • Deploy order: deploy the scheduler before or together with the control-plane. An older scheduler ignores the per-fork identities.

…ped before it existed

A freeform self-chat message landing within milliseconds of a draft being
staged was treated as the owner's deliberate replacement text and forwarded
verbatim to the draft's contact. Live incident: an owner message about an
unrelated SLA digest ("morning, those are auto-nudges not real tasks") landed
11ms after a draft was staged for contact Safiya and got queued to send to her
as-is — blocked only because the channel happened to be muted at that moment.

classifyPendingReply now requires >=3s between the draft's issuedAt and the
reply before falling through to "replace" (approve/reject are unaffected).
Shared in bridge-core, so both bridges get the fix from one change.
…; fail closed on Slack signing, Twilio auth off, claude-code in prod
…empotency keys, scheduler quota leak + tenant-scoped idempotency, SSE exec streaming
…N_WORKLOAD_UID

Every child ran as PID 1's root, so the SO_PEERCRED secrets check always
passed (workload uid == harness uid), the iptables REDIRECT (which exempts
the harness uid) did not apply to the workload, and tls.key was readable.
Spawn via CommandExt uid/gid (gid defaults to uid; std clears supplementary
groups). The secrets and OTLP sockets are chmod 0666 so the unprivileged
workload can still connect; SO_PEERCRED stays the authorization.
The kernel cmdline splits on whitespace and strips quotes, so argv like
'python main.py' and JSON egress lists arrived mangled. spawn.json
(workload_cmd, env, egress_rules, workload_uid/gid; all optional, unknown
fields ignored) now takes precedence over boot-args; typed fields beat
free-form env. Malformed file: fatal in prod, warn + cmdline in dev.

The cert material is copied into a root-only 0700 dir and the drive is
unmounted, so tls.key and spawn.json are unreachable by the workload uid
regardless of the image's file modes.
…d file tools

http_fetch buffered the whole body before checking the 1 MiB cap and
shell_exec used unbounded .output(). Both now stream with a bounded read
and truncate with a marker; shell_exec runs as the workload uid under the
timeout. New read_file / write_file / list_dir tools are confined to
LANTERN_WORKSPACE (default /workspace): canonicalize + prefix check, leaf
O_NOFOLLOW|O_NONBLOCK, hard links rejected, writes chowned to the workload.
…crypted sent-reply log; introspect 36h staleness alarm
…thropic streaming system prompts, per-request LLM idempotency keys
…x workflowEngine env nil in helm values; gofmt
…and tsc is enforced in CI; update stale group-reply tests
…start without a workload uid

Children inherited the harness's whole environment, including
LANTERN_HARNESS_EXEC_TOKEN, LANTERN_AGENT_INSTANCE_TOKEN and the TLS key
path. as_workload() now env_clear()s and passes the harness env minus the
LANTERN_* namespace (except a short non-secret allowlist: VM/run id, socket
paths, workspace, trace parent, LANTERN_WORKLOAD_*), then drops the uid.
Used by the supervisor, vm exec (tty and not) and shell_exec.

The prod 'LANTERN_WORKLOAD_UID missing' check ran in the secrets task,
racing the supervisor; parse_env now fails first. The secrets check stays.

fs tools open through openat2(RESOLVE_BENEATH|RESOLVE_NO_SYMLINKS) on a
root fd on Linux, closing the parent-dir swap window after the path check;
pre-5.6 kernels (ENOSYS) and non-Linux keep the plain open.
…t context

rust:1.85 cannot build the current Cargo.lock, build.rs needs protoc, and
'COPY . .' from services/harness could never reach ../../packages/proto.
Build from the repo root; a per-Dockerfile .dockerignore keeps the context
to packages/proto + services/harness.
…'s tenant

Adds field 8 so the runtime-manager can scope ExecTool to the VM's owning
tenant (ADR 0025). gen/go regenerated with the pinned protoc v6.33.4 +
protoc-gen-go v1.36.11 (byte-identical to the tracked stub before the change).
…xecTool/snapshots

- spawn.json (version/workload_cmd/env/egress_rules/workload_uid/gid) written
  at the cert-image root; cmdline kept for compat, manager-owned
  LANTERN_WORKLOAD_UID appended, caller-set value dropped
- RuntimeManager interceptor: VM certs (OU=lantern-vm) refused; shared
  x-lantern-service-token required (constant-time); prod refuses to start
  without LANTERN_GRPC_SERVICE_TOKEN, dev warns+allows
- ExecTool: tenant mismatch / empty-in-prod -> NotFound (vm_id and run_id paths)
- spawn reserves the wire vm_id before boot (concurrent duplicate -> Aborted)
- restore_snapshot_id restores through the post-gate backend with ownership
  check (NotFound cross-tenant/legacy/bogus id); non-restoring backend ->
  FailedPrecondition; bare firecracker resolves store:// ids
- snapshots record tenant_id + real agent_version_id; save streams artifacts
  (1 MiB buffer, streamed S3 multipart) instead of loading guest memory
- ExecTool harness call bounded by deadline + 1 MiB output cap; timeout capped
- warm pool cold-starts through the post-isolation-gate backend; keyed by tenant

Refs ADR 0025.
@dshakes dshakes closed this Oct 23, 2026
This was referenced Oct 23, 2026
@dshakes
dshakes deleted the feat/ga-nextgen branch October 28, 2026 03:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain:core Core library / business logic

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant