[dshakes]

Part 7 of 11 of the GA next-gen stack (after #276 · #277 · #278 · #279 · #280). Reordered so the pause/resume/fork server lands before the SDK/dashboard that call it (the reviewer blocked the old order: clients calling routes that did not exist yet). The final tree is byte-identical to the previous stack tip. Supersedes #281–#286. Base: ga4/06-vm-identity. Merge in order with a merge commit.

Context, verification and needs-human list: #254 and docs/roadmap/2026-10-nextgen-sandbox-and-harness.md.

Commits in this part:

• merge: per-fork agent identity, resume reconcile grace, orphaned-snapshot logging

[github-actions]

🔎 Codex cross-audit (agent:audit)

Blocking

• Fork persists live agent-instance bearer tokens in scheduler state. In lifecycle.go, fork children are stored with Spec: withIdentity(...); withIdentity injects LANTERN_AGENT_INSTANCE_TOKEN into Spec.Env at lifecycle.go. The write-through store then marshals and persists the whole spec into sched_vms.spec at pg.go, and List also returns Spec directly at scheduler_grpc.go. That violates the repo invariant that secrets never appear in run state and creates a durable credential leak for every fork. Store a redacted spec in scheduler state/persistence and only pass the token in the manager SpawnRequest.

High

• Pause frees node capacity before the VM is actually stopped. Pause writes PAUSED before Stop at lifecycle.go, and UpdateVMState immediately decrements node occupancy/releases reservations for RUNNING -> PAUSED at state.go. While Stop is still in flight, other placement can consume capacity that is still physically occupied; if Stop fails, the code restores RUNNING but the capacity reservation has already been dropped. The state should not release node capacity until manager stop succeeds, or it needs an intermediate non-freeing state.

• Pause/Fork dial a default manager when the owning node is missing. Pause ignores the GetNode boolean and calls Snapshot with node.Address even if the node was not found at lifecycle.go. Fork does the same for running sources before checking nodeOK at lifecycle.go. With an empty address, the dialer can fall back to LANTERN_DEFAULT_MANAGER_ADDR, so a stale VM row can snapshot/stop/spawn against the wrong manager instead of failing closed.

Verification

• I could not run the Go tests in this read-only sandbox: Go failed creating the module cache under both /home/runner/go and /tmp.

[github-actions]

🛡️ Vuln scan — ❌ vulnerable dependency found

      #41: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.Shutdown
      #42: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.Shutdown
      #43: internal/telemetry/tracer.go:92:21: telemetry.InitTracer calls trace.TracerProvider.Shutdown
      #44: internal/handlers/runtime.go:282:50: handlers.ManagerDialOptions calls otelgrpc.NewClientHandler, which calls trace.TracerProvider.Tracer
      #45: internal/telemetry/tracer.go:79:23: telemetry.InitTracer calls trace.WithBatcher
      #46: internal/telemetry/tracer.go:80:24: telemetry.InitTracer calls trace.WithResource
      #47: internal/telemetry/tracer.go:81:23: telemetry.InitTracer calls trace.WithSampler
      #48: internal/handlers/llm_proxy.go:3935:104: handlers.LlmProxyHandler.Transcribe calls trace.errUnsupportedSampler.Error
      #49: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.init
      #50: internal/telemetry/tracer.go:17:2: telemetry.init calls trace.init
      #51: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.logDropped
      #52: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.logDropped
      #53: internal/handlers/llm_proxy.go:3810:27: handlers.LlmProxyHandler.OCR calls io.ReadAll, which eventually calls trace.nonRecordingSpan.AddEvent
      #54: internal/handlers/runtime.go:1652:2: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.End
      #55: internal/middleware/span.go:34:22: middleware.EnrichSpan calls trace.nonRecordingSpan.IsRecording
      #56: internal/handlers/runtime.go:1660:19: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.RecordError
      #57: internal/handlers/runtime.go:1651:20: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.SetAttributes
      #58: cmd/server/main.go:848:38: server.main calls http.Server.ListenAndServe, which eventually calls trace.nonRecordingSpan.SetName
      #59: internal/handlers/runtime.go:1661:17: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.SetStatus
      #60: internal/handlers/runtime.go:1650:27: handlers.RuntimeHandler.Schedule calls noop.Tracer.Start, which calls trace.nonRecordingSpan.SpanContext
      #61: internal/handlers/llm_proxy.go:3801:24: handlers.LlmProxyHandler.OCR calls http.Client.Do, which eventually calls trace.nonRecordingSpan.TracerProvider
      #62: internal/handlers/llm_proxy.go:3810:27: handlers.LlmProxyHandler.OCR calls io.ReadAll, which eventually calls trace.recordingSpan.AddEvent
      #63: internal/handlers/runtime.go:1652:2: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.End
      #64: internal/middleware/span.go:34:22: middleware.EnrichSpan calls trace.recordingSpan.IsRecording
      #65: internal/handlers/runtime.go:1660:19: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.RecordError
      #66: internal/handlers/runtime.go:1651:20: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.SetAttributes
      #67: cmd/server/main.go:848:38: server.main calls http.Server.ListenAndServe, which eventually calls trace.recordingSpan.SetName
      #68: internal/handlers/runtime.go:1661:17: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.SetStatus
      #69: internal/handlers/runtime.go:1650:27: handlers.RuntimeHandler.Schedule calls noop.Tracer.Start, which calls trace.recordingSpan.SpanContext
      #70: internal/handlers/llm_proxy.go:3801:24: handlers.LlmProxyHandler.OCR calls http.Client.Do, which eventually calls trace.recordingSpan.TracerProvider
      #71: internal/handlers/llm_proxy.go:3935:104: handlers.LlmProxyHandler.Transcribe calls trace.samplerArgParseError.Error
      #72: internal/handlers/connector_health.go:177:15: handlers.PersistConnectorQuarantine calls errors.As, which eventually calls trace.samplerArgParseError.Unwrap
      #73: internal/handlers/runtime.go:1650:27: handlers.RuntimeHandler.Schedule calls trace.tracer.Start
      #74: internal/telemetry/tracer.go:86:27: telemetry.InitTracer calls otel.SetTextMapPropagator, which eventually calls trace.tracerProviderConfig.MarshalLog
      #75: internal/telemetry/tracer.go:79:23: telemetry.InitTracer calls trace.WithBatcher, which eventually calls x.Feature[string].Enabled
      #76: internal/telemetry/tracer.go:16:2: telemetry.init calls resource.init, which calls x.init

Vulnerability #15: GO-2026-6443
    Server panic via missing authority or Host headers in google.golang.org/grpc
  More info: https://pkg.go.dev/vuln/GO-2026-6443
  Module: google.golang.org/grpc
    Found in: google.golang.org/grpc@v1.82.1
    Fixed in: google.golang.org/grpc@v1.82.2
    Example traces found:
      #1: cmd/server/main.go:262:32: server.main calls grpc.Server.Serve, which eventually calls transport.http2Server.HandleStreams

Vulnerability #16: GO-2026-6348
    Heap Memory Exhaustion (OOM) via HTTP/2 DATA Frame Fragmentation in
    google.golang.org/grpc
  More info: https://pkg.go.dev/vuln/GO-2026-6348
  Module: google.golang.org/grpc
    Found in: google.golang.org/grpc@v1.82.1
    Fixed in: google.golang.org/grpc@v1.83.1
    Example traces found:
      #1: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.BufferSlice.MaterializeToBuffer
      #2: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.IsBelowBufferPoolingThreshold
      #3: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.NewBuffer
      #4: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls mem.ReadAll
      #5: internal/handlers/llm_proxy.go:3935:104: handlers.LlmProxyHandler.Transcribe calls multierr.multiError.Error, which eventually calls mem.writer.Write
      #6: internal/handlers/runtime.go:2081:27: handlers.RuntimeHandler.StreamLogs calls grpc.GenericClientStream[github.com/dshakes/lantern/gen/go/lantern/v1.LogsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.RuntimeLogLine].Recv, which eventually calls transport.ClientStream.Close
      #7: internal/handlers/runtime.go:440:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.ClientStream.Header
      #8: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ClientStream.Read
      #9: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ClientStream.RecvCompress
      #10: internal/handlers/runtime.go:440:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.ClientStream.TrailersOnly
      #11: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.NewHTTP2Client
      #12: cmd/server/main.go:262:32: server.main calls grpc.Server.Serve, which eventually calls transport.NewServerTransport
      #13: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ServerStream.Read
      #14: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.Stream.ReadMessageHeader
      #15: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.http2Client.Close
      #16: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.http2Client.GracefulClose
      #17: internal/handlers/runtime.go:440:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.http2Client.NewStream
      #18: cmd/server/main.go:913:25: server.main calls grpc.Server.GracefulStop, which eventually calls transport.http2Server.Close
      #19: cmd/server/main.go:262:32: server.main calls grpc.Server.Serve, which eventually calls transport.http2Server.HandleStreams

Your code is affected by 16 vulnerabilities from 3 modules and the Go standard library.
This scan also found 3 vulnerabilities in packages you import and 19
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.
make: *** [Makefile:270: audit] Error 3

[github-actions]

🔷 Gemini cross-audit (agent:audit-gemini)

Independent Audit Report: Sandbox Lifecycle (Pause / Resume / Fork) & Manager Authz

1. Blocking / Critical Issues

[BLOCKING] 1. Unregistered Node Check Order & Snapshot Orphanage in Fork

Location: services/runtime-scheduler/internal/handlers/lifecycle.go (lines 223–250)

In SchedulerService.Fork:

node, nodeOK := s.Store.GetNode(src.NodeName)
var snapID string
switch src.State {
case lanternv1.VmState_VM_STATE_RUNNING:
    snap, err := s.Dialer.Snapshot(ctx, node.Address, &lanternv1.SnapshotRequest{VmId: srcID, KeepRunning: true})
    ...
default:
    ...
}
if !nodeOK {
    return nil, status.Errorf(codes.Unavailable, "fork %s: node %q holding the snapshot is not registered", srcID, src.NodeName)
}
if !s.Store.TryReserveTenantVMs(tenantID, count, s.TenantHardCap) {
    ...
}

Two bugs:

1. If src.NodeName is not registered (!nodeOK), node is a zero-value cluster.Node with node.Address == "". When src.State == RUNNING, s.Dialer.Snapshot(ctx, "", ...) is executed before if !nodeOK is checked. Dialing with "" falls back to LANTERN_DEFAULT_MANAGER_ADDR, targeting the wrong manager node with a snapshot request for a foreign VM.
2. Even when nodeOK is true, taking an expensive snapshot (disk freeze, artifact streaming to S3) occurs before TryReserveTenantVMs. If the tenant is over quota, the call fails with ResourceExhausted while the newly created snapshot is left orphaned on the node.

• Fix: Move both if !nodeOK and if !s.Store.TryReserveTenantVMs above the switch src.State block.

---

[BLOCKING] 2. Spurious Node Capacity Increment & Leaked State in Terminate for Paused VMs

Location: services/runtime-scheduler/internal/handlers/scheduler_grpc.go (lines 390–405) & cluster/state.go

In SchedulerService.Terminate:

paused := vm.State == lanternv1.VmState_VM_STATE_PAUSED
quotaAlreadyFreed := vm.State == lanternv1.VmState_VM_STATE_FAILED || paused

s.Store.UpdateVMState(req.VmId, lanternv1.VmState_VM_STATE_DRAINING, req.Reason, nil, time.Now().UTC())

In InMemoryStore.UpdateVMState:

else if v.State == lanternv1.VmState_VM_STATE_PAUSED && occupiesNode(state) {
    if n, exists := s.nodes[v.NodeName]; exists {
        n.RunningVms++
    }
    v.RespawnedAt = at
}

occupiesNode(DRAINING) is true (!isTerminal(DRAINING) && DRAINING != PAUSED). When a PAUSED VM is terminated, transitioning it to DRAINING:

1. Increments RunningVms on the node for a container that is not running.
2. Sets v.RespawnedAt to now on a VM being deleted.
3. If stopUnlessPaused or any downstream logic fails, the VM is left in DRAINING with an inflated RunningVms count on the node, skewing placement scoring.

• Fix: A paused VM holds no live execution and must not transition to DRAINING. Either transition directly from PAUSED to TERMINATED (or bypass DRAINING for paused == true), or update occupiesNode / UpdateVMState so PAUSED -> DRAINING does not increment RunningVms.

---

2. High Severity / Correctness & Security Findings

[HIGH] 3. Resume Cannot Refresh LANTERN_AGENT_INSTANCE_TOKEN (Expired Credentials Post-Pause)

Location: packages/proto/lantern/v1/runtime.proto & services/runtime-scheduler/internal/handlers/lifecycle.go

Per ADR 0025, restoring a VM requires a new guest identity. While Fork receives fresh identities minted by the control plane (req.Identities), ResumeRequest carries only vm_id:

message ResumeRequest {
  string vm_id = 1;
}

In SchedulerService.spawnFromSnapshot, Resume clones vm.Spec as-is:

spec := &lanternv1.AgentSpec{}
if vm.Spec != nil {
    spec = proto.Clone(vm.Spec).(*lanternv1.AgentSpec)
}
spec.RestoreSnapshotId = snapID

vm.Spec.Env["LANTERN_AGENT_INSTANCE_TOKEN"] retains the token issued at initial spawn time. The default TTL (LANTERN_AGENT_IDENTITY_TTL) is 1 hour. Any VM paused for longer than 1 hour will resume with an expired JWT, causing all guest calls to POST /v1/runtime/secrets/resolve to fail with ErrTokenExpired.

• Fix: Extend ResumeRequest to accept a refreshed ForkIdentity (or AgentIdentity), have control-plane.ResumeVM mint a fresh token via h.identity.Issue, and inject it in spawnFromSnapshot matching withIdentity.

---

[HIGH] 4. Missing SnapshotID Validation in Resume Causes Silent Cold-Boot

Location: services/runtime-scheduler/internal/handlers/lifecycle.go (lines 160–180)

In SchedulerService.Fork:

case lanternv1.VmState_VM_STATE_PAUSED:
    snapID = src.SnapshotID
    if snapID == "" {
        return nil, status.Errorf(codes.FailedPrecondition, "vm %q is paused without a snapshot", srcID)
    }

In SchedulerService.Resume:

if vm.State != lanternv1.VmState_VM_STATE_PAUSED {
    return nil, status.Errorf(codes.FailedPrecondition, "vm %q is %s; only a PAUSED vm can be resumed", vmID, vm.State)
}
// Missing check for vm.SnapshotID != ""
if err := s.spawnFromSnapshot(ctx, node.Address, vm, vm.Handle, vm.SnapshotID); err != nil {

If a paused VM in s.Store has vm.SnapshotID == "" (e.g. state inconsistency or DB recovery anomaly), spawnFromSnapshot sets spec.RestoreSnapshotId = "" and calls Dialer.Spawn. The manager will cold-boot a clean container from scratch rather than failing with FailedPrecondition.

• Fix: Add if vm.SnapshotID == "" { return nil, status.Errorf(codes.FailedPrecondition, "vm %q has no snapshot to resume", vmID) } before invoking spawnFromSnapshot.

---

3. Medium & Low Severity Issues

[MEDIUM] 5. Unreverted SnapshotID on Pause Stop Failure

Location: services/runtime-scheduler/internal/handlers/lifecycle.go (lines 112–122)

In SchedulerService.Pause:

s.Store.SetVMSnapshot(vmID, snap.SnapshotId)
s.Store.UpdateVMState(vmID, lanternv1.VmState_VM_STATE_PAUSED, "pausing", nil, time.Now().UTC())

if _, err := s.Dialer.Stop(ctx, node.Address, &lanternv1.StopRequest{VmId: vmID, Reason: "paused"}); err != nil {
    s.Store.UpdateVMState(vmID, lanternv1.VmState_VM_STATE_RUNNING, "pause failed: "+err.Error(), nil, time.Now().UTC())
    return nil, status.Errorf(codes.Unavailable, "pause %s: manager stop failed: %v", vmID, err)
}

If Dialer.Stop fails, state is reverted to RUNNING, but SetVMSnapshot is not reverted (persisted both in-memory and in sched_vms). The VM remains RUNNING while retaining a snapshot_id representing a snapshot taken during an aborted pause.

• Fix: Call s.Store.SetVMSnapshot(vmID, "") when Stop returns an error.

---

[MEDIUM] 6. jsonb_set Silent Failure when spec.env is Missing

Location: services/control-plane/internal/handlers/runtime_lifecycle.go (line 2876)

jsonb_set(spec, '{env,LANTERN_AGENT_INSTANCE_ID}', to_jsonb($7::text))

In PostgreSQL, jsonb_set(target, path, val) requires the parent path element (env) to already exist in target. If spec does not have an "env" object (e.g. empty or null spec), jsonb_set does not create the intermediate object and leaves spec unchanged.

• Fix: Ensure the parent object exists: jsonb_set(COALESCE(spec, '{}'::jsonb), '{env,LANTERN_AGENT_INSTANCE_ID}', to_jsonb($7::text), true).

---

[LOW] 7. CLI lantern vm exec Lacks mTLS Options for Prod

Location: packages/cli/internal/cli/runtime.go (line 526)

runVmExec configures service token creds but hardcodes plaintext transport:

opts := []grpc.DialOption{grpc.WithTransportCredentials(insecure.NewCredentials())}

With the runtime manager mandating mTLS in production (ADR 0025 §2), direct lantern vm exec commands will fail TLS negotiation against production nodes. (The control-plane proxy route /v1/runtime/vms/{id}/exec correctly uses ManagerTransportCreds).

• Fix: Wire client certificate support or route CLI exec through the control plane REST/SSE stream endpoint in production.