[dshakes]

Part 9 of 11 of the GA next-gen stack. Replaces #264/#255–#263: same changes, history rewritten to drop a 43 MB build binary committed by mistake, plus part 11 fixing every confirmed Codex cross-audit finding. Base: ga2/08-vm-identity. Merge in order with a merge commit (not squash).

Context, verification and needs-human list: #254 and docs/roadmap/2026-10-nextgen-sandbox-and-harness.md.

Commits in this part:

• merge: per-fork agent identity, resume reconcile grace, orphaned-snapshot logging

[github-actions]

🔎 Codex cross-audit (agent:audit)

Blocking

• services/runtime-scheduler/internal/handlers/lifecycle.go: fork child specs are created with withIdentity, which writes LANTERN_AGENT_INSTANCE_TOKEN into child.Spec, then Store.CreateVM(child) persists that spec through pg.go. That stores a live bearer credential in scheduler state/DB, violating the repo’s “secrets never appear in run state” invariant. Use a redacted stored spec and pass the token-bearing spec only to the manager spawn call.

Findings

• services/runtime-scheduler/internal/handlers/lifecycle.go: Fork snapshots a running source before the authoritative scheduler quota reservation at line 243. If the tenant is over TenantHardCap, the call still performs and persists a snapshot before returning ResourceExhausted, leaking manager resources for a denied operation. Reserve first, then snapshot, and release the reservation if snapshot/spawn setup fails.

• services/runtime-scheduler/internal/handlers/lifecycle.go: Resume accepts any PAUSED VM without checking SnapshotID != "", then calls spawnFromSnapshot with an empty snapshot id at line 165. A corrupt/legacy/partially persisted paused row can resume as an invalid or fresh VM under the same id, risking sandbox state loss. Mirror the fork precondition and return FailedPrecondition when paused-without-snapshot.

Verification

• Could not run Go tests: the sandbox is read-only, so Go cannot create /home/runner/go or /tmp/gocache. git diff --check passed.

[github-actions]

🛡️ Vuln scan — ❌ vulnerable dependency found

      #41: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.Shutdown
      #42: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.Shutdown
      #43: internal/telemetry/tracer.go:92:21: telemetry.InitTracer calls trace.TracerProvider.Shutdown
      #44: internal/handlers/runtime.go:282:50: handlers.ManagerDialOptions calls otelgrpc.NewClientHandler, which calls trace.TracerProvider.Tracer
      #45: internal/telemetry/tracer.go:79:23: telemetry.InitTracer calls trace.WithBatcher
      #46: internal/telemetry/tracer.go:80:24: telemetry.InitTracer calls trace.WithResource
      #47: internal/telemetry/tracer.go:81:23: telemetry.InitTracer calls trace.WithSampler
      #48: internal/handlers/llm_proxy.go:3950:104: handlers.LlmProxyHandler.Transcribe calls trace.errUnsupportedSampler.Error
      #49: internal/telemetry/tracer.go:17:2: telemetry.init calls trace.init
      #50: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.init
      #51: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.logDropped
      #52: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.logDropped
      #53: internal/handlers/llm_proxy.go:3825:27: handlers.LlmProxyHandler.OCR calls io.ReadAll, which eventually calls trace.nonRecordingSpan.AddEvent
      #54: internal/handlers/runtime.go:1652:2: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.End
      #55: internal/middleware/span.go:34:22: middleware.EnrichSpan calls trace.nonRecordingSpan.IsRecording
      #56: internal/handlers/runtime.go:1660:19: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.RecordError
      #57: internal/handlers/runtime.go:1651:20: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.SetAttributes
      #58: cmd/server/main.go:851:38: server.main calls http.Server.ListenAndServe, which eventually calls trace.nonRecordingSpan.SetName
      #59: internal/handlers/runtime.go:1661:17: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.SetStatus
      #60: internal/handlers/runtime.go:1650:27: handlers.RuntimeHandler.Schedule calls noop.Tracer.Start, which calls trace.nonRecordingSpan.SpanContext
      #61: internal/handlers/llm_proxy.go:3816:24: handlers.LlmProxyHandler.OCR calls http.Client.Do, which eventually calls trace.nonRecordingSpan.TracerProvider
      #62: internal/handlers/llm_proxy.go:3825:27: handlers.LlmProxyHandler.OCR calls io.ReadAll, which eventually calls trace.recordingSpan.AddEvent
      #63: internal/handlers/runtime.go:1652:2: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.End
      #64: internal/middleware/span.go:34:22: middleware.EnrichSpan calls trace.recordingSpan.IsRecording
      #65: internal/handlers/runtime.go:1660:19: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.RecordError
      #66: internal/handlers/runtime.go:1651:20: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.SetAttributes
      #67: cmd/server/main.go:851:38: server.main calls http.Server.ListenAndServe, which eventually calls trace.recordingSpan.SetName
      #68: internal/handlers/runtime.go:1661:17: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.SetStatus
      #69: internal/handlers/runtime.go:1650:27: handlers.RuntimeHandler.Schedule calls noop.Tracer.Start, which calls trace.recordingSpan.SpanContext
      #70: internal/handlers/llm_proxy.go:3816:24: handlers.LlmProxyHandler.OCR calls http.Client.Do, which eventually calls trace.recordingSpan.TracerProvider
      #71: internal/handlers/llm_proxy.go:3950:104: handlers.LlmProxyHandler.Transcribe calls trace.samplerArgParseError.Error
      #72: internal/handlers/connector_health.go:177:15: handlers.PersistConnectorQuarantine calls errors.As, which eventually calls trace.samplerArgParseError.Unwrap
      #73: internal/handlers/runtime.go:1650:27: handlers.RuntimeHandler.Schedule calls trace.tracer.Start
      #74: internal/telemetry/tracer.go:86:27: telemetry.InitTracer calls otel.SetTextMapPropagator, which eventually calls trace.tracerProviderConfig.MarshalLog
      #75: internal/telemetry/tracer.go:79:23: telemetry.InitTracer calls trace.WithBatcher, which eventually calls x.Feature[string].Enabled
      #76: internal/telemetry/tracer.go:16:2: telemetry.init calls resource.init, which calls x.init

Vulnerability #15: GO-2026-6443
    Server panic via missing authority or Host headers in google.golang.org/grpc
  More info: https://pkg.go.dev/vuln/GO-2026-6443
  Module: google.golang.org/grpc
    Found in: google.golang.org/grpc@v1.82.1
    Fixed in: google.golang.org/grpc@v1.82.2
    Example traces found:
      #1: cmd/server/main.go:262:32: server.main calls grpc.Server.Serve, which eventually calls transport.http2Server.HandleStreams

Vulnerability #16: GO-2026-6348
    Heap Memory Exhaustion (OOM) via HTTP/2 DATA Frame Fragmentation in
    google.golang.org/grpc
  More info: https://pkg.go.dev/vuln/GO-2026-6348
  Module: google.golang.org/grpc
    Found in: google.golang.org/grpc@v1.82.1
    Fixed in: google.golang.org/grpc@v1.83.1
    Example traces found:
      #1: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.BufferSlice.MaterializeToBuffer
      #2: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.IsBelowBufferPoolingThreshold
      #3: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.NewBuffer
      #4: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls mem.ReadAll
      #5: internal/handlers/llm_proxy.go:3950:104: handlers.LlmProxyHandler.Transcribe calls multierr.multiError.Error, which eventually calls mem.writer.Write
      #6: internal/handlers/runtime.go:2081:27: handlers.RuntimeHandler.StreamLogs calls grpc.GenericClientStream[github.com/dshakes/lantern/gen/go/lantern/v1.LogsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.RuntimeLogLine].Recv, which eventually calls transport.ClientStream.Close
      #7: internal/handlers/runtime.go:440:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.ClientStream.Header
      #8: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ClientStream.Read
      #9: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ClientStream.RecvCompress
      #10: internal/handlers/runtime.go:440:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.ClientStream.TrailersOnly
      #11: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.NewHTTP2Client
      #12: cmd/server/main.go:262:32: server.main calls grpc.Server.Serve, which eventually calls transport.NewServerTransport
      #13: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ServerStream.Read
      #14: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.Stream.ReadMessageHeader
      #15: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.http2Client.Close
      #16: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.http2Client.GracefulClose
      #17: internal/handlers/runtime.go:440:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.http2Client.NewStream
      #18: cmd/server/main.go:916:25: server.main calls grpc.Server.GracefulStop, which eventually calls transport.http2Server.Close
      #19: cmd/server/main.go:262:32: server.main calls grpc.Server.Serve, which eventually calls transport.http2Server.HandleStreams

Your code is affected by 16 vulnerabilities from 3 modules and the Go standard library.
This scan also found 3 vulnerabilities in packages you import and 19
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.
make: *** [Makefile:270: audit] Error 3

[github-actions]

🔷 Gemini cross-audit (agent:audit-gemini)

Executive Summary

This PR introduces Wave 2 microVM lifecycle management (pause, resume, fork) across the control-plane, runtime-scheduler, CLI, and Go SDK, and closes mTLS/service-token authorization gaps when dialing the runtime-manager.

The architecture (ADR 0025, tenant isolation via RLS, non-leaking 404s, and atomic quota reservations) is well-structured. However, the audit identified two correctness/reliability regressions (one blocking metric corruption and one post-restart VM termination bug), along with a token expiration issue on long-paused VMs and a snapshot leak on quota rejection.

---

Blocking Issues

1. Metric Underflow & Gauge Corruption on Fork Failure

• File: services/runtime-scheduler/internal/handlers/lifecycle.go (Fork)
• Issue:
  In SchedulerService.Fork, quota is reserved via s.Store.TryReserveTenantVMs(tenantID, count, s.TenantHardCap). Note that TryReserveTenantVMs only adjusts the store count; it does not touch s.Metrics.ActiveVMs.
  In the per-child spawn loop, if a child fails:

  if err := s.spawnFromSnapshot(ctx, node.Address, child, handle, snapID); err != nil {
      failed++
      s.Store.UpdateVMState(handle.VmId, lanternv1.VmState_VM_STATE_FAILED, "spawn failed: "+err.Error(), nil, time.Now().UTC())
      s.Store.IncrTenantVMs(tenantID, -1)
      if s.Metrics != nil {
          s.Metrics.ActiveVMs.Dec() // <-- BUG: Dec() called without prior Inc()
      }
      ...

  After the loop:

  if s.Metrics != nil && count > failed {
      s.Metrics.ActiveVMs.Add(float64(count - failed))
  }

• Consequence:
  • If count = 1 and spawn fails: ActiveVMs.Dec() is called (-1). count > failed is false, so Add is skipped. Net change: -1.
  • If count = 2, 1 succeeds, 1 fails: in-loop Dec() (-1), post-loop Add(1). Net change: 0, despite 1 child running. When that child eventually terminates, Terminate calls Dec() again, driving the gauge to -1.
  • Every failed fork permanently corrupts the ActiveVMs Prometheus gauge.
• Remedy: Either increment ActiveVMs.Add(float64(count)) upfront when reserving quota slots (mirroring Resume), or remove ActiveVMs.Dec() from the child error branch and only add float64(count - failed) after the loop.

---

High Severity Issues

2. Resumed VMs Prematurely Terminated by Reconciler Following Scheduler Restart

• Files:
  • services/runtime-scheduler/internal/cluster/state.go
  • services/runtime-scheduler/internal/handlers/reconcile.go
  • services/runtime-scheduler/internal/store/migrate.go
  • services/runtime-scheduler/internal/store/pg.go
• Issue:
  v.RespawnedAt was introduced to grant a newly resumed VM the dispatch grace window (reconcileGrace = 60s) before being considered dead by node inventory reconciliation:

  born := createdAt(v)
  if v.RespawnedAt.After(born) {
      born = v.RespawnedAt
  }
  if !v.InventorySeenAt.After(born) && born.Add(reconcileGrace).After(now) {
      continue // inside dispatch window
  }

  However, RespawnedAt exists only in-memory on cluster.VM:
  1. sched_vms has no respawned_at column in migrate.go.
  2. persistVM does not save RespawnedAt.
  3. loadVMs does not load RespawnedAt (defaults to zero time.Time{}).
  4. createdAt(v) checks v.Handle.CreatedAt first, which holds the VM's initial creation time from hours or days ago.
• Consequence: If the runtime-scheduler restarts (e.g., rolling update or failover) shortly after a VM is resumed, v.RespawnedAt resets to zero, born is set to the historical CreatedAt, and InventorySeenAt is zero. The very next node heartbeat that arrives before the node registers the revived container will evaluate born.Add(reconcileGrace).After(now) as false and immediately terminate the resumed VM.
• Remedy: Either persist respawned_at in sched_vms and restore it in loadVMs, or update v.Handle.CreatedAt / set born = v.LastEventAt on reload if the state is RUNNING and has a snapshot_id.

---

Medium Severity Issues

3. Token Expiry on Resume for VMs Paused > Identity TTL

• Files:
  • services/control-plane/internal/handlers/runtime_lifecycle.go (ResumeVM)
  • services/runtime-scheduler/internal/handlers/lifecycle.go (Resume, spawnFromSnapshot)
  • packages/proto/lantern/v1/runtime.proto (ResumeRequest)
• Issue:
  For Fork, the control-plane mints fresh LANTERN_AGENT_INSTANCE_ID and LANTERN_AGENT_INSTANCE_TOKEN credentials for each child.
  For Resume, ResumeRequest passes only vm_id, and spawnFromSnapshot reuses the original vm.Spec. LANTERN_AGENT_INSTANCE_TOKEN is a signed JWT with a default 1-hour TTL (LANTERN_AGENT_IDENTITY_TTL).
• Consequence: If a user pauses a VM and resumes it after >1 hour (e.g., overnight or idle between tasks), the agent token in the VM's environment has expired. Any subsequent call to /v1/runtime/secrets/resolve fails authentication (verifyInstanceToken: invalid agent-instance token: token is expired).
• Remedy: Allow ResumeVM to mint a refreshed token for the VM's existing agent_instance_id (or issue a fresh incarnation token) and thread it through ResumeRequest to spawnFromSnapshot, matching ADR 0025 ("adopts the new incarnation's tokens").

4. Snapshot Leaked on Quota Denial in Fork

• File: services/runtime-scheduler/internal/handlers/lifecycle.go (Fork)
• Issue:
  In SchedulerService.Fork:

  snap, err := s.Dialer.Snapshot(ctx, node.Address, &lanternv1.SnapshotRequest{VmId: srcID, KeepRunning: true})
  ...
  // Quota is all-or-nothing: take count slots now; release on per-child failure.
  if !s.Store.TryReserveTenantVMs(tenantID, count, s.TenantHardCap) {
      return nil, status.Errorf(codes.ResourceExhausted, ...)
  }

  s.Dialer.Snapshot is invoked before s.Store.TryReserveTenantVMs.
• Consequence: If the tenant is at or near the hard cap, a full multi-gigabyte disk snapshot is created on the node. The call is then immediately rejected with ResourceExhausted. Because the source was running, it does not reference this snapshot; and because children were not created, no VM references it. Per ADR 0025, there is no snapshot deletion RPC yet, so the snapshot leaks on node disk.
• Remedy: Check s.Store.TryReserveTenantVMs before initiating s.Dialer.Snapshot.

---

Low Severity / Inconsistencies

1. GET /v1/runtime/vms (ListVMs) Omits snapshotId and parentVmId:
   In services/control-plane/internal/handlers/runtime.go, GetVM was updated to SELECT and scan snapshot_id, parent_vm_id, but ListVMs (~line 1750) was not updated. VM list queries via REST will consistently return nil for these fields.
2. Redundant DB Queries in ForkVM:
   lifecycleTarget queries runtime_vms to verify tenant ownership and state. ForkVM immediately performs a second query against runtime_vms on the exact same (vm_id, tenant_id) to extract run_id and agent_version_id. These can be selected in lifecycleTarget or consolidated.
3. Fork on Draining Nodes:
   SchedulerService.Fork does not check whether node.Draining is true. If the host node is draining, Fork will still attempt to launch children on it rather than rejecting early with codes.Unavailable.

---

Verification Summary

• Tenant Isolation (Invariant bridge: expose running build SHA at startup + on /health #7): Verified. All new lifecycle routes (pause, resume, fork) in control-plane enforce tenant ownership via WithTenant and return 404 for foreign or non-existent IDs. Scheduler methods (ownedVM) extract tenant from context metadata and reject cross-tenant access.
• Service Token & mTLS: Verified. managerTransportCreds correctly enforces mTLS in production and falls back to insecure in dev. The shared x-lantern-service-token header is properly attached when dialing runtime-manager from the scheduler, workflow engine, control-plane, and CLI.

[dshakes]

Superseded: the history rewrite on this stack accidentally re-signed master's commits (base diverged). Replaced by a range-limited rewrite on ga3/* — see the new GA 1/11 PR.