[dshakes]

Part 9 of 10 of the GA next-gen stack (split from #254 so each AI review fits its budget). Base: ga/08-vm-identity. Merge in order with a merge commit (not squash) so the stack stays intact.

Full context, verification results and needs-human list: #254 and docs/roadmap/2026-10-nextgen-sandbox-and-harness.md.

Commits in this part:

• merge: per-fork agent identity, resume reconcile grace, orphaned-snapshot logging

[github-actions]

🔎 Codex cross-audit (agent:audit)

Blocking

1. services/runtime-scheduler/internal/handlers/lifecycle.go:110 and line 165 make pause/resume crash-unsafe. Pause persists PAUSED before Stop; Resume starts the VM before persisting RUNNING. A scheduler crash in either window can leave a real running VM reloaded as PAUSED, and loadVMs then excludes it from quota/live accounting. This can orphan running untrusted code outside quota and normal reconciliation.

2. services/control-plane/internal/handlers/runtime_lifecycle.go:308 treats fork child runtime_vms inserts as best-effort after the scheduler has already spawned children. If that insert fails, the API still returns 200 at line 333, but the live child has no control-plane row, so later GET/DELETE, quota checks, reporting, and secret binding can miss it. Fork should fail/compensate, or the scheduler/control-plane must have durable child creation semantics before spawn.

High

3. services/runtime-scheduler/internal/handlers/lifecycle.go:225 snapshots a running source before checking/reserving fork quota at line 243. A quota-denied fork can still force snapshot I/O/storage side effects. The new test even expects this. Reserve quota before snapshot, then release on later failure.

4. services/control-plane/internal/handlers/runtime.go:1856 does not select snapshot_id or parent_vm_id for GET /v1/runtime/vms, while GET /v1/runtime/vms/{id} does. Fork/pause metadata is invisible in list responses, which is likely the primary dashboard/CLI view and makes fork lineage look absent.

No tests run; this was a targeted diff audit.

[github-actions]

🛡️ Vuln scan — ❌ vulnerable dependency found

      #41: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.Shutdown
      #42: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.Shutdown
      #43: internal/telemetry/tracer.go:92:21: telemetry.InitTracer calls trace.TracerProvider.Shutdown
      #44: internal/handlers/runtime.go:282:50: handlers.ManagerDialOptions calls otelgrpc.NewClientHandler, which calls trace.TracerProvider.Tracer
      #45: internal/telemetry/tracer.go:79:23: telemetry.InitTracer calls trace.WithBatcher
      #46: internal/telemetry/tracer.go:80:24: telemetry.InitTracer calls trace.WithResource
      #47: internal/telemetry/tracer.go:81:23: telemetry.InitTracer calls trace.WithSampler
      #48: internal/handlers/llm_proxy.go:3950:104: handlers.LlmProxyHandler.Transcribe calls trace.errUnsupportedSampler.Error
      #49: internal/telemetry/tracer.go:17:2: telemetry.init calls trace.init
      #50: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.init
      #51: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.logDropped
      #52: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.logDropped
      #53: internal/handlers/llm_proxy.go:3825:27: handlers.LlmProxyHandler.OCR calls io.ReadAll, which eventually calls trace.nonRecordingSpan.AddEvent
      #54: internal/handlers/runtime.go:1652:2: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.End
      #55: internal/middleware/span.go:34:22: middleware.EnrichSpan calls trace.nonRecordingSpan.IsRecording
      #56: internal/handlers/runtime.go:1660:19: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.RecordError
      #57: internal/handlers/runtime.go:1651:20: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.SetAttributes
      #58: cmd/server/main.go:851:38: server.main calls http.Server.ListenAndServe, which eventually calls trace.nonRecordingSpan.SetName
      #59: internal/handlers/runtime.go:1661:17: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.SetStatus
      #60: internal/handlers/runtime.go:1650:27: handlers.RuntimeHandler.Schedule calls noop.Tracer.Start, which calls trace.nonRecordingSpan.SpanContext
      #61: internal/handlers/llm_proxy.go:3816:24: handlers.LlmProxyHandler.OCR calls http.Client.Do, which eventually calls trace.nonRecordingSpan.TracerProvider
      #62: internal/handlers/llm_proxy.go:3825:27: handlers.LlmProxyHandler.OCR calls io.ReadAll, which eventually calls trace.recordingSpan.AddEvent
      #63: internal/handlers/runtime.go:1652:2: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.End
      #64: internal/middleware/span.go:34:22: middleware.EnrichSpan calls trace.recordingSpan.IsRecording
      #65: internal/handlers/runtime.go:1660:19: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.RecordError
      #66: internal/handlers/runtime.go:1651:20: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.SetAttributes
      #67: cmd/server/main.go:851:38: server.main calls http.Server.ListenAndServe, which eventually calls trace.recordingSpan.SetName
      #68: internal/handlers/runtime.go:1661:17: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.SetStatus
      #69: internal/handlers/runtime.go:1650:27: handlers.RuntimeHandler.Schedule calls noop.Tracer.Start, which calls trace.recordingSpan.SpanContext
      #70: internal/handlers/llm_proxy.go:3816:24: handlers.LlmProxyHandler.OCR calls http.Client.Do, which eventually calls trace.recordingSpan.TracerProvider
      #71: internal/handlers/llm_proxy.go:3950:104: handlers.LlmProxyHandler.Transcribe calls trace.samplerArgParseError.Error
      #72: internal/handlers/connector_health.go:177:15: handlers.PersistConnectorQuarantine calls errors.As, which eventually calls trace.samplerArgParseError.Unwrap
      #73: internal/handlers/runtime.go:1650:27: handlers.RuntimeHandler.Schedule calls trace.tracer.Start
      #74: internal/telemetry/tracer.go:86:27: telemetry.InitTracer calls otel.SetTextMapPropagator, which eventually calls trace.tracerProviderConfig.MarshalLog
      #75: internal/telemetry/tracer.go:79:23: telemetry.InitTracer calls trace.WithBatcher, which eventually calls x.Feature[string].Enabled
      #76: internal/telemetry/tracer.go:16:2: telemetry.init calls resource.init, which calls x.init

Vulnerability #15: GO-2026-6443
    Server panic via missing authority or Host headers in google.golang.org/grpc
  More info: https://pkg.go.dev/vuln/GO-2026-6443
  Module: google.golang.org/grpc
    Found in: google.golang.org/grpc@v1.82.1
    Fixed in: google.golang.org/grpc@v1.82.2
    Example traces found:
      #1: cmd/server/main.go:262:32: server.main calls grpc.Server.Serve, which eventually calls transport.http2Server.HandleStreams

Vulnerability #16: GO-2026-6348
    Heap Memory Exhaustion (OOM) via HTTP/2 DATA Frame Fragmentation in
    google.golang.org/grpc
  More info: https://pkg.go.dev/vuln/GO-2026-6348
  Module: google.golang.org/grpc
    Found in: google.golang.org/grpc@v1.82.1
    Fixed in: google.golang.org/grpc@v1.83.1
    Example traces found:
      #1: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.BufferSlice.MaterializeToBuffer
      #2: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.IsBelowBufferPoolingThreshold
      #3: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.NewBuffer
      #4: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls mem.ReadAll
      #5: internal/handlers/llm_proxy.go:3950:104: handlers.LlmProxyHandler.Transcribe calls multierr.multiError.Error, which eventually calls mem.writer.Write
      #6: internal/handlers/runtime.go:2081:27: handlers.RuntimeHandler.StreamLogs calls grpc.GenericClientStream[github.com/dshakes/lantern/gen/go/lantern/v1.LogsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.RuntimeLogLine].Recv, which eventually calls transport.ClientStream.Close
      #7: internal/handlers/runtime.go:440:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.ClientStream.Header
      #8: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ClientStream.Read
      #9: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ClientStream.RecvCompress
      #10: internal/handlers/runtime.go:440:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.ClientStream.TrailersOnly
      #11: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.NewHTTP2Client
      #12: cmd/server/main.go:262:32: server.main calls grpc.Server.Serve, which eventually calls transport.NewServerTransport
      #13: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ServerStream.Read
      #14: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.Stream.ReadMessageHeader
      #15: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.http2Client.Close
      #16: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.http2Client.GracefulClose
      #17: internal/handlers/runtime.go:440:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.http2Client.NewStream
      #18: cmd/server/main.go:916:25: server.main calls grpc.Server.GracefulStop, which eventually calls transport.http2Server.Close
      #19: cmd/server/main.go:262:32: server.main calls grpc.Server.Serve, which eventually calls transport.http2Server.HandleStreams

Your code is affected by 16 vulnerabilities from 3 modules and the Go standard library.
This scan also found 3 vulnerabilities in packages you import and 19
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.
make: *** [Makefile:270: audit] Error 3

[github-actions]

🔷 Gemini cross-audit (agent:audit-gemini)

Audit Summary: pr-diff.patch (MicroVM Sandbox Lifecycle: Pause, Resume, Fork)

Blocking Issues: None.
The PR correctly implements microVM pause, resume, and fork across control-plane, runtime-scheduler, workflow-engine, sdk-go, and cli. Architectural invariants (tenant isolation, secret containment, durable state, and quota accounting) are strictly preserved.

---

Correctness, Security & Edge Cases (Non-Blocking / Follow-ups)

1. CLI lantern vm exec lacks mTLS support in production

• Location: packages/cli/internal/cli/runtime.go:526 (runVmExec)
• Issue: While runVmExec was updated to attach LANTERN_GRPC_SERVICE_TOKEN, it still hardcodes grpc.WithTransportCredentials(insecure.NewCredentials()). In production where runtime-manager mandates mTLS (with client cert verification against LANTERN_RUNTIME_MANAGER_TLS_CA), direct lantern vm exec from the CLI will fail the TLS handshake.
• Recommendation: Add TLS credential resolution mirroring ManagerTransportCreds (or route CLI exec through control-plane's authenticated proxy).

2. SchedulerService.Pause misses explicit node registration check

• Location: services/runtime-scheduler/internal/handlers/lifecycle.go:96
• Issue: Pause ignores the boolean return from s.Store.GetNode(vm.NodeName):

  node, _ := s.Store.GetNode(vm.NodeName)
  snap, err := s.Dialer.Snapshot(ctx, node.Address, ...)

  Both Resume and Fork guard this with if !ok { return status.Errorf(codes.Unavailable, "node %q ... is not registered") }. If the node holding the VM has dropped out of store inventory, Pause calls Snapshot with node.Address == "", which falls back to LANTERN_DEFAULT_MANAGER_ADDR or fails with a confusing dial error instead of a clear codes.Unavailable.
• Recommendation: Check node, ok := s.Store.GetNode(vm.NodeName) and return codes.Unavailable on !ok.

3. jsonb_set in ForkVM child insert assumes {env} exists

• Location: services/control-plane/internal/handlers/runtime_lifecycle.go:207
• Issue: The SQL insert sets the child's identity via:

  jsonb_set(spec, '{env,LANTERN_AGENT_INSTANCE_ID}', to_jsonb($7::text))

  In PostgreSQL, jsonb_set with default create_missing = true only creates new keys if the parent path exists. If a VM's spec JSON lacks an top-level "env" key (e.g. {} or {"imageDigest":"..."}), jsonb_set returns spec unchanged. The child VM still receives its identity via the dedicated agent_instance_id column and the manager spawn spec, but its serialized JSONB spec in runtime_vms will omit LANTERN_AGENT_INSTANCE_ID.
• Recommendation: Use COALESCE(spec, '{}'::jsonb) with ensure-object or jsonb_set(jsonb_set(spec, '{env}', COALESCE(spec->'env', '{}'::jsonb), true), '{env,LANTERN_AGENT_INSTANCE_ID}', to_jsonb($7::text)) if JSON spec fidelity is required.

4. Unbounded sync.Map growth in s.vmLocks

• Location: services/runtime-scheduler/internal/handlers/lifecycle.go:43 (lockVM)
• Issue: Mutexes allocated per vm_id are stored in s.vmLocks and never evicted. As noted in the codebase comment (// ponytail), memory overhead per VM is small (~48 bytes), but for long-running schedulers processing millions of VMs over time, entries accumulate permanently.
• Recommendation: Clean up entry via s.vmLocks.Delete(req.VmId) in SchedulerService.Terminate after the VM is deleted from store.

---

Invariant & Architecture Verification

• Invariant bridge: expose running build SHA at startup + on /health #7 (Multi-tenant isolation & RLS): All control-plane queries use WithTenant and explicitly filter by tenant_id = claims.TenantID. Foreign VM IDs return 404 without existence leakage. Scheduler RPCs enforce MustTenantID and leader checks.
• Invariant bridge: read Apple Calendar.app for appointment queries (not just Google) #8 (Idempotent execution & quota safety): Pause on an already-paused VM is idempotent and returns the existing snapshot without calling the manager or double-freeing quota. Terminating a paused VM suppresses manager Stop and avoids double-decrementing quota. Partial fork failures accurately release slots for failed children only.
• Invariant twilio: send via Messaging Service (RCS + automatic SMS fallback) #10 (Secret protection): Fresh child agent instance tokens are minted in control-plane, passed to scheduler, and injected into the runtime-manager spawn spec. They are stripped from the stored AgentSpec and never written to Postgres or logs.
• Reconciliation & Grace Window: ReconcileNodeInventory explicitly ignores VM_STATE_PAUSED VMs so they are not wrongly reaped. Resumed VMs receive a fresh grace window based on RespawnedAt to prevent transient termination races before the node's next heartbeat.
• Transport Security: ManagerTransportCreds correctly fails closed in production (LANTERN_ENV=prod/production/staging) across control-plane, runtime-scheduler, and workflow-engine when TLS CA or client certificates are missing.

[dshakes]

Superseded by the rewritten stack #265–#275 (same changes; history rewritten to drop a 43 MB build binary committed by mistake; part 11 #275 fixes the confirmed Codex audit findings raised on this stack).