[dshakes]

Part 1 of 10 of the GA next-gen stack (split from #254 so each AI review fits its budget). Base: master. Merge the stack in order with a merge commit (not squash) so later parts stay intact. Next: #255.

Full context, verification results and needs-human list: #254 and docs/roadmap/2026-10-nextgen-sandbox-and-harness.md.

Commits in this part:

• merge: runtime spec plumbing, quota leak, tenant-scoped idempotency, SSE exec
• merge: model-router service-token auth (S2)
• docs(roadmap): 2026-10 next-gen sandbox + personal harness plan from the three audits
• fix(bridges): a fresh pending draft can't be replaced by a message typed before it existed

[github-actions]

🔎 Codex cross-audit (agent:audit)

Blocking

• services/control-plane/internal/handlers/runtime.go: scheduler idempotency can bind the wrong agent identity to a VM. scheduleAgentSpec mints a fresh agent_instance_id/token before calling Schedule; on an idempotent retry, the scheduler may return the cached original VM handle without respawning. If the first call reached the scheduler but the control-plane lost the response before inserting runtime_vms, the retry inserts the same vm_id with the new identity, while the actual VM booted with the old token. Secret vending / VM-bound auth will then fail because runtime_vms.agent_instance_id no longer matches the token inside the running VM. This is the exact ambiguous-response case idempotency is meant to handle. The scheduler cache return is at scheduler_grpc.go, and the control-plane insert uses the retry identity at runtime.go.

High

• services/control-plane/internal/handlers/runtime.go: the main microVM run path still drops manifest egress_rules and secrets. The PR adds map-to-proto support for those fields, but ScheduleRunInMicroVM only copies limits, confidential, env, and restoreSnapshotId from the manifest into agentSpecDTO, so agent manifests using the documented snake_case sandbox policy boot without the declared egress/secret policy. The HTTP schedule path has a similar gap: snake_case fallback only runs for image_digest / isolation_class and only when imageDigest is empty, so egress_rules / secret_uri from CLI-style YAML remain ignored at runtime.go.

I did not run tests; this was a read-only audit of git diff origin/master...HEAD.

[github-actions]

🛡️ Vuln scan — ❌ vulnerable dependency found

      #41: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.Shutdown
      #42: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.Shutdown
      #43: internal/telemetry/tracer.go:92:21: telemetry.InitTracer calls trace.TracerProvider.Shutdown
      #44: internal/handlers/runtime.go:186:50: handlers.NewGRPCSchedulerClient calls otelgrpc.NewClientHandler, which calls trace.TracerProvider.Tracer
      #45: internal/telemetry/tracer.go:79:23: telemetry.InitTracer calls trace.WithBatcher
      #46: internal/telemetry/tracer.go:80:24: telemetry.InitTracer calls trace.WithResource
      #47: internal/telemetry/tracer.go:81:23: telemetry.InitTracer calls trace.WithSampler
      #48: internal/handlers/llm_proxy.go:3919:104: handlers.LlmProxyHandler.Transcribe calls trace.errUnsupportedSampler.Error
      #49: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.init
      #50: internal/telemetry/tracer.go:17:2: telemetry.init calls trace.init
      #51: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.logDropped
      #52: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.logDropped
      #53: internal/handlers/llm_proxy.go:3794:27: handlers.LlmProxyHandler.OCR calls io.ReadAll, which eventually calls trace.nonRecordingSpan.AddEvent
      #54: internal/handlers/runtime.go:1556:2: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.End
      #55: internal/middleware/span.go:34:22: middleware.EnrichSpan calls trace.nonRecordingSpan.IsRecording
      #56: internal/handlers/runtime.go:1564:19: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.RecordError
      #57: internal/handlers/runtime.go:1555:20: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.SetAttributes
      #58: cmd/server/main.go:841:38: server.main calls http.Server.ListenAndServe, which eventually calls trace.nonRecordingSpan.SetName
      #59: internal/handlers/runtime.go:1565:17: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.SetStatus
      #60: internal/handlers/runtime.go:1554:27: handlers.RuntimeHandler.Schedule calls noop.Tracer.Start, which calls trace.nonRecordingSpan.SpanContext
      #61: internal/handlers/llm_proxy.go:3785:24: handlers.LlmProxyHandler.OCR calls http.Client.Do, which eventually calls trace.nonRecordingSpan.TracerProvider
      #62: internal/handlers/llm_proxy.go:3794:27: handlers.LlmProxyHandler.OCR calls io.ReadAll, which eventually calls trace.recordingSpan.AddEvent
      #63: internal/handlers/runtime.go:1556:2: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.End
      #64: internal/middleware/span.go:34:22: middleware.EnrichSpan calls trace.recordingSpan.IsRecording
      #65: internal/handlers/runtime.go:1564:19: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.RecordError
      #66: internal/handlers/runtime.go:1555:20: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.SetAttributes
      #67: cmd/server/main.go:841:38: server.main calls http.Server.ListenAndServe, which eventually calls trace.recordingSpan.SetName
      #68: internal/handlers/runtime.go:1565:17: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.SetStatus
      #69: internal/handlers/runtime.go:1554:27: handlers.RuntimeHandler.Schedule calls noop.Tracer.Start, which calls trace.recordingSpan.SpanContext
      #70: internal/handlers/llm_proxy.go:3785:24: handlers.LlmProxyHandler.OCR calls http.Client.Do, which eventually calls trace.recordingSpan.TracerProvider
      #71: internal/handlers/llm_proxy.go:3919:104: handlers.LlmProxyHandler.Transcribe calls trace.samplerArgParseError.Error
      #72: internal/handlers/connector_health.go:177:15: handlers.PersistConnectorQuarantine calls errors.As, which eventually calls trace.samplerArgParseError.Unwrap
      #73: internal/handlers/runtime.go:1554:27: handlers.RuntimeHandler.Schedule calls trace.tracer.Start
      #74: internal/telemetry/tracer.go:86:27: telemetry.InitTracer calls otel.SetTextMapPropagator, which eventually calls trace.tracerProviderConfig.MarshalLog
      #75: internal/telemetry/tracer.go:79:23: telemetry.InitTracer calls trace.WithBatcher, which eventually calls x.Feature[string].Enabled
      #76: internal/telemetry/tracer.go:16:2: telemetry.init calls resource.init, which calls x.init

Vulnerability #15: GO-2026-6443
    Server panic via missing authority or Host headers in google.golang.org/grpc
  More info: https://pkg.go.dev/vuln/GO-2026-6443
  Module: google.golang.org/grpc
    Found in: google.golang.org/grpc@v1.82.1
    Fixed in: google.golang.org/grpc@v1.82.2
    Example traces found:
      #1: cmd/server/main.go:263:32: server.main calls grpc.Server.Serve, which eventually calls transport.http2Server.HandleStreams

Vulnerability #16: GO-2026-6348
    Heap Memory Exhaustion (OOM) via HTTP/2 DATA Frame Fragmentation in
    google.golang.org/grpc
  More info: https://pkg.go.dev/vuln/GO-2026-6348
  Module: google.golang.org/grpc
    Found in: google.golang.org/grpc@v1.82.1
    Fixed in: google.golang.org/grpc@v1.83.1
    Example traces found:
      #1: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.BufferSlice.MaterializeToBuffer
      #2: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.IsBelowBufferPoolingThreshold
      #3: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.NewBuffer
      #4: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls mem.ReadAll
      #5: internal/handlers/llm_proxy.go:3919:104: handlers.LlmProxyHandler.Transcribe calls multierr.multiError.Error, which eventually calls mem.writer.Write
      #6: internal/handlers/runtime.go:1984:27: handlers.RuntimeHandler.StreamLogs calls grpc.GenericClientStream[github.com/dshakes/lantern/gen/go/lantern/v1.LogsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.RuntimeLogLine].Recv, which eventually calls transport.ClientStream.Close
      #7: internal/handlers/runtime.go:354:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.ClientStream.Header
      #8: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ClientStream.Read
      #9: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ClientStream.RecvCompress
      #10: internal/handlers/runtime.go:354:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.ClientStream.TrailersOnly
      #11: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.NewHTTP2Client
      #12: cmd/server/main.go:263:32: server.main calls grpc.Server.Serve, which eventually calls transport.NewServerTransport
      #13: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ServerStream.Read
      #14: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.Stream.ReadMessageHeader
      #15: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.http2Client.Close
      #16: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.http2Client.GracefulClose
      #17: internal/handlers/runtime.go:354:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.http2Client.NewStream
      #18: cmd/server/main.go:906:25: server.main calls grpc.Server.GracefulStop, which eventually calls transport.http2Server.Close
      #19: cmd/server/main.go:263:32: server.main calls grpc.Server.Serve, which eventually calls transport.http2Server.HandleStreams

Your code is affected by 16 vulnerabilities from 3 modules and the Go standard library.
This scan also found 3 vulnerabilities in packages you import and 19
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.
make: *** [Makefile:270: audit] Error 3

[github-actions]

🔷 Gemini cross-audit (agent:audit-gemini)

Blocking Issues

1. Quota double-decrement & node occupancy corruption on terminated VMs (scheduler_grpc.go)

• Location: services/runtime-scheduler/internal/handlers/scheduler_grpc.go:386, 424
• Issue: Terminate only marks quota already freed if vm.State == lanternv1.VmState_VM_STATE_FAILED:

  quotaAlreadyFreed := vm.State == lanternv1.VmState_VM_STATE_FAILED
  ...
  if !quotaAlreadyFreed {
      s.Store.IncrTenantVMs(tenantID, -1)
  }

  When batch agent workloads finish, ReconcileNodeInventory (reconcile.go) marks the VM VM_STATE_TERMINATED and calls store.IncrTenantVMs(v.TenantID, -1). If an operator or cleanup job subsequently invokes Terminate on that VM, quotaAlreadyFreed is false. Quota is decremented a second time, underflowing/corrupting active VM counts and enabling tenants to bypass TenantHardCap.
• Node occupancy leak: Before deleting the VM, Terminate unconditionally transitions the VM to VM_STATE_DRAINING then to VM_STATE_TERMINATED. In cluster/state.go, UpdateVMState decrements node occupancy (n.RunningVms--) on any transition from a non-terminal state (DRAINING) to a terminal state (TERMINATED). Transitioning an already-terminal VM back through DRAINING causes n.RunningVms on the node to decrement a second time, skewing node load scoring.
• Fix: Check isTerminalVMState(vm.State) (or vm.State == VM_STATE_FAILED || vm.State == VM_STATE_TERMINATED). If already terminal, skip state update to DRAINING, skip the node stop, and do not decrement tenant quota.

---

2. Incomplete R1 fix: ScheduleRunInMicroVM drops egress_rules and secrets (runtime.go)

• Location: services/control-plane/internal/handlers/runtime.go:710-740 (ScheduleRunInMicroVM)
• Issue: Finding R1 asserts that limits, egress_rules, and secrets were dropped when building AgentSpec. The PR added decoding in agentSpecFromMap, but ScheduleRunInMicroVM (the execution path for agent runs and recovery re-drives) still never maps manifest["egress_rules"] or manifest["secrets"] into agentSpecDTO:

  if lims, ok := manifest["limits"].(map[string]any); ok {
      spec.Limits = lims
  }
  // manifest["egress_rules"] and manifest["secrets"] are completely omitted

  Consequently, specMap["egress_rules"] and specMap["secrets"] remain empty. Every agent run dispatched to a microVM still boots without egress allowlists or secret references, leaving invariant bridge: stop sending the model's no-reply reasoning to the contact #5 / invariant twilio: send via Messaging Service (RCS + automatic SMS fallback) #10 violated in production.
• Fix: Extract egress_rules/egressRules and secrets from manifest into spec.EgressRules and spec.Secrets in ScheduleRunInMicroVM.

---

Non-Blocking Edge Cases & Correctness Issues

3. Dropped camelCase egressRules in agentSpecFromMap (runtime.go)

• Location: services/control-plane/internal/handlers/runtime.go:553
• Issue: agentSpecFromMap hardcodes mapSlice(spec["egress_rules"]). While nested fields use firstOf(r, "http_methods", "httpMethods"), the top-level key only checks snake_case. If an incoming spec dictionary uses camelCase (egressRules), all egress rules are dropped.
• Fix: Use mapSlice(firstOf(spec, "egress_rules", "egressRules")).

4. runScheduleKey type switch misses int64 (runtime.go)

• Location: services/control-plane/internal/handlers/runtime.go:2617
• Issue: runScheduleKey handles int and float64, but omits int64 and json.Number. If manifest["attempt"] is stored/unmarshaled as int64, it silently defaults to attempt 1, causing crash-recovery attempts to collide with the original crashed VM's idempotency key.
• Fix: Use int64Of(manifest["attempt"]).

5. Whitespace-only service token bypasses model-router startup check (auth.rs)

• Location: services/model-router/src/auth.rs:23
• Issue: check_startup uses token.is_none_or(str::is_empty). If LANTERN_GRPC_SERVICE_TOKEN=" ", str::is_empty is false, allowing the server to start in production.
• Fix: Check token.is_none_or(|t| t.trim().is_empty()).

[dshakes]

Superseded by the rewritten stack #265–#275 (same changes; history rewritten to drop a 43 MB build binary committed by mistake; part 11 #275 fixes the confirmed Codex audit findings raised on this stack).