[dshakes]

Part 5 of 11 of the GA next-gen stack. Replaces #264/#255–#263: same changes, history rewritten to drop a 43 MB build binary committed by mistake, plus part 11 fixing every confirmed Codex cross-audit finding. Base: ga2/04-harness. Merge in order with a merge commit (not squash).

Context, verification and needs-human list: #254 and docs/roadmap/2026-10-nextgen-sandbox-and-harness.md.

Commits in this part:

• merge: runtime-manager authz, spawn contract, tenant-scoped ExecTool, honest snapshot restore (ADR 0025)

[github-actions]

🔎 Codex cross-audit (agent:audit)

Blocking:

• services/runtime-scheduler/internal/dialer/manager.go: GRPCDialer still dials runtime-manager without x-lantern-service-token. This PR makes RuntimeManagerServer require that token in prod, so real scheduler Spawn, Stop, and Snapshot calls will fail with Unauthenticated, breaking VM placement.
• services/workflow-engine/cmd/server/main.go: workflow-engine also dials runtime-manager without per-RPC credentials. Prod already requires LANTERN_GRPC_SERVICE_TOKEN, but tool-call steps will now be rejected by the new manager interceptor.
• services/control-plane/internal/handlers/runtime.go: control-plane’s direct runtime-manager client for Logs/Exec similarly omits the token, so dashboard/runtime proxy paths will fail against an authenticated manager.

These all stem from the new interceptor at services/runtime-manager/src/main.rs plus token enforcement in authz.rs. The fix is to thread the existing LANTERN_GRPC_SERVICE_TOKEN into every RuntimeManager client dial, not just scheduler/model-router calls.

I did not run tests; filesystem is read-only in this audit environment.

[github-actions]

🛡️ Vuln scan — ❌ vulnerable dependency found

      #41: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.Shutdown
      #42: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.Shutdown
      #43: internal/telemetry/tracer.go:92:21: telemetry.InitTracer calls trace.TracerProvider.Shutdown
      #44: internal/handlers/runtime.go:186:50: handlers.NewGRPCSchedulerClient calls otelgrpc.NewClientHandler, which calls trace.TracerProvider.Tracer
      #45: internal/telemetry/tracer.go:79:23: telemetry.InitTracer calls trace.WithBatcher
      #46: internal/telemetry/tracer.go:80:24: telemetry.InitTracer calls trace.WithResource
      #47: internal/telemetry/tracer.go:81:23: telemetry.InitTracer calls trace.WithSampler
      #48: internal/handlers/llm_proxy.go:3935:104: handlers.LlmProxyHandler.Transcribe calls trace.errUnsupportedSampler.Error
      #49: internal/telemetry/tracer.go:17:2: telemetry.init calls trace.init
      #50: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.init
      #51: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.logDropped
      #52: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.logDropped
      #53: internal/handlers/llm_proxy.go:3810:27: handlers.LlmProxyHandler.OCR calls io.ReadAll, which eventually calls trace.nonRecordingSpan.AddEvent
      #54: internal/handlers/runtime.go:1556:2: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.End
      #55: internal/middleware/span.go:34:22: middleware.EnrichSpan calls trace.nonRecordingSpan.IsRecording
      #56: internal/handlers/runtime.go:1564:19: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.RecordError
      #57: internal/handlers/runtime.go:1555:20: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.SetAttributes
      #58: cmd/server/main.go:841:38: server.main calls http.Server.ListenAndServe, which eventually calls trace.nonRecordingSpan.SetName
      #59: internal/handlers/runtime.go:1565:17: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.SetStatus
      #60: internal/handlers/runtime.go:1554:27: handlers.RuntimeHandler.Schedule calls noop.Tracer.Start, which calls trace.nonRecordingSpan.SpanContext
      #61: internal/handlers/llm_proxy.go:3801:24: handlers.LlmProxyHandler.OCR calls http.Client.Do, which eventually calls trace.nonRecordingSpan.TracerProvider
      #62: internal/handlers/llm_proxy.go:3810:27: handlers.LlmProxyHandler.OCR calls io.ReadAll, which eventually calls trace.recordingSpan.AddEvent
      #63: internal/handlers/runtime.go:1556:2: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.End
      #64: internal/middleware/span.go:34:22: middleware.EnrichSpan calls trace.recordingSpan.IsRecording
      #65: internal/handlers/runtime.go:1564:19: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.RecordError
      #66: internal/handlers/runtime.go:1555:20: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.SetAttributes
      #67: cmd/server/main.go:841:38: server.main calls http.Server.ListenAndServe, which eventually calls trace.recordingSpan.SetName
      #68: internal/handlers/runtime.go:1565:17: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.SetStatus
      #69: internal/handlers/runtime.go:1554:27: handlers.RuntimeHandler.Schedule calls noop.Tracer.Start, which calls trace.recordingSpan.SpanContext
      #70: internal/handlers/llm_proxy.go:3801:24: handlers.LlmProxyHandler.OCR calls http.Client.Do, which eventually calls trace.recordingSpan.TracerProvider
      #71: internal/handlers/llm_proxy.go:3935:104: handlers.LlmProxyHandler.Transcribe calls trace.samplerArgParseError.Error
      #72: internal/handlers/connector_health.go:177:15: handlers.PersistConnectorQuarantine calls errors.As, which eventually calls trace.samplerArgParseError.Unwrap
      #73: internal/handlers/runtime.go:1554:27: handlers.RuntimeHandler.Schedule calls trace.tracer.Start
      #74: internal/telemetry/tracer.go:86:27: telemetry.InitTracer calls otel.SetTextMapPropagator, which eventually calls trace.tracerProviderConfig.MarshalLog
      #75: internal/telemetry/tracer.go:79:23: telemetry.InitTracer calls trace.WithBatcher, which eventually calls x.Feature[string].Enabled
      #76: internal/telemetry/tracer.go:16:2: telemetry.init calls resource.init, which calls x.init

Vulnerability #15: GO-2026-6443
    Server panic via missing authority or Host headers in google.golang.org/grpc
  More info: https://pkg.go.dev/vuln/GO-2026-6443
  Module: google.golang.org/grpc
    Found in: google.golang.org/grpc@v1.82.1
    Fixed in: google.golang.org/grpc@v1.82.2
    Example traces found:
      #1: cmd/server/main.go:263:32: server.main calls grpc.Server.Serve, which eventually calls transport.http2Server.HandleStreams

Vulnerability #16: GO-2026-6348
    Heap Memory Exhaustion (OOM) via HTTP/2 DATA Frame Fragmentation in
    google.golang.org/grpc
  More info: https://pkg.go.dev/vuln/GO-2026-6348
  Module: google.golang.org/grpc
    Found in: google.golang.org/grpc@v1.82.1
    Fixed in: google.golang.org/grpc@v1.83.1
    Example traces found:
      #1: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.BufferSlice.MaterializeToBuffer
      #2: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.IsBelowBufferPoolingThreshold
      #3: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.NewBuffer
      #4: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls mem.ReadAll
      #5: internal/handlers/llm_proxy.go:3935:104: handlers.LlmProxyHandler.Transcribe calls multierr.multiError.Error, which eventually calls mem.writer.Write
      #6: internal/handlers/runtime.go:1984:27: handlers.RuntimeHandler.StreamLogs calls grpc.GenericClientStream[github.com/dshakes/lantern/gen/go/lantern/v1.LogsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.RuntimeLogLine].Recv, which eventually calls transport.ClientStream.Close
      #7: internal/handlers/runtime.go:354:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.ClientStream.Header
      #8: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ClientStream.Read
      #9: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ClientStream.RecvCompress
      #10: internal/handlers/runtime.go:354:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.ClientStream.TrailersOnly
      #11: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.NewHTTP2Client
      #12: cmd/server/main.go:263:32: server.main calls grpc.Server.Serve, which eventually calls transport.NewServerTransport
      #13: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ServerStream.Read
      #14: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.Stream.ReadMessageHeader
      #15: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.http2Client.Close
      #16: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.http2Client.GracefulClose
      #17: internal/handlers/runtime.go:354:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.http2Client.NewStream
      #18: cmd/server/main.go:906:25: server.main calls grpc.Server.GracefulStop, which eventually calls transport.http2Server.Close
      #19: cmd/server/main.go:263:32: server.main calls grpc.Server.Serve, which eventually calls transport.http2Server.HandleStreams

Your code is affected by 16 vulnerabilities from 3 modules and the Go standard library.
This scan also found 3 vulnerabilities in packages you import and 19
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.
make: *** [Makefile:270: audit] Error 3

[github-actions]

🔷 Gemini cross-audit (agent:audit-gemini)

Blocking Issues

1. Production Startup / Call Deadlock in RuntimeManager Service Auth (authz.rs)

• Files: services/runtime-manager/src/authz.rs, services/runtime-manager/src/main.rs, services/runtime-scheduler/internal/dialer/manager.go, services/control-plane/internal/handlers/runtime.go, services/workflow-engine/cmd/server/main.go
• Problem: In services/runtime-manager/src/authz.rs, ManagerAuth::from_parts enforces that in production (LANTERN_ENV=prod/production/staging), LANTERN_GRPC_SERVICE_TOKEN must be set or runtime-manager terminates immediately with anyhow::bail!. When the token is configured, ManagerAuth intercepts every RPC on RuntimeManagerServer and requires x-lantern-service-token metadata. However, none of the three clients dialing RuntimeManager send x-lantern-service-token:
  • services/runtime-scheduler (GRPCDialer.connFor): connects with cleartext insecure.NewCredentials() without credentials or token metadata.
  • services/control-plane (grpcSchedulerClient.managerClient): connects with insecure.NewCredentials() and no credentials.
  • services/workflow-engine (cmd/server/main.go): dials cfg.RuntimeManagerAddr with insecure.NewCredentials() without passing serviceTokenCreds (even though modelClient right above it does pass serviceTokenCreds).
• Impact: In production (LANTERN_ENV=prod/production/staging), runtime-manager will either crash on boot (if token is unset) or reject 100% of incoming RPCs from the scheduler (Spawn, Stop, Snapshot), control plane (Exec), and workflow engine (ExecTool) with Status::unauthenticated.

---

2. Protocol Field Name Mismatch in ExecTool Harness Envelope Breaks All Tool Invocations

• Files: services/runtime-manager/src/service.rs, services/harness/src/tool_runner.rs
• Problem: In service.rs (run_tool_via_harness_inner), the JSON payload sent via the in-guest harness Exec stream (command: "__lantern_tool_call__") uses field name "tool_name":

  let envelope = serde_json::json!({
      "tool_name": tool_name,
      "args": args_json,
      "idempotency_key": idempotency_key,
      "timeout_sec": timeout_sec,
  });

  In services/harness/src/tool_runner.rs, ToolEnvelope strictly deserializes field "tool":

  #[derive(Debug, Deserialize)]
  pub struct ToolEnvelope {
      pub tool: String,
      ...
  }

  ToolEnvelope does not specify #[serde(alias = "tool_name")].
• Impact: serde_json::from_str::<ToolEnvelope> in harness/src/tool_runner.rs fails with missing field 'tool', returning Status::invalid_argument. Every ExecTool call through run_tool_via_harness fails with exec_tool: stream error from harness.

---

Non-Blocking Issues & Regressions

3. Idempotent Spawn Retries Fail with Spurious ABORTED Under Contention (service.rs)

• File: services/runtime-manager/src/service.rs
• Problem: In spawn(), self.registry.reserve(id) is acquired before checking self.registry.get(&h.vm_id).is_some().
• Scenario: If a VM vm-1 has already finished booting and is registered in self.registry, two concurrent retries or polling requests for vm-1 will contend on self.registry.reserve("vm-1"). The winner checks get() and returns Ok(handle), but the second request fails reserve() and returns Status::aborted("spawn for vm_id vm-1 is already in progress; retry").
• Fix: Check self.registry.get(&h.vm_id) before reserving, and re-check after reserving (double-checked locking), so retries for already-active VMs return immediately without contending on the reservation set.

---

4. Linear Unindexed S3 Bucket Scan on SnapshotStore::find (snapshot_store.rs, backends/firecracker.rs)

• Files: services/runtime-manager/src/snapshot_store.rs, services/runtime-manager/src/backends/firecracker.rs
• Problem: SnapshotStore::find(&self, snapshot_id: &str) falls back to S3 via s3.list(None), iterating every object in the bucket to find a key ending in /{snapshot_id}/meta.json. Similarly, fetch_snapshot_from_s3 in firecracker.rs issues s3.list(Some(&prefix)).try_collect().await? which loads all snapshot keys into a single in-memory Vec<String>.
• Impact: On a node that does not have a snapshot locally cached, every microVM restore performs a linear scan over all snapshots across all tenants/agents in S3. This causes significant latency, memory pressure, and high S3 LIST costs. Note that AgentSpec already carries agent_version_id, which should be used to prefix S3 queries.

---

5. Bare Firecracker Snapshots Leak Memory Dumps in tmpfs (backends/firecracker.rs)

• File: services/runtime-manager/src/backends/firecracker.rs
• Problem: FirecrackerBackend::snapshot writes snapshots to /run/lantern/snapshots/{handle_id}/. SnapshotStore::put_files copies those files into $SNAPSHOT_DIR. When jailing is disabled (self.jailer is None), cancel() does not clean up /run/lantern/snapshots/{handle_id}.
• Impact: Because /run is typically a tmpfs mount, full guest memory files (mem, 512MiB–2GiB) remain in RAM indefinitely on bare launches, leading to memory exhaustion.

---

6. Cross-Tenant Run ID Ambiguity in ExecTool Resolution (service.rs)

• File: services/runtime-manager/src/service.rs
• Problem: When req.vm_id is empty, exec_tool resolves the VM using self.registry.find_by_run_id(&req.run_id), which scans without tenant scoping. If two tenants have the same run_id, find_by_run_id may select another tenant's VM, fail the exec_tool_tenant_ok check, and return NOT_FOUND even if the requesting tenant owns an active VM for that run_id.
• Fix: Scope find_by_run_id by (tenant_id, run_id) or pass req.tenant_id to the registry query.

[dshakes]

Superseded: the history rewrite on this stack accidentally re-signed master's commits (base diverged). Replaced by a range-limited rewrite on ga3/* — see the new GA 1/11 PR.