[dshakes]

Part 2 of 11 of the GA next-gen stack. Replaces #264/#255–#263: same changes, history rewritten to drop a 43 MB build binary committed by mistake, plus part 11 fixing every confirmed Codex cross-audit finding. Base: ga2/01-scheduler-modelrouter. Merge in order with a merge commit (not squash).

Context, verification and needs-human list: #254 and docs/roadmap/2026-10-nextgen-sandbox-and-harness.md.

Commits in this part:

• merge: control-plane security + cost fixes (S1 S3-S6, C1-C3, C5)

[github-actions]

🔎 Codex cross-audit (agent:audit)

Blocking

• infra/helm/lantern/values.yaml now defaults the chart to LANTERN_ENV: "production", but values-dev.yaml does not override it. The documented Kind/dev install will inherit prod mode and hit fatal startup guards for missing receipt secret, credential key, RLS app password, gRPC token, TLS, and scheduler addr in main.go. Add explicit dev env overrides in values-dev.yaml or keep the base chart non-prod and reserve prod mode for values-prod.yaml.

High

• services/control-plane/internal/handlers/api_keys.go hard-codes knownScopes without the existing runtime scopes defined in runtime.go. After this PR, POST /v1/api-keys rejects runtime:read, runtime:write, and runtime:admin as unknown, so operators cannot mint scoped service keys for the runtime APIs that enforce those scopes. Centralize scope registration or include the runtime scopes.

• The idempotency fix still collides across tool-loop turns. The new helper promises base | provider | payloadHash changes when the request body changes, but both tool callers pass the original messages, not the evolving request body: OpenAI at llm_proxy.go, Anthropic at llm_proxy.go. In a run-scoped context, turn 2 can reuse turn 1’s provider idempotency key and get a replayed first response. Use msgs/the actual Anthropic request messages for the key.

Checks

• Ran git diff --check origin/ga2/01-scheduler-modelrouter...HEAD; no whitespace errors.

[github-actions]

🛡️ Vuln scan — ❌ vulnerable dependency found

      #41: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.Shutdown
      #42: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.Shutdown
      #43: internal/telemetry/tracer.go:92:21: telemetry.InitTracer calls trace.TracerProvider.Shutdown
      #44: internal/handlers/runtime.go:186:50: handlers.NewGRPCSchedulerClient calls otelgrpc.NewClientHandler, which calls trace.TracerProvider.Tracer
      #45: internal/telemetry/tracer.go:79:23: telemetry.InitTracer calls trace.WithBatcher
      #46: internal/telemetry/tracer.go:80:24: telemetry.InitTracer calls trace.WithResource
      #47: internal/telemetry/tracer.go:81:23: telemetry.InitTracer calls trace.WithSampler
      #48: internal/handlers/llm_proxy.go:3935:104: handlers.LlmProxyHandler.Transcribe calls trace.errUnsupportedSampler.Error
      #49: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.init
      #50: internal/telemetry/tracer.go:17:2: telemetry.init calls trace.init
      #51: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.logDropped
      #52: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.logDropped
      #53: internal/handlers/llm_proxy.go:3810:27: handlers.LlmProxyHandler.OCR calls io.ReadAll, which eventually calls trace.nonRecordingSpan.AddEvent
      #54: internal/handlers/runtime.go:1556:2: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.End
      #55: internal/middleware/span.go:34:22: middleware.EnrichSpan calls trace.nonRecordingSpan.IsRecording
      #56: internal/handlers/runtime.go:1564:19: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.RecordError
      #57: internal/handlers/runtime.go:1555:20: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.SetAttributes
      #58: cmd/server/main.go:841:38: server.main calls http.Server.ListenAndServe, which eventually calls trace.nonRecordingSpan.SetName
      #59: internal/handlers/runtime.go:1565:17: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.SetStatus
      #60: internal/handlers/runtime.go:1554:27: handlers.RuntimeHandler.Schedule calls noop.Tracer.Start, which calls trace.nonRecordingSpan.SpanContext
      #61: internal/handlers/llm_proxy.go:3801:24: handlers.LlmProxyHandler.OCR calls http.Client.Do, which eventually calls trace.nonRecordingSpan.TracerProvider
      #62: internal/handlers/llm_proxy.go:3810:27: handlers.LlmProxyHandler.OCR calls io.ReadAll, which eventually calls trace.recordingSpan.AddEvent
      #63: internal/handlers/runtime.go:1556:2: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.End
      #64: internal/middleware/span.go:34:22: middleware.EnrichSpan calls trace.recordingSpan.IsRecording
      #65: internal/handlers/runtime.go:1564:19: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.RecordError
      #66: internal/handlers/runtime.go:1555:20: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.SetAttributes
      #67: cmd/server/main.go:841:38: server.main calls http.Server.ListenAndServe, which eventually calls trace.recordingSpan.SetName
      #68: internal/handlers/runtime.go:1565:17: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.SetStatus
      #69: internal/handlers/runtime.go:1554:27: handlers.RuntimeHandler.Schedule calls noop.Tracer.Start, which calls trace.recordingSpan.SpanContext
      #70: internal/handlers/llm_proxy.go:3801:24: handlers.LlmProxyHandler.OCR calls http.Client.Do, which eventually calls trace.recordingSpan.TracerProvider
      #71: internal/handlers/llm_proxy.go:3935:104: handlers.LlmProxyHandler.Transcribe calls trace.samplerArgParseError.Error
      #72: internal/handlers/connector_health.go:177:15: handlers.PersistConnectorQuarantine calls errors.As, which eventually calls trace.samplerArgParseError.Unwrap
      #73: internal/handlers/runtime.go:1554:27: handlers.RuntimeHandler.Schedule calls trace.tracer.Start
      #74: internal/telemetry/tracer.go:86:27: telemetry.InitTracer calls otel.SetTextMapPropagator, which eventually calls trace.tracerProviderConfig.MarshalLog
      #75: internal/telemetry/tracer.go:79:23: telemetry.InitTracer calls trace.WithBatcher, which eventually calls x.Feature[string].Enabled
      #76: internal/telemetry/tracer.go:16:2: telemetry.init calls resource.init, which calls x.init

Vulnerability #15: GO-2026-6443
    Server panic via missing authority or Host headers in google.golang.org/grpc
  More info: https://pkg.go.dev/vuln/GO-2026-6443
  Module: google.golang.org/grpc
    Found in: google.golang.org/grpc@v1.82.1
    Fixed in: google.golang.org/grpc@v1.82.2
    Example traces found:
      #1: cmd/server/main.go:263:32: server.main calls grpc.Server.Serve, which eventually calls transport.http2Server.HandleStreams

Vulnerability #16: GO-2026-6348
    Heap Memory Exhaustion (OOM) via HTTP/2 DATA Frame Fragmentation in
    google.golang.org/grpc
  More info: https://pkg.go.dev/vuln/GO-2026-6348
  Module: google.golang.org/grpc
    Found in: google.golang.org/grpc@v1.82.1
    Fixed in: google.golang.org/grpc@v1.83.1
    Example traces found:
      #1: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.BufferSlice.MaterializeToBuffer
      #2: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.IsBelowBufferPoolingThreshold
      #3: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.NewBuffer
      #4: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls mem.ReadAll
      #5: internal/handlers/llm_proxy.go:3935:104: handlers.LlmProxyHandler.Transcribe calls multierr.multiError.Error, which eventually calls mem.writer.Write
      #6: internal/handlers/runtime.go:1984:27: handlers.RuntimeHandler.StreamLogs calls grpc.GenericClientStream[github.com/dshakes/lantern/gen/go/lantern/v1.LogsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.RuntimeLogLine].Recv, which eventually calls transport.ClientStream.Close
      #7: internal/handlers/runtime.go:354:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.ClientStream.Header
      #8: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ClientStream.Read
      #9: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ClientStream.RecvCompress
      #10: internal/handlers/runtime.go:354:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.ClientStream.TrailersOnly
      #11: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.NewHTTP2Client
      #12: cmd/server/main.go:263:32: server.main calls grpc.Server.Serve, which eventually calls transport.NewServerTransport
      #13: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ServerStream.Read
      #14: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.Stream.ReadMessageHeader
      #15: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.http2Client.Close
      #16: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.http2Client.GracefulClose
      #17: internal/handlers/runtime.go:354:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.http2Client.NewStream
      #18: cmd/server/main.go:906:25: server.main calls grpc.Server.GracefulStop, which eventually calls transport.http2Server.Close
      #19: cmd/server/main.go:263:32: server.main calls grpc.Server.Serve, which eventually calls transport.http2Server.HandleStreams

Your code is affected by 16 vulnerabilities from 3 modules and the Go standard library.
This scan also found 3 vulnerabilities in packages you import and 19
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.
make: *** [Makefile:270: audit] Error 3

[github-actions]

🔷 Gemini cross-audit (agent:audit-gemini)

Independent Audit Report

Target: pr-diff.patch
Focus: Correctness regressions, security gaps, and edge cases.

---

🚨 Blocking Issues

1. Helm values-dev.yaml Broken by LANTERN_ENV: "production" Default (CrashLoopBackOff)

• Files: infra/helm/lantern/values.yaml, infra/helm/lantern/templates/control-plane/deployment.yaml, infra/helm/lantern/templates/workflow-engine/deployment.yaml
• Mechanism: values.yaml sets controlPlane.env.LANTERN_ENV: "production" and workflowEngine.env.LANTERN_ENV: "production". Deployment templates also set default "production".
• Bug: values-dev.yaml was not updated to explicitly opt out (LANTERN_ENV: ""). It also leaves credentialKey, receiptSecret, and grpcServiceToken empty.
• Impact: Running helm install -f values.yaml -f values-dev.yaml causes control-plane and workflow-engine to run in production mode without required production secrets. runStartupGuards() calls logger.Fatal() on missing secrets at startup, crashing both pods immediately in dev/kind environments.
• Remedy: In infra/helm/lantern/values-dev.yaml, explicitly set LANTERN_ENV: "" (or "dev") under both controlPlane.env and workflowEngine.env.

---

⚠️ Correctness & Security Findings

2. Workflow Runs Never Increment runs_count in agent_usage_daily (Budget Bypass)

• Files: services/control-plane/internal/handlers/budgets.go, services/control-plane/internal/handlers/rest.go
• Mechanism: meteredLLMCall records intra-run workflow LLM spend using RecordUsageSpend(..., runsDelta=0) under the rationale: "whose run is already counted, so runs/day budgets are not inflated".
• Bug: A workflow run is never counted anywhere in agent_usage_daily:
  • CreateRun does not touch agent_usage_daily.
  • In executeRunInlineSync, runWorkflowIfPresent handles workflow execution and returns early (return wfResult, ...). The RecordUsage call at the bottom of executeRunInlineSync (which passes runsDelta=1) is unreachable for workflow agents.
  • runWorkflowIfPresent only updates runs.status to succeeded/failed; it never calls recordUsage with runsDelta=1.
• Impact: For all workflow agents, agent_usage_daily.runs_count remains 0. Any budget policy setting max_runs_per_day is completely bypassed. Daily usage analytics will report 0 runs.
• Remedy: When a workflow completes in runWorkflowIfPresent, record the run increment: recordUsage(ctx, h.srv.Pool, tenantID, agentName, 1, 0, 0, 0, nil).

3. Privilege Escalation Gap: DELETE /v1/api-keys/{id} Missing Role Check

• Files: services/control-plane/cmd/server/main.go, services/control-plane/internal/handlers/api_keys.go
• Mechanism: DELETE /v1/api-keys/{id} is wrapped with WithScope(ScopeAdmin, ...).
• Bug: For interactive JWT users, HasScope bypasses scope checks unconditionally (if claims.Role != "service" { return true }). CreateApiKey protects against this by explicitly verifying claims.Role == "owner" || claims.Role == "admin", but RevokeApiKey omits this check.
• Impact: Any authenticated user with role "member" can call DELETE /v1/api-keys/{id} and revoke administrative or system API keys for the tenant.
• Remedy: In RevokeApiKey, add if claims.Role != "service" && claims.Role != "owner" && claims.Role != "admin" { ... return 403 }.

4. Scope Contradiction: WithScope(ScopeAdmin) Blocks svc("write") API Key Creation

• Files: services/control-plane/cmd/server/main.go, services/control-plane/internal/handlers/api_keys.go, api_keys_scope_test.go
• Mechanism: POST /v1/api-keys is guarded with WithScope(ScopeAdmin). validateGrantableScopes and its test TestValidateGrantableScopes ({"subset allowed", svc("write"), []string{"agents:write", "read"}, false}) assume a write-scoped service key can mint sub-scoped keys.
• Discrepancy: When LANTERN_AUTHZ_ENFORCE=1 is active, WithScope(ScopeAdmin) blocks any key without ScopeAdmin before CreateApiKey / validateGrantableScopes is reached. The sub-granting path for non-admin service keys is dead code under enforcement.

---

✅ Positive Observations

• LLM Idempotency Bugfix: Suffixing model and hash(messages) into llmIdempotencyKey resolves a critical bug where multi-turn tool loops under the same execution base reused the same idempotency key, causing providers to return stale Turn 1 completions.
• Tool Token Accumulation: withToolUsage properly captures tokens spent during web_search and adds them to named returns tokensIn/tokensOut in callLLMWithTools / callAnthropicWithTools.
• Slack Signature Verification: Refusing unverified commands and eliminating the devTenantID fallback in production closes an unauthenticated tenant-impersonation hole.

[dshakes]

Superseded: the history rewrite on this stack accidentally re-signed master's commits (base diverged). Replaced by a range-limited rewrite on ga3/* — see the new GA 1/11 PR.