Skip to content

GA 6/11 · tenant-wide spend cap + runtime SDK namespaces (TS/Python) + dashboard pause/resume/fork - #270

Closed
dshakes wants to merge 5 commits into
ga2/05-manager-authzfrom
ga2/06-tenant-cap-sdk-dashboard
Closed

dshakes wants to merge 5 commits into
ga2/05-manager-authzfrom
ga2/06-tenant-cap-sdk-dashboard

Conversation

@dshakes

@dshakes dshakes commented Oct 23, 2026

Copy link
Copy Markdown
Owner

Part 6 of 11 of the GA next-gen stack. Replaces #264/#255–#263: same changes, history rewritten to drop a 43 MB build binary committed by mistake, plus part 11 fixing every confirmed Codex cross-audit finding. Base: ga2/05-manager-authz. Merge in order with a merge commit (not squash).

Context, verification and needs-human list: #254 and docs/roadmap/2026-10-nextgen-sandbox-and-harness.md.

Commits in this part:

  • merge: runtime SDK namespaces (TS/Python) + dashboard pause/resume/fork
  • merge: tenant-wide daily spend cap (C6)

@github-actions

Copy link
Copy Markdown
Contributor

🔎 Codex cross-audit (agent:audit)

Blocking Findings

  • packages/sdk-ts/src/client.ts and page.tsx: the PR exposes/calls POST /v1/runtime/vms/{id}/pause, /resume, and /fork, but the server only registers schedule/list/get/logs/delete/exec routes in main.go. There are no corresponding RuntimeHandler methods. These UI buttons and SDK methods will 404 in production.

  • budgets.go: CheckBudget returns immediately for any tenant-wide overage, even when the tenant cap is soft (HardFail=false). Callers only block on !Allowed && HardFail, so a soft tenant cap can mask a hard agent cap that would otherwise block. Example: tenant soft cap exceeded plus agent hard daily/run limit exceeded now continues instead of returning 402. Evaluate the agent budget when the tenant violation is soft, or otherwise preserve hard-fail precedence.

Other Risk

  • budgets.go: * is introduced as a reserved sentinel agent name, but agent creation only rejects empty names. A real/auto-created agent named * can collide with the tenant budget row and be hidden from ListBudgets. Add validation/reservation or use a separate table/column.

@github-actions

Copy link
Copy Markdown
Contributor

🛡️ Vuln scan — ❌ vulnerable dependency found

      #41: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.Shutdown
      #42: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.Shutdown
      #43: internal/telemetry/tracer.go:92:21: telemetry.InitTracer calls trace.TracerProvider.Shutdown
      #44: internal/handlers/runtime.go:186:50: handlers.NewGRPCSchedulerClient calls otelgrpc.NewClientHandler, which calls trace.TracerProvider.Tracer
      #45: internal/telemetry/tracer.go:79:23: telemetry.InitTracer calls trace.WithBatcher
      #46: internal/telemetry/tracer.go:80:24: telemetry.InitTracer calls trace.WithResource
      #47: internal/telemetry/tracer.go:81:23: telemetry.InitTracer calls trace.WithSampler
      #48: internal/handlers/llm_proxy.go:3950:104: handlers.LlmProxyHandler.Transcribe calls trace.errUnsupportedSampler.Error
      #49: internal/telemetry/tracer.go:17:2: telemetry.init calls trace.init
      #50: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.init
      #51: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.logDropped
      #52: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.logDropped
      #53: internal/handlers/llm_proxy.go:3825:27: handlers.LlmProxyHandler.OCR calls io.ReadAll, which eventually calls trace.nonRecordingSpan.AddEvent
      #54: internal/handlers/runtime.go:1556:2: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.End
      #55: internal/middleware/span.go:34:22: middleware.EnrichSpan calls trace.nonRecordingSpan.IsRecording
      #56: internal/handlers/runtime.go:1564:19: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.RecordError
      #57: internal/handlers/runtime.go:1555:20: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.SetAttributes
      #58: cmd/server/main.go:844:38: server.main calls http.Server.ListenAndServe, which eventually calls trace.nonRecordingSpan.SetName
      #59: internal/handlers/runtime.go:1565:17: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.SetStatus
      #60: internal/handlers/runtime.go:1554:27: handlers.RuntimeHandler.Schedule calls noop.Tracer.Start, which calls trace.nonRecordingSpan.SpanContext
      #61: internal/handlers/llm_proxy.go:3816:24: handlers.LlmProxyHandler.OCR calls http.Client.Do, which eventually calls trace.nonRecordingSpan.TracerProvider
      #62: internal/handlers/llm_proxy.go:3825:27: handlers.LlmProxyHandler.OCR calls io.ReadAll, which eventually calls trace.recordingSpan.AddEvent
      #63: internal/handlers/runtime.go:1556:2: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.End
      #64: internal/middleware/span.go:34:22: middleware.EnrichSpan calls trace.recordingSpan.IsRecording
      #65: internal/handlers/runtime.go:1564:19: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.RecordError
      #66: internal/handlers/runtime.go:1555:20: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.SetAttributes
      #67: cmd/server/main.go:844:38: server.main calls http.Server.ListenAndServe, which eventually calls trace.recordingSpan.SetName
      #68: internal/handlers/runtime.go:1565:17: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.SetStatus
      #69: internal/handlers/runtime.go:1554:27: handlers.RuntimeHandler.Schedule calls noop.Tracer.Start, which calls trace.recordingSpan.SpanContext
      #70: internal/handlers/llm_proxy.go:3816:24: handlers.LlmProxyHandler.OCR calls http.Client.Do, which eventually calls trace.recordingSpan.TracerProvider
      #71: internal/handlers/llm_proxy.go:3950:104: handlers.LlmProxyHandler.Transcribe calls trace.samplerArgParseError.Error
      #72: internal/handlers/connector_health.go:177:15: handlers.PersistConnectorQuarantine calls errors.As, which eventually calls trace.samplerArgParseError.Unwrap
      #73: internal/handlers/runtime.go:1554:27: handlers.RuntimeHandler.Schedule calls trace.tracer.Start
      #74: internal/telemetry/tracer.go:86:27: telemetry.InitTracer calls otel.SetTextMapPropagator, which eventually calls trace.tracerProviderConfig.MarshalLog
      #75: internal/telemetry/tracer.go:79:23: telemetry.InitTracer calls trace.WithBatcher, which eventually calls x.Feature[string].Enabled
      #76: internal/telemetry/tracer.go:16:2: telemetry.init calls resource.init, which calls x.init

Vulnerability #15: GO-2026-6443
    Server panic via missing authority or Host headers in google.golang.org/grpc
  More info: https://pkg.go.dev/vuln/GO-2026-6443
  Module: google.golang.org/grpc
    Found in: google.golang.org/grpc@v1.82.1
    Fixed in: google.golang.org/grpc@v1.82.2
    Example traces found:
      #1: cmd/server/main.go:263:32: server.main calls grpc.Server.Serve, which eventually calls transport.http2Server.HandleStreams

Vulnerability #16: GO-2026-6348
    Heap Memory Exhaustion (OOM) via HTTP/2 DATA Frame Fragmentation in
    google.golang.org/grpc
  More info: https://pkg.go.dev/vuln/GO-2026-6348
  Module: google.golang.org/grpc
    Found in: google.golang.org/grpc@v1.82.1
    Fixed in: google.golang.org/grpc@v1.83.1
    Example traces found:
      #1: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.BufferSlice.MaterializeToBuffer
      #2: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.IsBelowBufferPoolingThreshold
      #3: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.NewBuffer
      #4: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls mem.ReadAll
      #5: internal/handlers/llm_proxy.go:3950:104: handlers.LlmProxyHandler.Transcribe calls multierr.multiError.Error, which eventually calls mem.writer.Write
      #6: internal/handlers/runtime.go:1984:27: handlers.RuntimeHandler.StreamLogs calls grpc.GenericClientStream[github.com/dshakes/lantern/gen/go/lantern/v1.LogsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.RuntimeLogLine].Recv, which eventually calls transport.ClientStream.Close
      #7: internal/handlers/runtime.go:354:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.ClientStream.Header
      #8: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ClientStream.Read
      #9: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ClientStream.RecvCompress
      #10: internal/handlers/runtime.go:354:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.ClientStream.TrailersOnly
      #11: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.NewHTTP2Client
      #12: cmd/server/main.go:263:32: server.main calls grpc.Server.Serve, which eventually calls transport.NewServerTransport
      #13: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ServerStream.Read
      #14: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.Stream.ReadMessageHeader
      #15: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.http2Client.Close
      #16: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.http2Client.GracefulClose
      #17: internal/handlers/runtime.go:354:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.http2Client.NewStream
      #18: cmd/server/main.go:909:25: server.main calls grpc.Server.GracefulStop, which eventually calls transport.http2Server.Close
      #19: cmd/server/main.go:263:32: server.main calls grpc.Server.Serve, which eventually calls transport.http2Server.HandleStreams

Your code is affected by 16 vulnerabilities from 3 modules and the Go standard library.
This scan also found 3 vulnerabilities in packages you import and 19
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.
make: *** [Makefile:270: audit] Error 3

@compass-sdlc-bot compass-sdlc-bot Bot added the agent:needs-fix Reviewer found Blocking issues -> triggers the auto-fix loop label Oct 23, 2026
@compass-sdlc-bot

Copy link
Copy Markdown

🔒 Reviewer verdict: BLOCKING — Tenant-wide budget cap looks solid, but the new runtime pause/resume/fork feature is client-only — no matching control-plane endpoints exist, so every SDK/dashboard call will 404.

@github-actions github-actions Bot added the domain:api Touches API / SDK / public contracts label Oct 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔷 Gemini cross-audit (agent:audit-gemini)

Blocking Issues

1. Missing Backend Handlers for MicroVM Pause, Resume, and Fork

  • Files: apps/web/app/(dashboard)/runtime/[vmId]/page.tsx, packages/sdk-ts/src/client.ts, packages/sdk-python/lantern/client.py, packages/sdk-python/lantern/integrations/runtime_client.py
  • Defect: The web UI and both SDKs implement calls to:
    • POST /v1/runtime/vms/{id}/pause
    • POST /v1/runtime/vms/{id}/resume
    • POST /v1/runtime/vms/{id}/fork
      None of these endpoints exist in services/control-plane (cmd/server/main.go or internal/handlers/runtime.go). There are no RPCs or state transitions for pause/resume/fork in packages/proto/lantern/v1/runtime.proto (enum VmState has no paused), nor in runtime-scheduler or runtime-manager.
  • Impact: Every button click in the cockpit ("Pause", "Resume", "Fork") and every SDK method invocation will fail with HTTP 404 Not Found in any non-mocked environment.
  • Lineage Data Loss: Fields snapshotId and parentVmId added to VmRow / RuntimeVm / VmDetail do not exist in the runtime_vms database schema and are neither queried nor returned by GET /v1/runtime/vms/{id}.

2. Soft Tenant Budget Breach Disables All Hard-Fail Per-Agent Budgets

  • File: services/control-plane/internal/handlers/budgets.go:314-319
  • Defect: In CheckBudget:
    if tc := CheckTenantBudget(ctx, pool, tenantID, estCostUsd); !tc.Allowed {
        return tc
    }
    return checkAgentBudget(ctx, pool, tenantID, agentName, estCostUsd)
    When a tenant configures a tenant budget with hard_fail = false (soft cap) and daily spend exceeds max_cost_usd_per_day, CheckTenantBudget returns BudgetCheckResult{Allowed: false, HardFail: false, ...}.
    Because !tc.Allowed evaluates to true, CheckBudget returns tc immediately without ever calling checkAgentBudget.
  • Impact: Any caller checking budgets (rest.go for runs, sessions.go for interactive messages, voice.go for calls, llm_proxy.go for completions) checks if !bc.Allowed && bc.HardFail. Since bc.HardFail is false, execution proceeds. Exceeding a soft tenant budget bypasses all strict per-agent hard-fail limits (daily spend, cost-per-run, max runs/day), allowing runaway agent spend.
  • Fix: CheckBudget should only return tc early if tc.HardFail is true. If tc is a soft breach (!tc.Allowed && !tc.HardFail), it must continue to checkAgentBudget so agent hard-fail policies take precedence.

Correctness & Security Findings

3. Pre-run Cost Forecast (/v1/runs/forecast) Completely Ignores Tenant Budget

  • File: services/control-plane/internal/handlers/forecaster.go:250-280
  • Defect: ForecastHandler.loadBudget only queries agent_budgets for agent_name = $2 (req.AgentName) and checks usage for that specific agent. It does not invoke CheckTenantBudget.
  • Impact: When a tenant-wide hard-fail budget is exhausted, POST /v1/runs/forecast returns wouldExceedBudget: false, falsely indicating the run is permitted. The subsequent POST /v1/runs immediately fails with HTTP 402, breaking Wedge WhatsApp: agent avatar + 'my voice' persona + handoff disclosure #1 pre-run budget predictability.

4. Sentinel Collision & Mutation via Per-Agent Budget Endpoints (agent_name = "*")

  • File: services/control-plane/internal/handlers/budgets.go:22-26, 68-75, 145-155
  • Defect: TenantBudgetAgent = "*" is stored directly in agent_budgets. However, UpsertBudget (PUT /v1/agents/{name}/budget), GetBudget, and DeleteBudget do not guard against r.PathValue("name") == "*", nor does CreateAgent in agents.go restrict * as an agent name.
  • Impact:
    • A caller with standard agent budget scopes can call DELETE /v1/agents/*/budget to silently delete the tenant-wide budget.
    • A caller can call PUT /v1/agents/*/budget with {"hardFail": false} (omitting maxCostUsdPerDay), which updates the * row setting max_cost_usd_per_day = NULL and effectively disables the tenant spend limit.
    • GET /v1/agents/*/budget returns the tenant budget disguised as an agent budget.
  • Fix: Explicitly reject agentName == TenantBudgetAgent ("*") in UpsertBudget, GetBudget, DeleteBudget, and CreateAgent.

5. SSE Multiline Data Overwrite in Both SDKs

  • Files: packages/sdk-python/lantern/client.py:440-445, packages/sdk-ts/src/client.ts:655-660
  • Defect: In LanternClient._Runtime.exec_stream and rawSSEWithEvent:
    • Python: elif line.startswith("data:"): data = line[5:].strip()
    • TS: else if (line.startsWith("data:")) { pendingData = line.slice(5).trim(); }
  • Impact: Per the W3C SSE standard, consecutive data: lines within the same event block must append with a newline (\n). Reassigning the variable overwrites earlier lines, corrupting multiline payloads if chunking or proxy buffering emits multiple data: lines.

6. Missing Client & UI Surface for Tenant Budgets

  • Files: packages/sdk-ts, packages/sdk-python, apps/web/app/(dashboard)/budgets
  • Defect: While control-plane endpoints PUT/GET/DELETE /v1/budgets/tenant are introduced, neither SDK exposes client.budgets.getTenant() / client.budgets.upsertTenant(), and the dashboard budgets page has no UI to inspect or set the tenant cap.

@dshakes

dshakes commented Oct 24, 2026

Copy link
Copy Markdown
Owner Author

Superseded: the history rewrite on this stack accidentally re-signed master's commits (base diverged). Replaced by a range-limited rewrite on ga3/* — see the new GA 1/11 PR.

@dshakes dshakes closed this Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:needs-fix Reviewer found Blocking issues -> triggers the auto-fix loop domain:api Touches API / SDK / public contracts

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant