[dshakes]

Part 8 of 11 of the GA next-gen stack (supersedes #255–#275). History rewritten (range-limited, master untouched) to drop a 43 MB build binary committed by mistake; part 11 fixes every confirmed Codex cross-audit finding. Base: ga3/07-web-deps-docs. Merge in order with a merge commit (not squash).

Context, verification and needs-human list: #254 and docs/roadmap/2026-10-nextgen-sandbox-and-harness.md.

Commits in this part:

• merge: VM identity = wire vm_id, re-identify on restore, per-incarnation exec tokens, fail-closed bare fork

[github-actions]

🔎 Codex cross-audit (agent:audit)

Blocking

• services/runtime-manager/src/service.rs: Jailed forks of a still-running source are allowed, but the stale-cert protection does not work in that case. The fork initially presents the source VM’s still-current cert, so cert_matches_pin passes for the source registry entry in Heartbeat/VendSecret (heartbeat, VendSecret). Until re-identify wins the race, the fork can poison harness_addrs for the source VM and potentially vend any source-declared secret. This violates the “one identity per VM” claim for live-source forks. I would fail closed for all live-source forks, or mark the source cert/pin superseded before allowing the fork to resume.

High

• services/harness/src/main.rs: egress fail-closed preflight only runs at cold boot. On restore, reidentify just swaps rules, but it cannot inject proxy env into the already-running workload and does not re-run enforcement_preflight. A snapshot taken with no egress rules can be restored/forked with new egress rules on an image lacking iptables REDIRECT, bypassing the prod fail-closed behavior described in egress.rs. Restore should reject rule changes unless REDIRECT is present, or enforce that at manager restore time.

• services/harness/src/reidentify.rs: network re-address failure is logged but treated as non-fatal, after which the harness still swaps identity and tears down manager streams. For a fork with a new /30, failing to update eth0/default route likely leaves the VM unable to reach the manager, while secrets have already been reset and the workload keeps running. This should probably abort/retry re-identify before mutating identity, not proceed.

Medium

• services/runtime-manager/src/backends/firecracker.rs: restore unconditionally sends network_overrides; there is no Firecracker version/capability check or clearer preflight. Older VMMs will reject PUT /snapshot/load at runtime. If GA supports variable Firecracker versions, gate this with a version probe or fail early with an actionable error.

I could not run cargo check: rustup tried to create files under /home/runner/.rustup/tmp, but the sandbox is read-only.

[github-actions]

🛡️ Vuln scan — ❌ vulnerable dependency found

      #41: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.Shutdown
      #42: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.Shutdown
      #43: internal/telemetry/tracer.go:92:21: telemetry.InitTracer calls trace.TracerProvider.Shutdown
      #44: internal/handlers/runtime.go:186:50: handlers.NewGRPCSchedulerClient calls otelgrpc.NewClientHandler, which calls trace.TracerProvider.Tracer
      #45: internal/telemetry/tracer.go:79:23: telemetry.InitTracer calls trace.WithBatcher
      #46: internal/telemetry/tracer.go:80:24: telemetry.InitTracer calls trace.WithResource
      #47: internal/telemetry/tracer.go:81:23: telemetry.InitTracer calls trace.WithSampler
      #48: internal/handlers/llm_proxy.go:3950:104: handlers.LlmProxyHandler.Transcribe calls trace.errUnsupportedSampler.Error
      #49: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.init
      #50: internal/telemetry/tracer.go:17:2: telemetry.init calls trace.init
      #51: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.logDropped
      #52: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls trace.logDropped
      #53: internal/handlers/llm_proxy.go:3825:27: handlers.LlmProxyHandler.OCR calls io.ReadAll, which eventually calls trace.nonRecordingSpan.AddEvent
      #54: internal/handlers/runtime.go:1556:2: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.End
      #55: internal/middleware/span.go:34:22: middleware.EnrichSpan calls trace.nonRecordingSpan.IsRecording
      #56: internal/handlers/runtime.go:1564:19: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.RecordError
      #57: internal/handlers/runtime.go:1555:20: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.SetAttributes
      #58: cmd/server/main.go:844:38: server.main calls http.Server.ListenAndServe, which eventually calls trace.nonRecordingSpan.SetName
      #59: internal/handlers/runtime.go:1565:17: handlers.RuntimeHandler.Schedule calls trace.nonRecordingSpan.SetStatus
      #60: internal/handlers/runtime.go:1554:27: handlers.RuntimeHandler.Schedule calls noop.Tracer.Start, which calls trace.nonRecordingSpan.SpanContext
      #61: internal/handlers/llm_proxy.go:3816:24: handlers.LlmProxyHandler.OCR calls http.Client.Do, which eventually calls trace.nonRecordingSpan.TracerProvider
      #62: internal/handlers/llm_proxy.go:3825:27: handlers.LlmProxyHandler.OCR calls io.ReadAll, which eventually calls trace.recordingSpan.AddEvent
      #63: internal/handlers/runtime.go:1556:2: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.End
      #64: internal/middleware/span.go:34:22: middleware.EnrichSpan calls trace.recordingSpan.IsRecording
      #65: internal/handlers/runtime.go:1564:19: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.RecordError
      #66: internal/handlers/runtime.go:1555:20: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.SetAttributes
      #67: cmd/server/main.go:844:38: server.main calls http.Server.ListenAndServe, which eventually calls trace.recordingSpan.SetName
      #68: internal/handlers/runtime.go:1565:17: handlers.RuntimeHandler.Schedule calls trace.recordingSpan.SetStatus
      #69: internal/handlers/runtime.go:1554:27: handlers.RuntimeHandler.Schedule calls noop.Tracer.Start, which calls trace.recordingSpan.SpanContext
      #70: internal/handlers/llm_proxy.go:3816:24: handlers.LlmProxyHandler.OCR calls http.Client.Do, which eventually calls trace.recordingSpan.TracerProvider
      #71: internal/handlers/llm_proxy.go:3950:104: handlers.LlmProxyHandler.Transcribe calls trace.samplerArgParseError.Error
      #72: internal/handlers/connector_health.go:177:15: handlers.PersistConnectorQuarantine calls errors.As, which eventually calls trace.samplerArgParseError.Unwrap
      #73: internal/handlers/runtime.go:1554:27: handlers.RuntimeHandler.Schedule calls trace.tracer.Start
      #74: internal/telemetry/tracer.go:86:27: telemetry.InitTracer calls otel.SetTextMapPropagator, which eventually calls trace.tracerProviderConfig.MarshalLog
      #75: internal/telemetry/tracer.go:79:23: telemetry.InitTracer calls trace.WithBatcher, which eventually calls x.Feature[string].Enabled
      #76: internal/telemetry/tracer.go:16:2: telemetry.init calls resource.init, which calls x.init

Vulnerability #15: GO-2026-6443
    Server panic via missing authority or Host headers in google.golang.org/grpc
  More info: https://pkg.go.dev/vuln/GO-2026-6443
  Module: google.golang.org/grpc
    Found in: google.golang.org/grpc@v1.82.1
    Fixed in: google.golang.org/grpc@v1.82.2
    Example traces found:
      #1: cmd/server/main.go:263:32: server.main calls grpc.Server.Serve, which eventually calls transport.http2Server.HandleStreams

Vulnerability #16: GO-2026-6348
    Heap Memory Exhaustion (OOM) via HTTP/2 DATA Frame Fragmentation in
    google.golang.org/grpc
  More info: https://pkg.go.dev/vuln/GO-2026-6348
  Module: google.golang.org/grpc
    Found in: google.golang.org/grpc@v1.82.1
    Fixed in: google.golang.org/grpc@v1.83.1
    Example traces found:
      #1: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.BufferSlice.MaterializeToBuffer
      #2: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.IsBelowBufferPoolingThreshold
      #3: internal/handlers/runs.go:447:25: handlers.RunService.StreamRunEvents calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.StreamRunEventsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.StreamEvent].Send, which eventually calls mem.NewBuffer
      #4: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls mem.ReadAll
      #5: internal/handlers/llm_proxy.go:3950:104: handlers.LlmProxyHandler.Transcribe calls multierr.multiError.Error, which eventually calls mem.writer.Write
      #6: internal/handlers/runtime.go:1984:27: handlers.RuntimeHandler.StreamLogs calls grpc.GenericClientStream[github.com/dshakes/lantern/gen/go/lantern/v1.LogsRequest, github.com/dshakes/lantern/gen/go/lantern/v1.RuntimeLogLine].Recv, which eventually calls transport.ClientStream.Close
      #7: internal/handlers/runtime.go:354:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.ClientStream.Header
      #8: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ClientStream.Read
      #9: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ClientStream.RecvCompress
      #10: internal/handlers/runtime.go:354:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.ClientStream.TrailersOnly
      #11: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.NewHTTP2Client
      #12: cmd/server/main.go:263:32: server.main calls grpc.Server.Serve, which eventually calls transport.NewServerTransport
      #13: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.ServerStream.Read
      #14: internal/handlers/dataplane.go:519:30: handlers.DataPlaneService.RunStream calls grpc.GenericServerStream[github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamClientMsg, github.com/dshakes/lantern/gen/go/lantern/v1.DpRunStreamServerMsg].Recv, which eventually calls transport.Stream.ReadMessageHeader
      #15: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.http2Client.Close
      #16: internal/handlers/schedules.go:237:33: handlers.RESTHandler.UpdateSchedule calls time.LoadLocation, which eventually calls transport.http2Client.GracefulClose
      #17: internal/handlers/runtime.go:354:28: handlers.pumpExec calls grpc.clientStream.CloseSend, which eventually calls transport.http2Client.NewStream
      #18: cmd/server/main.go:909:25: server.main calls grpc.Server.GracefulStop, which eventually calls transport.http2Server.Close
      #19: cmd/server/main.go:263:32: server.main calls grpc.Server.Serve, which eventually calls transport.http2Server.HandleStreams

Your code is affected by 16 vulnerabilities from 3 modules and the Go standard library.
This scan also found 3 vulnerabilities in packages you import and 19
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.
make: *** [Makefile:270: audit] Error 3

[github-actions]

🔷 Gemini cross-audit (agent:audit-gemini)

Audit Summary

The patch introduces critical architecture for microVM pause, resume, and fork by re-keying Firecracker microVMs at boot to the scheduler wire vm_id and allowing restored guests to dynamically re-identify via an ext4 cert-drive swap.

However, there are two blocking correctness/security issues and several missed edge cases in services/runtime-manager and services/harness:

---

Blocking Issues

1. Live VM Forking Steers Source VM's Exec / ExecTool Traffic to Fork Child (Exec Hijacking Race)

• Location: services/runtime-manager/src/service.rs (RuntimeHarnessGrpc::heartbeat & route_exec)
• Mechanism:
  When a live VM (vm-src) is forked into a new VM (vm-fork), the fork child begins execution with vm-src's memory state and original client cert. Before the in-guest watcher detects the ext4 drive change (up to a ~1s poll interval):
  1. The child's harness sends a HeartbeatRequest with vm_id = "vm-src" using vm-src's client certificate.
  2. In RuntimeHarnessGrpc::heartbeat:
     • authorize_vm_cert("vm-src", peer_cert) passes (CN is "vm-src").
     • registry.get("vm-src") returns vm-src's HandleInfo (the source is still running).
     • cert_matches_pin(info.cert_sha256, peer_cert) passes because peer_cert is vm-src's original cert, which matches vm-src's recorded pin in the registry.
  3. The manager updates harness_addrs.insert("vm-src", fork_child_ip).
• Impact: Any operator Exec or workflow ExecTool targeting vm-src is routed to the fork child instead of the source VM. Furthermore, the manager sends vm-src's exec token, which the fork child (still holding vm-src's exec token in memory) accepts, causing mutations and commands to execute inside the wrong VM.
• Remediation: Do not trust incoming peer_ip to overwrite harness_addrs for a VM when the peer IP does not match the allocated point-to-point network for that HandleInfo. Alternatively, verify peer_ip == info.expected_guest_ip or drop heartbeats whose remote address does not belong to the interface assigned to that vm_id.

2. Stale IP in harness_addrs on Pause/Resume Causes Exec/ExecTool to Dial Dead or Reallocated IP

• Location: services/runtime-manager/src/service.rs (stop & route_exec)
• Mechanism:
  When a VM is stopped (e.g., during Pause: Snapshot + Stop), RuntimeManagerGrpc::stop calls self.registry.deregister(&req.vm_id), but self.harness_addrs is never cleared.
  1. When the VM is later resumed with the same wire vm_id, VmNetwork::allocate() assigns a new /30 subnet and a new guest IP.
  2. If an Exec or ExecTool request arrives right after resume (before the in-guest harness establishes its new heartbeat stream), route_exec calls self.harness_addrs.get(vm_id).
  3. Instead of returning failed_precondition: harness not connected yet (which signals the caller to retry), route_exec retrieves the stale guest IP from the previous incarnation.
• Impact: The manager dials the old IP, resulting in immediate connection failures, timeouts, or dialing an unrelated VM if the subnet was recycled.
• Remediation: Remove vm_id from both self.harness_addrs and self.heartbeat_cache on stop / cancel (and defensively upon registering a new incarnation in schedule/restore).

---

High & Medium Issues

3. Bare-Mode Collision Guard Misses Concurrent/Successive Forks of the Same Snapshot

• Location: services/runtime-manager/src/service.rs (restore_owned_snapshot, line 689)
• Mechanism:

  if backend.restore_needs_stopped_source() && self.registry.get(&meta.vm_id).is_some() { ... }

  This check only verifies whether the original source VM (meta.vm_id) is currently running.
  In bare Firecracker, the snapshot pins the host vsock UDS path (/run/lantern/vsock/{meta.vm_id}.sock). If a paused/stopped VM is forked with count > 1, or if a second fork is started while the first fork is running:
  • self.registry.get(&meta.vm_id) is None (the source is stopped).
  • Both forks are admitted on the same node in bare mode.
  • The second fork attempts to bind /run/lantern/vsock/{meta.vm_id}.sock, colliding with the first fork (EADDRINUSE).
• Remediation: In bare mode, check if any active VM in the registry is already restoring from or running with snapshot_source_vm == meta.vm_id, or refuse multiple bare restores from the same snapshot.

4. Leaked Incarnation on TAP Setup Failure in restore_with_identity

• Location: services/runtime-manager/src/backends/firecracker.rs (restore_with_identity)
• Mechanism:

  self.provision_identity(&cfg).await?;
  setup_tap_device(
      &cfg.tap_dev,
      self.bridge_name.as_deref(),
      Some(&cfg.network.host_ip),
  ).await?;

  provision_identity sets self.incarnations[&cfg.vm_id]. If setup_tap_device fails, the ? returns early without clearing the entry. Unlike boot_vm and restore_after_tap, restore_with_identity does not call self.set_incarnation(&vm_id, None) on failure.
• Remediation: Wrap setup_tap_device in error handling that clears the incarnation on failure.

5. In-Guest replace_default_route Silently Swallows EEXIST when Deletion Misses

• Location: services/harness/src/reidentify.rs (netcfg::replace_default_route)
• Mechanism:
  SIOCDELRT is called with rt_flags = RTF_UP without RTF_GATEWAY or an interface specification. If the kernel did not automatically flush the old gateway route upon SIOCSIFADDR, SIOCDELRT will not match the existing default gateway route. Subsequently, SIOCADDRT fails with EEXIST, which is swallowed (Err(e) if e.raw_os_error() == Some(libc::EEXIST) => Ok(())).
• Impact: The function returns Ok(()) while the guest's default route still points to the old, unreachable host gateway.
• Remediation: Specify rt_dev or delete the existing default route with RTF_UP | RTF_GATEWAY, and avoid treating EEXIST on route add as success unless verified to point to the new gateway.

---

Recommendation

Do not merge in current form. Address Blocking Issues #1 and #2 (clearing harness_addrs on deregistration and preventing the live-fork heartbeat IP overwrite) before deploying.

[dshakes]

Superseded by the reordered parts #287–#292 (server-side pause/resume/fork now lands before the SDK/dashboard that call it; final tree identical).