fix(runtime): reclaim TAP devices orphaned by a crashed manager

[dshakes]

Closes the last TAP leak path — the one the earlier teardown fixes structurally could not cover.

The gap

cancel() removes a VM's TAP, but a manager that is SIGKILLed or crashes never runs it. The device — and the /30 assigned to it — then stays on the host forever: the subnet allocator skips that block, and orphans accumulate until the host has duplicate routes for the same network and replies leave via a dead device.

No in-process cleanup can fix this, because the process is gone. So the manager now sweeps once at startup, before any VM is scheduled.

The safety question

Not deleting a TAP that a live VM still uses — that would cut its only link.

A TAP whose VMM is alive holds its carrier and reports UP; when the process goes away the carrier drops and it falls to DOWN. Only DOWN devices with the fc- prefix are reclaimed, so a running VM's TAP is never touched, and neither is any host networking.

Parsing is split into a pure orphaned_tap_names so that rule is directly testable:

• a live (UP) TAP is never selected
• non-Lantern devices (lo, eth0, docker0, tap0) are ignored
• dev@if7 suffixes and malformed lines are survivable

Verified on real KVM — both directions

Scenario	Result
2 planted DOWN taps + a non-Lantern nottap0	found 2, removed 2; nottap0 untouched
VM left running, manager SIGKILLed, then restarted	removed nothing — the VM's TAP and its firecracker process both still alive

The second is the one that matters: it's the exact scenario reconciliation could get catastrophically wrong.

350 manager tests pass (4 new), clippy and rustfmt clean.

[github-actions]

🔎 Codex cross-audit (agent:audit)

Blocking

• firecracker.rs: startup reconciliation deletes every fc-* TAP in DOWN state, but fc- is only a naming convention, not an ownership proof. A runtime-manager running with CAP_NET_ADMIN can remove unrelated host TAPs from other Firecracker/userspace networking. This is amplified by main.rs, where cleanup runs before FirecrackerBackend::new() checks that this node is actually a valid Firecracker node. Require a Lantern-owned marker, a stricter persisted registry, or at least a verifiable alias/address/path check before deletion.

Non-blocking

• firecracker.rs: reconcile_orphaned_taps increments removed when teardown_tap_device returns Ok, but teardown_tap_device logs “could NOT be removed” and still returns Ok. The startup log can report successful cleanup for devices that remain leaked, masking the allocator problem this PR is meant to fix.

I did not run tests; this was a read-only audit of git diff origin/master...HEAD.

[github-actions]

🧪 QA — ❌ tests failing

ok  	github.com/dshakes/lantern/services/control-plane/internal/agentidentity	1.036s
ok  	github.com/dshakes/lantern/services/control-plane/internal/db	1.014s
ok  	github.com/dshakes/lantern/services/control-plane/internal/grounding	1.010s
ok  	github.com/dshakes/lantern/services/control-plane/internal/handlers	1.320s
ok  	github.com/dshakes/lantern/services/control-plane/internal/middleware	1.019s
ok  	github.com/dshakes/lantern/services/control-plane/internal/scheduler	1.025s
ok  	github.com/dshakes/lantern/services/control-plane/internal/secrets	1.013s
?   	github.com/dshakes/lantern/services/control-plane/internal/server	[no test files]
?   	github.com/dshakes/lantern/services/control-plane/internal/storage	[no test files]
ok  	github.com/dshakes/lantern/services/control-plane/internal/telemetry	1.013s
ok  	github.com/dshakes/lantern/services/control-plane/internal/workflow	1.022s
cd services/workflow-engine && go test -race -count=1 ./...
go: downloading github.com/jackc/pgx/v5 v5.7.4
go: downloading go.opentelemetry.io/otel v1.43.0
go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20260414002931-afd174a4e478
go: downloading golang.org/x/net v0.53.0
go: downloading golang.org/x/sys v0.43.0
go: downloading golang.org/x/crypto v0.50.0
go: downloading go.opentelemetry.io/otel/metric v1.43.0
go: downloading go.opentelemetry.io/otel/trace v1.43.0
?   	github.com/dshakes/lantern/services/workflow-engine/cmd/server	[no test files]
?   	github.com/dshakes/lantern/services/workflow-engine/internal/db	[no test files]
ok  	github.com/dshakes/lantern/services/workflow-engine/internal/engine	1.034s
?   	github.com/dshakes/lantern/services/workflow-engine/internal/handlers	[no test files]
ok  	github.com/dshakes/lantern/services/workflow-engine/internal/journal	1.013s
ok  	github.com/dshakes/lantern/services/workflow-engine/internal/middleware	1.014s
?   	github.com/dshakes/lantern/services/workflow-engine/internal/server	[no test files]
cd services/scheduler && go test -race -count=1 ./...
?   	github.com/dshakes/lantern/services/scheduler/cmd/server	[no test files]
ok  	github.com/dshakes/lantern/services/scheduler/internal/cron	1.015s
?   	github.com/dshakes/lantern/services/scheduler/internal/db	[no test files]
ok  	github.com/dshakes/lantern/services/scheduler/internal/handlers	1.030s
?   	github.com/dshakes/lantern/services/scheduler/internal/middleware	[no test files]
ok  	github.com/dshakes/lantern/services/scheduler/internal/runclient	1.035s
?   	github.com/dshakes/lantern/services/scheduler/internal/server	[no test files]
cd services/runtime-scheduler && CGO_ENABLED=0 go test ./...
go: downloading github.com/prometheus/client_golang v1.23.2
go: downloading github.com/prometheus/client_model v0.6.2
go: downloading github.com/prometheus/common v0.66.1
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/prometheus/procfs v0.16.1
go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822
go: downloading go.yaml.in/yaml/v2 v2.4.2
?   	github.com/dshakes/lantern/services/runtime-scheduler/cmd/scheduler	[no test files]
ok  	github.com/dshakes/lantern/services/runtime-scheduler/internal/cluster	0.003s
ok  	github.com/dshakes/lantern/services/runtime-scheduler/internal/dialer	0.006s
ok  	github.com/dshakes/lantern/services/runtime-scheduler/internal/handlers	0.059s
ok  	github.com/dshakes/lantern/services/runtime-scheduler/internal/leader	0.006s
ok  	github.com/dshakes/lantern/services/runtime-scheduler/internal/metrics	0.017s
ok  	github.com/dshakes/lantern/services/runtime-scheduler/internal/middleware	0.007s
ok  	github.com/dshakes/lantern/services/runtime-scheduler/internal/placement	0.005s
ok  	github.com/dshakes/lantern/services/runtime-scheduler/internal/scoring	0.004s
ok  	github.com/dshakes/lantern/services/runtime-scheduler/internal/store	0.003s
cd services/gateway && cargo test
info: syncing channel updates for 1.93-x86_64-unknown-linux-gnu
info: latest update on 2026-02-12 for version 1.93.1 (01f6ddf75 2026-02-11)
info: downloading component clippy
error: 'cargo' is not installed for the toolchain '1.93-x86_64-unknown-linux-gnu'.
help: run `rustup component add --toolchain 1.93-x86_64-unknown-linux-gnu cargo` to install it
make: *** [Makefile:186: test-rust] Error 1

[github-actions]

🔷 Gemini cross-audit (agent:audit-gemini)

Audit Report: Startup Reconciliation of Orphaned TAP Devices

---

Blocking Issues

• None. The implementation is logically sound for a standard single-instance Linux host, incorporates safe process execution patterns (avoiding shell injection risks), and provides solid unit-test coverage.

---

Correctness, Performance & Robustness Bottlenecks

1. Unnecessary Retries and Startup Stalls on Orphan Reconciliation (Performance / Robustness)

• The Issue: reconcile_orphaned_taps iterates over detected orphaned TAPs and calls teardown_tap_device(tap).await sequentially. teardown_tap_device has a hardcoded retry loop (10 attempts, 150ms delay) to handle races with a terminating VMM process holding the TAP file descriptor. However, during startup reconciliation, the VMM is already dead, and any failure to delete the TAP is almost certainly permanent (e.g., permission denied, mismatched capabilities, or system lock).
• The Impact: If a host has accumulated many orphans (e.g., dozens from a prior crashed manager) or if permissions are incorrect, the manager will uselessly loop and sleep for 1.5 seconds per TAP. This sequential blocking can stall the runtime-manager startup for tens of seconds or minutes, delaying the boot of any new scheduled microVMs.
• Recommendation:
  • Pass a retry: bool flag to teardown_tap_device and set it to false during startup reconciliation (fast-path delete).
  • Alternatively, execute the startup teardowns concurrently using futures::future::join_all so that multiple permanent failures do not block sequentially.

---

Missed Edge Cases

2. Race Conditions in Multi-Instance Manager Deployments (Operational Edge Case)

• The Issue: The TAP prefix fc- and reconciliation logic assume a single runtime-manager instance operates on the host. If multiple manager instances are co-located on the same bare-metal/VM host (e.g., multi-tenant, multi-replica, or local development environments), a newly starting instance will run reconcile_orphaned_taps.
• The Impact: If Instance A is in the middle of booting a VM, it calls setup_tap_device (which brings the TAP interface up, but it starts as DOWN with NO-CARRIER until Firecracker attaches). If Instance B starts up at that exact microsecond, its reconciliation scan will see this transiently DOWN TAP, identify it as an "orphan", and delete it—breaking Instance A's boot process.
• Recommendation: Include a manager-specific identifier or deployment namespace suffix in the TAP prefix (e.g., fc-<manager-id>-<vm-id>) so that managers only reconcile TAPs they are authorized to manage.

3. Hard Dependency on iproute2 in Minimal Container Environments (Deployment Robustness)

• The Issue: The cleanup relies on spawning the external ip utility. While reconcile_orphaned_taps handles command execution errors gracefully (by logging a warning and returning 0), setup_tap_device and teardown_tap_device will fail hard if ip is missing.
• The Impact: In highly stripped-down production environments (such as distroless or minimal containers), iproute2 may not be installed, rendering the manager incapable of managing network interfaces.
• Recommendation: Explicitly document iproute2 as a mandatory system dependency, or implement interface management directly in Rust using netlink protocols (e.g., the rtnetlink crate) to eliminate the fork overhead and dependency.

---

Strengths of the PR

• Excellent Test Coverage: Pure unit tests cleanly validate formatting quirks (such as the ifindex suffix @) and invalid formats without executing real command-line calls.
• Cross-Platform Safety: Properly guards command execution failures to ensure the manager starts up gracefully on non-compatible platforms (like macOS dev environments) rather than panicking or crashing.