ci: gate the exec path on a real container, no KVM required

[dshakes]

Item #2 from the remaining-work list. The exec e2e test has existed since the exec dispatch fix but only ran by hand (LANTERN_RUNTIME_E2E=1), so the regression it guards was unprotected.

That regression is worth remembering: grpcSchedulerClient.Exec returned a canned string with exit code 0, making a missing implementation indistinguishable from a command that ran and printed nothing. Mocks can't catch that — only a real workload can.

Why this needs no KVM

The runtime-manager's docker backend spawns a real container, and hosted runners have Docker. So this is an ordinary PR gate, while the Firecracker boot job still waits on a nested-virt runner. The job builds the manager and scheduler, starts both, and runs the test against them.

No Postgres service needed — with DATABASE_URL unset the scheduler uses its in-memory store and runs always-leader, which is exactly what a single-node test wants.

The gate asserts the test actually RAN

go test reports a skipped test as a pass. Without this the job would go green while proving nothing — the same failure mode as the boot assertion that reported PASS for kernel-panicking VMs all session. So the step:

• fails on any --- SKIP
• requires the --- PASS: TestRuntimeExecE2E_RealVM line

Verified both directions before pushing

env unset	test SKIPs → gate correctly FAILS
env set	2/2 subtests PASS against a live manager + scheduler → gate passes

The test's own cleanup left no containers behind. Scheduler build command and workflow YAML both checked separately.

What this covers

The 6 subtests: stdout is real command output, the command runs inside the container, nonzero exit propagates, stderr stays separate, filesystem writes persist across execs, and an unknown vm_id errors instead of reporting success.

Workflow-only change.

[github-actions]

🔎 Codex cross-audit (agent:audit)

Blocking

1. runtime-exec-e2e.yml:61 builds services/runtime-manager, but the workflow never installs protoc. runtime-manager/build.rs calls tonic_build::compile_protos(...) build.rs:11, and existing QA/lint workflows install protobuf-compiler for this reason. On a clean hosted runner this build step fails before the e2e can run.

2. runtime-exec-e2e.yml:72 starts the manager without SCHEDULER_URL, so it never self-registers with the scheduler. The manager heartbeat explicitly returns when scheduler_url is empty scheduler_heartbeat.rs:195, while scheduler placement fails if no nodes are registered placement.go:35. After fixing protoc, Schedule should fail with “no nodes registered in cluster” instead of spawning the container. Set SCHEDULER_URL=http://localhost:8085 for the manager, or seed a node another way before running the test.

No other PR-introduced security/correctness issues found in the added workflow.

[github-actions]

🔷 Gemini cross-audit (agent:audit-gemini)

Audit Report: pr-diff.patch

---

🚨 Blocking Issues

1. Misconfigured GHA Workflow: Missing SCHEDULER_URL for runtime-manager Node Registration (Correctness/Failure)

• Impact: The E2E test suite will fail completely under CI.
• Details:
  • In the Start runtime-manager (docker backend) step of .github/workflows/runtime-exec-e2e.yml, the environment variable SCHEDULER_URL is not provided to the lantern-runtime-manager process.
  • Without SCHEDULER_URL configured, the runtime-manager's background heartbeat loop in services/runtime-manager/src/scheduler_heartbeat.rs skips execution:

    if cfg.scheduler_url.is_empty() {
        tracing::info!("scheduler_url not set; skipping self-register heartbeat");
        return;
    }

  • As a result, the node never registers with the runtime-scheduler (which listens on HTTP :8085).
  • When the Go E2E test calls Schedule, the scheduler's placement engine (Pick) will have 0 registered nodes, failing immediately with "no nodes registered in cluster":

    nodes := e.Store.ListNodes()
    if len(nodes) == 0 {
        return nil, fmt.Errorf("no nodes registered in cluster")
    }

• Resolution: In the workflow, set SCHEDULER_URL to point to the scheduler's HTTP endpoint during the manager's startup step:

        - name: Start runtime-manager (docker backend)
          run: |
            set -euo pipefail
            RUNTIME_BACKEND=docker \
            LISTEN_ADDR=0.0.0.0:50054 \
            SCHEDULER_URL=http://localhost:8085 \
            AGENT_IMAGE=python:3.11-slim \
            LOG_LEVEL=info \
            ./services/runtime-manager/target/debug/lantern-runtime-manager \
              > /tmp/manager.log 2>&1 &

---

🔍 Other Observations & Missed Edge Cases

2. Hardcoded Rust Toolchain Version vs. rust-toolchain.toml

• Impact: Minor maintenance/future-proofing concern.
• Details: The workflow set up Rust step specifies toolchain: "1.93". While this is consistent with the current rust-toolchain.toml value, it introduces a duplicate source of truth. If the project updates the toolchain in rust-toolchain.toml, this workflow will continue to compile with 1.93 instead of the upgraded toolchain, violating the step's name: Set up Rust (pinned by rust-toolchain.toml).
• Resolution: Omit the explicit toolchain key under dtolnay/rust-toolchain to let it automatically parse and honor rust-toolchain.toml.

3. Bash-Specific /dev/tcp Network Redirections

• Impact: Robustness.
• Details: The startup readiness check uses the bash-specific net redirection (exec 3<>/dev/tcp/127.0.0.1/<port>). While standard GitHub hosted Ubuntu runners have bash and net redirections enabled, this fails under pure sh environments or non-standard shells.
• Resolution: Standard tools like nc -z localhost <port> or standard polling using curl are more robust and portable.