feat(microvm): make the in-guest tool path usable and assert it

[dshakes]

Correction first

The in-guest tool runner was never missing. I listed it as unimplemented in my earlier status — that was wrong. The whole chain already existed: the manager sends a __lantern_tool_call__ envelope, the guest exec server routes it to tool_runner::dispatch, and the typed result comes back.

exec_tool returned UNAVAILABLE because it needs a heartbeat-derived guest address to dial back on — and the harness couldn't open a heartbeat stream until the networking and mTLS fixes landed. Verified end to end now:

ExecTool shell_exec -> TOOL_STATUS_OK
result.stdout = "hello-from-inside-the-microvm\naarch64\n0\n"
result.exit_code = 0

Two things were still missing to make that path exercisable.

1. The test image had no shell

shell_exec spawns /bin/sh -c, so every tool call in this image failed with spawn failed: No such file or directory — the in-guest tool path couldn't be exercised at all.

build-image.sh now installs a static busybox as /bin/sh (the rootfs has no libc, so it must be static). It prefers FC_BUSYBOX_BIN, falls back to a host /bin/busybox that file confirms is statically linked, and when neither exists builds exactly as before and says so.

2. Nothing asserted it

The integration test now invokes a tool through ExecTool and requires the tool's own output back. That covers the workflow-engine's tool_call path end to end — and since ExecTool can only reach the guest via the heartbeat-derived address, it also proves the harness is reachable, not merely started.

The assertion degrades to a skip, not a pass, when the image has no shell — so it can never report success for a path it didn't run. That's the same failure mode as the boot assertion this session already fixed.

Verified on real KVM — 5/5

PASS: Firecracker backend available (Linux + firecracker + /dev/kvm)
PASS: microVM booted AND guest is live
PASS: harness live + secret vended over mTLS (CN=vm_id; value never logged)
PASS: tool invoked in-guest and returned its output (ExecTool -> shell_exec)
PASS: ALL ASSERTIONS PASSED

Shell/infra only — no Rust or Go changes.

[github-actions]

🔎 Codex cross-audit (agent:audit)

Blocking: None found.

Findings:

• P2: New ExecTool coverage is skipped in the normal CI path. The workflow installs musl-tools e2fsprogs jq curl, but not busybox-static, and build-image.sh only installs /bin/sh if FC_BUSYBOX_BIN or host /bin/busybox exists. On a typical KVM runner this produces a rootfs without /bin/sh, then the new ExecTool assertion treats No such file or directory as SKIP, so the PR can pass without ever proving shell_exec works in-guest. See build-image.sh, integration-test.sh, and workflow context at .github/workflows/microvm-integration.yml. Install/provide a static BusyBox in the workflow and make this assertion hard-fail for CI.

• P3: FC_BUSYBOX_BIN is not validated as static. The auto-detected /bin/busybox path checks file ... statically linked, but a caller-provided FC_BUSYBOX_BIN only needs to be executable. A dynamic BusyBox copied into this libc-less rootfs will fail at runtime with the same misleading “No such file or directory” path and the integration test may skip it. Validate FC_BUSYBOX_BIN with the same static check before copying.

[github-actions]

🔷 Gemini cross-audit (agent:audit-gemini)

1. Executive Summary

The proposed unified diff in pr-diff.patch is clean, secure, and well-designed. It correctly installs a statically-linked busybox binary as /bin/sh into the minimal guest rootfs, addressing the bug where the in-guest shell_exec tool failed with "No such file or directory" due to the lack of a guest shell.

No correctness regressions or security vulnerabilities were introduced. However, there are a few important, non-blocking edge cases and missed improvements regarding CI coverage, host binary detection, and in-guest command capabilities.

---

2. Blocking Issues

• None.

---

3. Important Edge Cases & Recommendations (Non-Blocking)

A. Missing CI Dependency (busybox-static) in GitHub Actions Workflow

• Issue: In .github/workflows/microvm-integration.yml, the workflow installs tools via sudo apt-get install -y musl-tools e2fsprogs jq curl. It does not install busybox-static.
• Impact: On the live self-hosted or KVM runner, /bin/busybox will not be present or statically linked. build-image.sh will fall back to not installing /bin/sh, causing the new integration test assertion Invoke a tool in the guest (ExecTool -> shell_exec) to be silently skipped rather than fully executed and verified.
• Recommendation: Install busybox-static in the workflow's package setup step to guarantee complete end-to-end coverage in CI:

  sudo apt-get update -y
  sudo apt-get install -y musl-tools e2fsprogs jq curl busybox-static

B. Robustness in Host Busybox Detection

• Issue: In infra/firecracker/build-image.sh, the check for the host busybox path is hardcoded to /bin/busybox:

  elif [ -x /bin/busybox ] && file /bin/busybox 2>/dev/null | grep -q "statically linked"; then

  On some Linux distributions, static busybox may reside under /usr/bin/busybox or another location in $PATH.
• Recommendation: Use which or command -v to search for busybox dynamically across the host system:

  elif BUSYBOX_PATH=$(which busybox 2>/dev/null) && [ -x "${BUSYBOX_PATH}" ] && file "${BUSYBOX_PATH}" 2>/dev/null | grep -q "statically linked"; then
    BUSYBOX_SRC="${BUSYBOX_PATH}"

C. In-Guest Shell Applet Symlinks for Exec Capabilities

• Issue: Although /bin/sh is successfully created as a symlink to /bin/busybox, standard shell applets (such as ls, cat, grep, etc.) are not symlinked in the guest's /bin/ or /usr/bin/.
• Impact: If the static busybox binary was compiled without FEATURE_SH_STANDALONE enabled, running any standard command other than built-in shell functions (like echo) inside a shell_exec tool call will fail with "command not found".
• Recommendation: Consider populating standard applet symlinks inside the guest rootfs mount during build_rootfs() using busybox's automatic installation feature:

  sudo chroot "${mnt}" /bin/busybox --install -s /bin 2>/dev/null || true

  (Or explicitly symlink a few critical utilities if chrooting is not preferred).