GA 8/11 · VM identity = wire vm_id, re-identify on restore, per-incarnation exec tokens, fail-closed bare fork

docs/adr/0025-sandbox-spawn-contract-and-manager-authz.md

@@ -114,7 +114,135 @@ check and boot two VMs.
 - A guest that leaks its cert can no longer control other VMs. Exfiltrating its
   cert gains nothing beyond its own harness channel.
 - Callers need a coordinated rollout to send the token (see §2).
-- A restored VM still runs with the *source* VM's in-guest identity (vm_id,
-  cert). Re-issuing identity on restore is a follow-up that belongs with the ADR
-  0007 drive-state work. Until then, restore is meant for same-tenant warm starts,
-  which ownership enforcement makes safe.
+- ~~A restored VM still runs with the *source* VM's in-guest identity.~~
+  Superseded by the addendum below: a restore re-issues identity.
+
+## Addendum (2026-10-23) — one identity per VM: wire vm_id at boot, re-identity on restore
+
+Wave 2 adds pause (Snapshot + Stop), resume (Spawn with the **same** wire
+vm_id and `restore_snapshot_id`) and fork (Spawn **N new** wire vm_ids, each
+restoring the same snapshot). Two defects blocked that.
+
+### A. Boot under the wire vm_id (identity gap)
+
+The Firecracker backend booted every VM under its own random UUID, which became
+the cert CN and the harness `vm_id`. The registry was then rekeyed to the
+scheduler's wire vm_id. As a result, harness Heartbeat and VendSecret lookups
+by the UUID missed after the rekey.
+
+**Decision.** `Spawn` copies `SpawnRequest.handle.vm_id` into the internal
+`ScheduleRequest.vm_id`. Firecracker boots under that id, so the cert CN/SAN,
+the `lantern.vm_id` boot-arg, `spawn.json.vm_id`, the `Handle.id` and the
+registry key are all the same value. The id is validated before it names host
+paths: 1-64 chars of `[A-Za-z0-9-]` with at least 12 hex digits. The
+scheduler's `vm-<uuid>` passes. The TAP name is `fc-` + the first 8 **hex
+digits** of the id, which matches the old naming for UUID ids. The rekey stays
+for backends that mint their own ids (docker/k8s) and is a no-op when the ids
+are equal.
+
+### B. A restore is a new identity
+
+A snapshot freezes the whole guest identity: in memory, the vm_id, the private
+key, the secret cache and the `ip=` addressing; in the vmstate, the NIC's TAP
+and the cert drive. Resuming it as-is would run N forks as one VM.
+
+**Mechanism (smallest correct one under Firecracker's snapshot model):**
+
+1. **Manager, before the guest runs.** `backend.restore` builds a full
+   `VmConfig` for the restoring request's vm_id. That gives it a fresh /30 +
+   TAP, a freshly issued cert (CN/SAN = new vm_id, new key) and a fresh
+   `certs.img` whose `spawn.json` carries the **restoring request's** contract
+   (`vm_id`, `declared_secrets`, `egress_rules`, env, `network`). The API
+   sequence is pure and unit-tested (`plan_identity_restore`):
+   `PUT /snapshot/load {resume_vm:false, network_overrides:[eth0 → own TAP]}`,
+   then `PATCH /drives/certs {path_on_host: new image}`, then
+   `PATCH /vm {Resumed}`. Jailed restores also stage the new `certs.img` (and
+   the base rootfs) at the in-jail basenames the vmstate references.
+2. **Manager, authz.** The registry pins the SHA-256 of the cert issued for the
+   VM's *current incarnation* (`HandleInfo.cert_sha256`, from
+   `RuntimeBackend::identity_pin`). Heartbeat, VendSecret and Report require
+   that exact cert. A cert frozen into a snapshot has the right CN but a
+   superseded fingerprint, so it is refused. This covers a resume under the
+   same id, and someone replaying the snapshot's stolen key. Secrets vended to
+   the old identity cannot be refreshed. The pin is dropped on Stop.
+3. **Guest, re-identify** (`services/harness/src/reidentify.rs`). Every
+   `mke2fs` image gets a new ext4 UUID. The harness records the cert drive's
+   UUID at boot and polls it every second (`posix_fadvise(DONTNEED)` + a 2 KiB
+   read). When the UUID changes, the harness:
+   - remounts the drive and overwrites the harness-only cert/key/CA copies in
+     place. TLS config is re-read on every connect, so no environment mutation
+     is needed.
+   - clears the secret cache and adopts the new declared secrets and egress
+     rules.
+   - re-addresses eth0 (SIOCSIFADDR/NETMASK, default route) and re-pins
+     `manager.lantern.internal` in `/etc/hosts`.
+   - swaps the shared `vm_id` and bumps an identity generation, which tears
+     down the Heartbeat and Report streams so they reconnect under the new
+     cert.
+   - adopts the new incarnation's tokens, held in the same shared state as
+     `vm_id`, so every reader sees the live value:
+     - `exec_token`: a typed `spawn.json` field that becomes
+       `LANTERN_HARNESS_EXEC_TOKEN`. The exec server's interceptor reads it per
+       call, so the previous token is rejected as soon as the swap lands.
+     - `LANTERN_AGENT_INSTANCE_TOKEN` from the new contract's `env`. The
+       control plane mints this per fork child.
+
+     If exec was authenticated and the new drive carries no exec token, the
+     re-identify is refused rather than keeping the old token.
+
+   A drive whose `spawn.json` has no `vm_id` is refused, and the harness
+   retries on the next tick. The fail-closed property comes from item 2, not
+   from the poll.
+
+**Not re-applied:** workload argv/env/uid. The restored workload is already
+running, and that state is what the snapshot exists to keep. Its tenant and
+contract, as enforced by the manager (registry tenant, secrets allowlist,
+egress), are the restoring request's.
+
+### Exec token per incarnation
+
+The manager mints a fresh exec token for every spawn **and** every restore,
+including a resume under the same vm_id. It is 244 bits taken from two v4
+UUIDs. The token is written only to the 0600 cert image, never to the kernel
+cmdline, and is recorded as `HandleInfo.exec_token`. The manager's two harness
+exec clients (Exec and ExecTool) send it as `authorization: Bearer`. Before
+this change the manager sent no token, so prod exec, which requires a token,
+could never authenticate.
+
+### Bare-mode fork of a running source fails closed
+
+A bare (non-jailed) restore reuses the snapshot's host paths verbatim,
+including the source VM's vsock UDS and `certs.img`. So when the snapshot's
+source vm is still registered on this node and the backend reports
+`restore_needs_stopped_source()` (bare Firecracker), the restore returns
+`FAILED_PRECONDITION` with the message "fork of a running VM requires the
+jailer". A resume is unaffected, because its source was stopped first. Jailed
+restores give each VM its own paths and are allowed.
+
+### Snapshot id for pause → resume/fork
+
+`RuntimeManager.Snapshot` returns `SnapshotResponse.snapshot_id` (field 1).
+That value is the bare snapshot-store id (a UUID, no `store://` prefix). Pass it
+back verbatim as `AgentSpec.restore_snapshot_id`. The snapshot is recorded
+under the registry tenant of the snapshotted VM, and restore is
+ownership-checked against `AgentSpec.tenant_id` (§4). `SnapshotResponse.sha256`
+is currently empty. Snapshot now reads the artifacts from the path the backend
+reports (`fc://…`, chroot-expanded for jailed VMs). Before this change, the
+service read the bare `/run/lantern/snapshots/<id>` path, so every jailed pause
+failed.
+
+### Residual risks (need a real KVM host to close)
+
+- `network_overrides` needs Firecracker >= 1.12. Older VMMs reject the load.
+- `PATCH /drives` between load and resume is assumed allowed. The guest must
+  observe the new backing file; it unmounts the drive after every read, which
+  drops its page cache, and also fadvises.
+- The vmstate's drive and vsock paths must resolve at load. A bare fork of a
+  *live* source is refused (see above). A bare fork of a stopped source still
+  needs the source's `certs.img` on disk. Jailed mode stages the **base**
+  rootfs, not the source VM's disk state (ADR 0007 Tier 2).
+- Fork of a live source: until the fork re-identifies (≤ ~1 s), it holds the
+  source's still-valid cert and IP. Exposure is same-tenant only, because
+  ownership gates the restore.
+- The guest re-addressing ioctls and the default-route swap are compiled and
+  unit-adjacent only. They are not exercised on a real guest kernel.

services/harness/src/bootenv.rs

@@ -66,6 +66,24 @@ pub struct SpawnContract {
     pub egress_rules: Option<Vec<crate::proto::EgressRule>>,
     pub workload_uid: Option<u32>,
     pub workload_gid: Option<u32>,
+    /// This VM's identity (== the cert CN on the same drive). Re-read on
+    /// resume detection — see `reidentify.rs`.
+    pub vm_id: Option<String>,
+    pub declared_secrets: Option<Vec<crate::proto::SecretRef>>,
+    /// Point-to-point addressing; only used to re-address a restored guest
+    /// (a cold boot is configured by the kernel's `ip=`).
+    pub network: Option<GuestNetwork>,
+    /// Per-incarnation exec-server bearer (→ `LANTERN_HARNESS_EXEC_TOKEN`).
+    pub exec_token: Option<String>,
+}
+
+/// `spawn.json` `network` — this VM's /30 (manager = `host_ip`).
+#[derive(Clone, Debug, Default, PartialEq, Eq, serde::Deserialize)]
+#[serde(default)]
+pub struct GuestNetwork {
+    pub guest_ip: String,
+    pub host_ip: String,
+    pub netmask: String,
 }
 
 pub fn parse_spawn_contract(bytes: &[u8]) -> serde_json::Result<SpawnContract> {
@@ -92,6 +110,18 @@ pub fn contract_env_pairs(c: &SpawnContract) -> serde_json::Result<Vec<(String,
     if let Some(gid) = c.workload_gid {
         pairs.push(("LANTERN_WORKLOAD_GID".into(), gid.to_string()));
     }
+    if let Some(t) = c.exec_token.as_ref().filter(|t| !t.is_empty()) {
+        pairs.push(("LANTERN_HARNESS_EXEC_TOKEN".into(), t.clone()));
+    }
+    if let Some(vm_id) = c.vm_id.as_ref().filter(|v| !v.is_empty()) {
+        pairs.push(("LANTERN_VM_ID".into(), vm_id.clone()));
+    }
+    if let Some(secrets) = c.declared_secrets.as_ref().filter(|s| !s.is_empty()) {
+        pairs.push((
+            "LANTERN_DECLARED_SECRETS".into(),
+            serde_json::to_string(secrets)?,
+        ));
+    }
     let typed = pairs.len();
     for (k, v) in &c.env {
         if !pairs[..typed].iter().any(|(t, _)| t == k) {
@@ -292,7 +322,7 @@ fn assemble_declared_secrets() {
 /// Name the manager is reached by from inside the guest.
 ///
 /// Must match a DNS SAN on the manager's server certificate.
-const MANAGER_HOSTNAME: &str = "manager.lantern.internal";
+pub(crate) const MANAGER_HOSTNAME: &str = "manager.lantern.internal";
 
 /// Make the manager reachable by NAME rather than by address.
 ///
@@ -426,6 +456,70 @@ fn ensure_procfs() {}
 /// Returns the raw `spawn.json` from the drive root (`Ok(None)` when there is
 /// no drive or no file). The drive is UNMOUNTED before returning, after its
 /// cert material is copied into the harness-only `PRIVATE_DIR` — see there.
+/// The block device the cert drive was found on at boot — what the
+/// re-identify watcher polls (`reidentify.rs`).
+static CERT_DEVICE: std::sync::OnceLock<&'static str> = std::sync::OnceLock::new();
+
+pub fn cert_device() -> Option<&'static str> {
+    CERT_DEVICE.get().copied()
+}
+
+/// What one mount of the cert drive yielded.
+#[cfg(target_os = "linux")]
+pub struct CertDriveLoad {
+    /// Raw `spawn.json` (`Ok(None)` when absent).
+    pub spawn: std::io::Result<Option<Vec<u8>>>,
+    /// Paths of the harness-only copies (cert, key, CA), when stashed.
+    pub private: Option<Vec<std::path::PathBuf>>,
+}
+
+/// Mount `device`, verify it is the cert drive, copy its material into the
+/// harness-only dir (overwriting — the paths stay the same, so every later
+/// connect presents whatever identity is there now), read `spawn.json`,
+/// unmount. `None` when `device` is not the cert drive.
+#[cfg(target_os = "linux")]
+pub fn load_cert_device(device: &str) -> Option<CertDriveLoad> {
+    if !std::path::Path::new(device).exists() || !mount_ro_ext4(device, CERT_MOUNT) {
+        return None;
+    }
+    // Verify this really is the cert drive before trusting anything —
+    // probing means we may have just mounted some other volume.
+    let is_cert_drive = CERT_FILES
+        .iter()
+        .all(|name| std::path::Path::new(&format!("{CERT_MOUNT}/{name}")).exists());
+    if !is_cert_drive {
+        // Wrong volume: leave nothing mounted behind for the next candidate.
+        umount(CERT_MOUNT);
+        return None;
+    }
+    let spawn = match std::fs::read(format!("{CERT_MOUNT}/{SPAWN_FILE}")) {
+        Ok(b) => Ok(Some(b)),
+        Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None),
+        Err(e) => Err(e),
+    };
+    let private = match stash_private(
+        std::path::Path::new(CERT_MOUNT),
+        std::path::Path::new(PRIVATE_DIR),
+        &CERT_FILES,
+    ) {
+        Ok(p) => Some(p),
+        Err(e) => {
+            tracing::warn!(error = %e, "certs: cannot stash cert material");
+            None
+        }
+    };
+    // Nothing on the drive may stay reachable by the workload uid.
+    if !umount(CERT_MOUNT) {
+        tracing::error!(
+            mount = CERT_MOUNT,
+            error = %std::io::Error::last_os_error(),
+            "certs: SECURITY could not unmount the cert drive — its files remain \
+             subject to the image's file modes"
+        );
+    }
+    Some(CertDriveLoad { spawn, private })
+}
+
 #[cfg(target_os = "linux")]
 fn mount_cert_drive() -> std::io::Result<Option<Vec<u8>>> {
     // Already resolvable — a real environment was injected (docker/K8s), or
@@ -443,60 +537,30 @@ fn mount_cert_drive() -> std::io::Result<Option<Vec<u8>>> {
     }
 
     for device in CERT_DEVICE_CANDIDATES {
-        if !std::path::Path::new(device).exists() || !mount_ro_ext4(device, CERT_MOUNT) {
+        let Some(load) = load_cert_device(device) else {
             continue;
-        }
-
-        // Verify this really is the cert drive before publishing anything —
-        // probing means we may have just mounted some other volume.
-        let paths: Vec<String> = CERT_FILES
-            .iter()
-            .map(|name| format!("{CERT_MOUNT}/{name}"))
-            .collect();
-        if paths.iter().all(|p| std::path::Path::new(p).exists()) {
-            let spawn = match std::fs::read(format!("{CERT_MOUNT}/{SPAWN_FILE}")) {
-                Ok(b) => Ok(Some(b)),
-                Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None),
-                Err(e) => Err(e),
-            };
-            match stash_private(
-                std::path::Path::new(CERT_MOUNT),
-                std::path::Path::new(PRIVATE_DIR),
-                &CERT_FILES,
-            ) {
-                Ok(private) => {
-                    // SAFETY: single-threaded startup — see the contract on `main`.
-                    unsafe {
-                        std::env::set_var("LANTERN_VM_TLS_CERT", &private[0]);
-                        std::env::set_var("LANTERN_VM_TLS_KEY", &private[1]);
-                        std::env::set_var("LANTERN_MANAGER_TLS_CA", &private[2]);
-                    }
-                    tracing::info!(
-                        device,
-                        private_dir = PRIVATE_DIR,
-                        "certs: per-VM cert drive copied to harness-only dir — \
-                         harness→manager channel is mTLS"
-                    );
+        };
+        let _ = CERT_DEVICE.set(device);
+        match load.private {
+            Some(private) => {
+                // SAFETY: single-threaded startup — see the contract on `main`.
+                unsafe {
+                    std::env::set_var("LANTERN_VM_TLS_CERT", &private[0]);
+                    std::env::set_var("LANTERN_VM_TLS_KEY", &private[1]);
+                    std::env::set_var("LANTERN_MANAGER_TLS_CA", &private[2]);
                 }
-                Err(e) => tracing::warn!(
-                    error = %e,
-                    "certs: cannot stash cert material — harness→manager channel will be PLAINTEXT"
-                ),
-            }
-            // Nothing on the drive may stay reachable by the workload uid.
-            if !umount(CERT_MOUNT) {
-                tracing::error!(
-                    mount = CERT_MOUNT,
-                    error = %std::io::Error::last_os_error(),
-                    "certs: SECURITY could not unmount the cert drive — its files remain \
-                     subject to the image's file modes"
+                tracing::info!(
+                    device,
+                    private_dir = PRIVATE_DIR,
+                    "certs: per-VM cert drive copied to harness-only dir — \
+                     harness→manager channel is mTLS"
                 );
             }
-            return spawn;
+            None => tracing::warn!(
+                "certs: cannot stash cert material — harness→manager channel will be PLAINTEXT"
+            ),
         }
-
-        // Wrong volume: leave nothing mounted behind for the next candidate.
-        umount(CERT_MOUNT);
+        return load.spawn;
     }
 
     tracing::warn!(
@@ -860,6 +924,40 @@ mod tests {
         let _ = std::fs::remove_dir_all(&base);
     }
 
+    #[test]
+    fn contract_identity_and_secrets_beat_the_cmdline() {
+        // spawn.json is the identity of record: the cmdline copy is frozen in
+        // guest memory and may name a snapshot's source VM.
+        let c = parse_spawn_contract(
+            br#"{"vm_id":"vm-new","declared_secrets":[{"env_name":"K","secret_uri":"u"}]}"#,
+        )
+        .expect("valid");
+        let got = layer_boot_env(
+            env_from_cmdline("lantern.vm_id=vm-old"),
+            contract_env_pairs(&c).expect("encodable"),
+        );
+        let get = |k: &str| got.iter().find(|(n, _)| n == k).map(|(_, v)| v.as_str());
+        assert_eq!(get("LANTERN_VM_ID"), Some("vm-new"));
+        assert_eq!(
+            get("LANTERN_DECLARED_SECRETS"),
+            Some(r#"[{"env_name":"K","secret_uri":"u"}]"#)
+        );
+        let c = parse_spawn_contract(
+            br#"{"exec_token":"tok","env":{"LANTERN_HARNESS_EXEC_TOKEN":"forged"}}"#,
+        )
+        .expect("valid");
+        let pairs = contract_env_pairs(&c).expect("ok");
+        let toks: Vec<&str> = pairs
+            .iter()
+            .filter(|(k, _)| k == "LANTERN_HARNESS_EXEC_TOKEN")
+            .map(|(_, v)| v.as_str())
+            .collect();
+        assert_eq!(toks, vec!["tok"], "typed token beats free-form env");
+        // An empty vm_id never overrides the cmdline.
+        let c = parse_spawn_contract(br#"{"vm_id":""}"#).expect("valid");
+        assert!(contract_env_pairs(&c).expect("ok").is_empty());
+    }
+
     #[test]
     fn keeps_values_containing_equals() {
         // split_once means only the FIRST '=' delimits; base64/query-ish

services/harness/src/cc_attest.rs

@@ -189,7 +189,7 @@ pub async fn run(manager: &ManagerClient) {
     );
 
     let frame = build_frame(
-        &manager.vm_id,
+        &manager.vm_id(),
         &cc_tech,
         &runtime_class,
         measurement.as_deref(),