Skip to content

fix(harness): verify the manager by name; let rustls explain itself - #201

Merged
dshakes merged 2 commits into
masterfrom
chore/rustls-logging
Jul 30, 2026
Merged

dshakes merged 2 commits into
masterfrom
chore/rustls-logging

Conversation

@dshakes

@dshakes dshakes commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Two commits: re-enable rustls's own logging, and fix the server-name fault it exposed.

1. Let rustls explain itself

default-features = false on rustls had dropped logging, one of its defaults. Without it rustls cannot report why it rejects a chain — which is why several rounds of debugging inferred its reasoning from outside and kept landing on wrong answers (DER encoding, musl, entropy, clock skew, crypto provider — all checked, all wrong).

One Cargo line, and it named the cause in a single run.

2. Verify the manager by name, not by address

rustls was checking the manager's certificate against IpAddress(172.16.x.1) — the VM's gateway — which is not, and can never be, one of its SANs. Every microVM sits on its own point-to-point link, so the manager's address differs per VM; only a name can be on the certificate.

ClientTlsConfig::domain_name exists for exactly this and is set. It observably does not reach the connection: with logging on, the ServerName was IpAddress on every attempt, never DnsName. I verified the config is built correctly (identity + CA + domain_name) and that the running binary contains the change, so this isn't a stale build.

Rather than depend on that override, the name now goes in the URI: the harness pins manager.lantern.internal to its gateway in /etc/hosts at boot and dials that. The TLS server name is then correct by construction, one certificate with a DNS:manager.lantern.internal SAN covers every VM, and no library override sits in the path. Best-effort — an unparseable address or unwritable /etc/hosts leaves the original behaviour untouched.

Verified on real KVM

before:  5x IpAddress(V4(172.16.0.1))          0x DnsName
after:   5x DnsName("manager.lantern.internal")  0x IpAddress

VendSecret is still NOT fixed

Stating that plainly. UnknownIssuer persists with the name now correct, and the client still sends no certificate ("Client auth requested but no cert/sigscheme available").

Those three facts — wrong server name, no trusted roots, no client identity — now point at one cause rather than three: the whole ClientTlsConfig not reaching the connection, with tonic falling back to a default. That the ServerName tracked my URI change while the roots stayed empty fits that reading. It's a single hypothesis instead of a pile of symptoms, but it is not proven, and no fix for it is attempted here.

153 harness tests pass, clippy and rustfmt clean.

dshakes added 2 commits July 30, 2026 08:56
…ature)

`default-features = false` on rustls dropped `logging`, one of its defaults.
Without it rustls cannot say WHY it rejects a chain, which is why three rounds
of debugging inferred its reasoning from the outside and kept landing on wrong
answers. Turning it back on took one Cargo change and produced the actual cause
in a single run.

What rustls reports from inside the guest:

  Got CertificateRequest ... signature_algorithms: [ECDSA_NISTP384_SHA384,
    ECDSA_NISTP256_SHA256, ..., RSA_PKCS1_SHA256]
  Client auth requested but no cert/sigscheme available
  No cached session for IpAddress(V4(Ipv4Addr([172, 16, 0, 1])))

Two findings, neither of which was visible before:

1. rustls is verifying against **IpAddress(172.16.0.1)**, not
   DnsName("manager.lantern.internal"). The `domain_name` set on
   ClientTlsConfig is not reaching the connection, so the server certificate is
   checked against the VM's gateway address — which is not one of its SANs
   (IP:127.0.0.1, DNS:localhost, DNS:manager.lantern.internal). This is the
   name-pinning added for per-VM point-to-point links failing to take effect.

2. The client never presents its certificate: "Client auth requested but no
   cert/sigscheme available", even though the server advertises RSA_PKCS1_SHA256
   and the VM's key is RSA-2048. So mTLS client auth is also not working, and
   this is a separate fault from (1).

The connection still dies reporting UnknownIssuer, so at least one of these is
mis-surfaced by tonic rather than reported as a name or client-auth failure.
VendSecret is NOT fixed by this commit.

Also noted, not fixed here: tracing is initialised BEFORE
`bootenv::bootstrap()`, so RUST_LOG arriving via boot-args can never affect the
filter — boot-args cannot configure logging in a microVM at all. Diagnosing this
needed the filter default patched instead, which is why it is worth fixing
properly.
…ns to have

rustls was checking the manager's server certificate against
IpAddress(172.16.x.1) — the VM's gateway — which is not, and can never be, one
of its SANs. Every microVM sits on its own point-to-point link, so the manager's
address differs per VM; only a NAME can be on the certificate.

`ClientTlsConfig::domain_name` was supposed to handle exactly this, and it is
set. It observably does not reach the connection: with rustls logging enabled
the ServerName was IpAddress on every attempt, never DnsName. Rather than
depend on that override taking effect, put the name in the URI so the TLS
server name is correct by construction.

The harness now pins `manager.lantern.internal` to its gateway in /etc/hosts at
boot and dials that name. One certificate with a DNS:manager.lantern.internal
SAN then covers every VM, with no per-VM IP SANs and no library override in the
path. Best-effort: an unparseable address or an unwritable /etc/hosts leaves the
original address in place, so the connection behaves exactly as before.

Verified on real KVM with rustls trace logging:

  before: 5x IpAddress(V4(172.16.0.1)),  0x DnsName
  after:  5x DnsName("manager.lantern.internal"),  0x IpAddress

VendSecret still fails, and NOT because of this. `UnknownIssuer` persists with
the name now correct, and the client still sends no certificate ("Client auth
requested but no cert/sigscheme available"). Taken together — wrong server name,
no trusted roots, no client identity — the evidence now points at the whole
ClientTlsConfig not reaching the connection, with tonic falling back to a
default. That the ServerName tracked the URI change while the roots stayed empty
fits that reading. It is a single remaining hypothesis rather than three
symptoms, but it is not yet proven, and no fix for it is attempted here.
@github-actions github-actions Bot added the domain:core Core library / business logic label Jul 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔷 Gemini cross-audit (agent:audit-gemini)

[Blocking] Soundness Issue: Concurrency & Undefined Behavior in unsafe { std::env::set_var(...) }

  • The Problem: The safety comments in bootenv.rs (including the newly added one in pin_manager_hostname) assert that std::env::set_var is safe to call because it is invoked "before any thread is spawned." This is incorrect.
  • The Root Cause: main is annotated with the #[tokio::main] attribute macro (without specifying flavor = "current_thread"). The macro expands to generate code that builds and spawns a multi-threaded Tokio runtime prior to executing the async main body. This means background worker threads are already running when bootenv::bootstrap() is called.
  • The Risk: Calling std::env::set_var (which is unsafe starting in Rust 1.79.0) in a multi-threaded process is a data race on environ under the hood. Concurrent environment queries (such as those done by tracing, standard library, or other dependencies) will trigger undefined behavior (UB), resulting in potential memory unsafety or hard process crashes.
  • The Solution: Avoid #[tokio::main] in main.rs. Instead, initialize your synchronous bootstrap routines first, and then construct/start the tokio runtime manually:
    fn main() -> Result<()> {
        tracing_subscriber::fmt().init(); // (single-threaded here)
    
        // Guarantee single-threaded safety for environment mutation
        bootenv::bootstrap();
    
        let rt = tokio::runtime::Builder::new_multi_thread()
            .enable_all()
            .build()?;
        rt.block_on(async_main())
    }

[Blocking] Correctness/Stale Configuration Bug: Persistent /etc/hosts & IP Mismatch

  • The Problem: pin_manager_hostname checks if !existing.contains(MANAGER_HOSTNAME) before appending to /etc/hosts.
  • The Risk: If the guest's /etc/hosts is persistent (e.g., across VM boots on reusable images, snapshots, or local dev/test container setups), and a previous run already mapped manager.lantern.internal to a different IP address:
    1. existing.contains(MANAGER_HOSTNAME) will evaluate to true.
    2. The function will bypass appending the updated IP entry.
    3. The harness will still rewrite LANTERN_MANAGER_ADDR to use the hostname manager.lantern.internal.
  • The Result: The guest's DNS resolver will route traffic to the outdated IP address, breaking the gRPC channel to the manager.
  • The Solution: Ensure any stale entries for manager.lantern.internal are purged or modified. For example, parse /etc/hosts line-by-line, filter out lines containing MANAGER_HOSTNAME, and append the correct mapping:
    let clean_hosts: Vec<String> = existing
        .lines()
        .filter(|line| !line.contains(MANAGER_HOSTNAME))
        .map(|line| line.to_string())
        .collect();
    let contents = format!("{}\n{}", clean_hosts.join("\n"), entry);

[Edge Case] Fragile Substring Verification in /etc/hosts

  • The Problem: A raw .contains(MANAGER_HOSTNAME) substring check can be easily fooled.
  • The Risk: Any comment like # manager.lantern.internal configuration or a partial domain name matching not-manager.lantern.internal inside /etc/hosts will trigger a false positive, causing the host pinning to skip entirely.
  • The Solution: Match against exact whitespace-delimited tokens or boundary-accurate lines.

[Non-Blocking/Improvement] Rustls Log Diagnostics

  • Feedback: Adding "logging" to rustls in Cargo.toml is a great diagnostic improvement that makes handshake rejections understandable within guest trace logs.

@compass-sdlc-bot compass-sdlc-bot Bot added the agent:reviewed-clean Reviewer found no Blocking issues this round label Jul 30, 2026
@dshakes
dshakes merged commit 5f80a78 into master Jul 30, 2026
18 of 19 checks passed
@dshakes
dshakes deleted the chore/rustls-logging branch August 4, 2026 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent:reviewed-clean Reviewer found no Blocking issues this round domain:core Core library / business logic

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant