fix(harness): retry the boot-time secret prefetch

[dshakes]

Follow-up to the finding Codex raised while auditing #211. The mechanism checks out: a failed prefetch was permanent for the life of the VM.

Why it couldn't recover

prefetch_declared warms the secret cache moments after boot, so the mTLS vend path runs on every boot rather than only when a workload asks. It tried each declared secret once and logged a warning on failure.

That looks recoverable — refresh_loop runs every 15s — but it isn't. That loop iterates entries already in the cache and re-vends the ones near expiry. A prefetch that never landed leaves nothing to iterate. One lost race at boot and the vend never happens again.

The window is real: this runs immediately after the guest's network comes up, against a manager that may not be accepting yet.

Impact

	effect
production	cold cache, not broken — the workload still vends on demand
CI	the integration test asserts on that boot vend (EXPECT_VENDSECRET=1, now always on), so a lost race reads as a failed security contract, not a slow start

The second is the sharper edge: a flaky gate on the one assertion that proves the mTLS boot contract works.

Change

Retry up to 5 times with doubling backoff — 250ms, 500ms, 1s, 2s (~3.75s total, capped so a wedged manager can't stretch it out). Only failures are retried. Still best-effort: spawned off the boot path, bounded, and giving up never blocks the VM — it logs that the workload will vend on demand.

Not using a shared helper because the repo's canonical lantern-retry crate doesn't exist yet (same gap as @lantern/retry in TS, already noted in bridge-core). Small local loop, to be replaced when it lands.

Verification

• Backoff extracted as a pure function so the schedule is testable without a live manager. The test asserts it both backs off and stays bounded — removing the cap makes it fail (verified by mutation, then reverted).
• Real KVM: 5/5 assertions, including harness live + secret vended over mTLS.
• The binary was checked to actually contain the change first — an earlier build silently reused a two-day-old root-owned artifact, which would have "verified" nothing.
• 156 harness tests pass (1 new), clippy + rustfmt clean on 1.94.

[github-actions]

🔎 Codex cross-audit (agent:audit)

Blocking: none found.

I audited git diff origin/master...HEAD; the PR only changes services/harness/src/secrets.rs to retry boot-time declared-secret prefetches. I did not find a correctness, security, or missed-edge regression in the changed logic.

Verification: git diff --check origin/master...HEAD is clean. I did not run cargo test because this session’s filesystem is read-only.

[github-actions]

🧪 QA — ❌ tests failing

	/home/runner/work/lantern/lantern/services/control-plane/internal/handlers/signals.go:128
github.com/dshakes/lantern/services/control-plane/internal/handlers.postSignal
	/home/runner/work/lantern/lantern/services/control-plane/internal/handlers/signals_test.go:44
github.com/dshakes/lantern/services/control-plane/internal/handlers.TestSignalIngest.func1
	/home/runner/work/lantern/lantern/services/control-plane/internal/handlers/signals_test.go:192
testing.tRunner
	/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.5.linux-amd64/src/testing/testing.go:2036
2026-08-02T16:03:43.950Z	WARN	signals	handlers/signals.go:100	signal rejected (401)	{"reason": "request missing x-lantern-signal-token header", "from": "192.0.2.1:1234"}
github.com/dshakes/lantern/services/control-plane/internal/handlers.(*SignalHandler).authorize
	/home/runner/work/lantern/lantern/services/control-plane/internal/handlers/signals.go:100
github.com/dshakes/lantern/services/control-plane/internal/handlers.(*SignalHandler).IngestSignal
	/home/runner/work/lantern/lantern/services/control-plane/internal/handlers/signals.go:128
github.com/dshakes/lantern/services/control-plane/internal/handlers.postSignal
	/home/runner/work/lantern/lantern/services/control-plane/internal/handlers/signals_test.go:44
github.com/dshakes/lantern/services/control-plane/internal/handlers.TestSignalIngest.func1
	/home/runner/work/lantern/lantern/services/control-plane/internal/handlers/signals_test.go:192
testing.tRunner
	/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.5.linux-amd64/src/testing/testing.go:2036
2026-08-02T16:03:43.951Z	WARN	signals	handlers/signals.go:100	signal rejected (401)	{"reason": "LANTERN_SIGNAL_TOKEN unset on the server — this API instance cannot accept signals", "from": "192.0.2.1:1234"}
github.com/dshakes/lantern/services/control-plane/internal/handlers.(*SignalHandler).authorize
	/home/runner/work/lantern/lantern/services/control-plane/internal/handlers/signals.go:100
github.com/dshakes/lantern/services/control-plane/internal/handlers.(*SignalHandler).IngestSignal
	/home/runner/work/lantern/lantern/services/control-plane/internal/handlers/signals.go:128
github.com/dshakes/lantern/services/control-plane/internal/handlers.postSignal
	/home/runner/work/lantern/lantern/services/control-plane/internal/handlers/signals_test.go:44
github.com/dshakes/lantern/services/control-plane/internal/handlers.TestSignalIngest.func1
	/home/runner/work/lantern/lantern/services/control-plane/internal/handlers/signals_test.go:192
testing.tRunner
	/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.5.linux-amd64/src/testing/testing.go:2036
2026-08-02T16:03:43.960Z	WARN	signals	handlers/signals.go:100	signal rejected (401)	{"reason": "request missing x-lantern-signal-token header", "from": "192.0.2.1:1234"}
github.com/dshakes/lantern/services/control-plane/internal/handlers.(*SignalHandler).authorize
	/home/runner/work/lantern/lantern/services/control-plane/internal/handlers/signals.go:100
github.com/dshakes/lantern/services/control-plane/internal/handlers.(*SignalHandler).ListSignals
	/home/runner/work/lantern/lantern/services/control-plane/internal/handlers/signals.go:211
github.com/dshakes/lantern/services/control-plane/internal/handlers.TestSignalListTokenGate
	/home/runner/work/lantern/lantern/services/control-plane/internal/handlers/signals_test.go:408
testing.tRunner
	/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.5.linux-amd64/src/testing/testing.go:2036
2026-08-02T16:03:43.978Z	INFO	llm_proxy	handlers/llm_proxy.go:1237	streaming: provider failed before first delta, failing over	{"from": "openai", "error": "openai stream 500: "}
2026-08-02T16:03:43.980Z	ERROR	llm_proxy	handlers/llm_proxy.go:1280	completeSSE: streaming failed	{"error": "streaming: openai failed after 1 delta(s): simulated mid-stream provider failure"}
github.com/dshakes/lantern/services/control-plane/internal/handlers.(*LlmProxyHandler).completeSSE
	/home/runner/work/lantern/lantern/services/control-plane/internal/handlers/llm_proxy.go:1280
github.com/dshakes/lantern/services/control-plane/internal/handlers.TestCompleteSSE_ErrorAfterFirstDelta
	/home/runner/work/lantern/lantern/services/control-plane/internal/handlers/stream_test.go:422
testing.tRunner
	/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.5.linux-amd64/src/testing/testing.go:2036
2026-08-02T16:03:43.992Z	DEBUG	middleware/tenant.go:99	tenant extracted	{"tenant_id": "tenant-1", "method": "/test.Service/TestMethod"}
2026-08-02T16:03:43.992Z	DEBUG	middleware/tenant.go:99	tenant extracted	{"tenant_id": "tenant-1", "method": "/test.Service/TestMethod"}
2026-08-02T16:03:43.992Z	DEBUG	middleware/tenant.go:99	tenant extracted	{"tenant_id": "tenant-1", "method": "/test.Service/TestMethod"}
FAIL
FAIL	github.com/dshakes/lantern/services/control-plane/internal/handlers	0.340s
ok  	github.com/dshakes/lantern/services/control-plane/internal/middleware	1.021s
ok  	github.com/dshakes/lantern/services/control-plane/internal/scheduler	1.017s
ok  	github.com/dshakes/lantern/services/control-plane/internal/secrets	1.013s
?   	github.com/dshakes/lantern/services/control-plane/internal/server	[no test files]
?   	github.com/dshakes/lantern/services/control-plane/internal/storage	[no test files]
ok  	github.com/dshakes/lantern/services/control-plane/internal/telemetry	1.013s
ok  	github.com/dshakes/lantern/services/control-plane/internal/workflow	1.023s
FAIL
make: *** [Makefile:180: test-go] Error 1

[github-actions]

🔎 Codex cross-audit (agent:audit)

Blocking: none found.

Findings: no correctness or security regressions attributable to this PR. The retry loop in services/harness/src/secrets.rs preserves best-effort boot behavior, retries only pending names, removes successful vends from subsequent attempts, and keeps the sleep schedule bounded.

Verification: git diff --check origin/master...HEAD passed. I could not run Rust tests because the read-only sandbox prevents rustup from creating files under ~/.rustup/tmp, so compile/test verification is still pending in CI or a writable dev environment.

[github-actions]

🔷 Gemini cross-audit (agent:audit-gemini)

An independent audit of the unified diff in pr-diff.patch against the base codebase has been completed.

The implementation of the boot-time retry mechanism in secrets.rs is highly idiomatic and correctly implements saturating arithmetic and capped exponential backoffs. However, two critical security/correctness edge cases were identified:

---

1. Metadata Leak (Secret URIs) in Warning Logs

• Category: Security / Missed Edge Case
• Severity: Medium
• Description:
  The retry loop in prefetch_declared contains an explicit security comment to guard against logging sensitive URIs:

  // Names only — never the value or the URI.
  tracing::warn!(env = %name, attempt, error = %e, "secrets: prefetch failed");

  However, get_or_vend propagates the anyhow::Error returned by the vend_secret RPC client in manager_client.rs. That gRPC call explicitly appends the URI to the error context:

  .with_context(|| format!("vend_secret RPC failed for secret_uri '{}'", req.secret_uri))?

  Because the warning prints the full error chain (error = %e), the secret URI is leaked into the warnings logs (e.g., lantern.secret://tenant/t1/key/K), directly violating the developer's stated security boundary.
• Recommendation:
  Sanitize the error representation in the warning block of secrets.rs by logging only the root cause or a sanitized error string (or update the warning block to avoid formatting %e entirely if URI privacy is strict).

---

2. Sequential & Unbounded Connection Timeouts (The "Bounded Wait" Illusion)

• Category: Correctness / Performance
• Severity: Medium
• Description:
  The prefetch_declared retry loop processes all pending secrets sequentially:

  for name in std::mem::take(&mut pending) {
      match self.get_or_vend(&name).await { ... }
  }

  The comment asserts that the prefetch logic is best-effort and bounded to ~3.75s (also enforced in unit tests). However, there are no connection (connect_timeout) or request (timeout) timeouts configured on the tonic Endpoint inside manager_client.rs.
  If the manager is down or unresponsive (e.g., packet-dropping firewall), each sequential connection attempt will block for the default TCP handshake timeout (typically 20–120s). If multiple secrets are declared, the first attempt alone can block the background task for minutes—making the ~3.75s sleep schedule a false ceiling.
• Recommendation:
  1. Configure explicit timeouts (e.g., .connect_timeout(...) and .timeout(...)) when constructing gRPC endpoints in manager_client.rs.
  2. Warm the declared secrets concurrently (e.g., spawning independent tokio tasks or utilizing a FuturesUnordered stream) rather than processing them sequentially.

[github-actions]

🔷 Gemini cross-audit (agent:audit-gemini)

Audit Report: Boot-Time Secret Prefetch Retry Support

After a thorough, independent static analysis of the unified diff in pr-diff.patch against the base repository context, there are no blocking issues, correctness regressions, or security vulnerabilities introduced by this change.

The implementation is highly robust, correct, and idiomatic. Below is the detailed audit summary:

1. Security Analysis (No Issues)

• No Information Disclosure: The warning and informational logs within the retry loop only output the declared environment variable names (env = %name). Sensitive secret values or underlying secret URIs are never logged.
• Integrity Maintained: The retry mechanism does not bypass any peer-authentication validation. It sits entirely within the established security boundary of the guest harness calling get_or_vend.

2. Correctness & Edge-Case Verification (No Issues)

• Sequential Processing (Safe & Concurrency-Friendly): The loop vends the pending secrets sequentially. This prevents sudden spikes in connection limits or hammering of the manager's gRPC listener during VM boot.
• Selective Retry Logic: Using std::mem::take(&mut pending) correctly extracts the list of pending items and reconstructs pending with only the failed ones at the end of each iteration. This guarantees that successfully prefetched secrets are never vended again in subsequent attempts.
• Bounded Off-Boot Execution: Since prefetch_declared is spawned as a background task off the boot path (as verified in main.rs), any latency or unreachable manager state during the backoff period (~3.75s total) will not block the startup of other crucial VM systems or workload supervisor spawning.
• Robust Backoff Calculation: The prefetch_backoff function handles potential arithmetic overflows safely via saturating_sub, saturating_mul, and a cap on shift (at min(3)). It is mathematically bounded and returns consistent values for any u32 input.

3. Test Coverage & Quality

• Deterministic Verification: The newly added prefetch_backoff_doubles_then_caps unit test accurately verifies both the exponential doubling, the capping behaviors, and the cumulative sleep schedule bounds (3750ms).

Verdict: Approved. The PR is fully safe, correct, and ready for integration.