Skip to content

test: cover middleware effects and export - #14441

Open
jyaunches wants to merge 4 commits into
NVIDIA:v1from
jyaunches:codex/middleware-boundary-tests
Open

jyaunches wants to merge 4 commits into
NVIDIA:v1from
jyaunches:codex/middleware-boundary-tests

Conversation

@jyaunches

@jyaunches jyaunches commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Extend NemoClaw's tests to observe middleware request effects in the docker-dummy live lifecycle and to keep an authored middleware binding through bundled CLI export and SDK reapply.

Problem

Existing middleware tests check configuration projection and document compilation. They do not observe middleware interception and destination effects in the live Docker lifecycle. The bundled CLI export/reapply journey has no authored middleware binding.

Change

  • docker-dummy's gateway registers an external gRPC middleware that hcl_live::case serves on the host, beside an HTTP effect recorder; both write the requests they see to middleware-effects.json in the run's reports. One sandbox command sends three requests, and the same run block asserts on the counts:
    • a request the middleware allows reaches the middleware and the recorder once;
    • a request it denies returns 403 and reaches only the middleware;
    • a request to the default Docker bridge's gateway, which the policy permits and the binding excludes, reaches only the recorder.
  • cargo ci live-docker binds the middleware and recorder ports and the two addresses to each run.
  • The export/reapply integration journey's second sandbox gets an explicit policy with a middleware binding; the first sandbox keeps the network-grant coverage. The existing CLI export comparison and SDK no-change/no-additional-effects assertions cover it.
  • The live test guide describes the new checks.
  • The case adds no reapply: its unchanged plan already checks unchanged reconciliation.
  • Dependency pins, Hermes, and native controls are unchanged. The live case uses Fabric's dummy adapter; it does not qualify Hermes or OpenClaw behavior.

Beyond CI

cargo ci live-docker passed on linux/arm64 against v1's Fabric pin, b4725580 (NVIDIA/NeMo-Fabric#632's head, which includes NVIDIA/NeMo-Fabric#627). With the destination expectation deliberately changed to include /deny, the middleware run block failed and reported {"destination":{"/allow":1,"/bypass":1},"middleware":{"/allow":1,"/deny":1}}.

@copy-pr-bot

copy-pr-bot Bot commented Nov 12, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@jyaunches

Copy link
Copy Markdown
Contributor Author
Authoring session — opening

Authoring cutoff: 2026-11-12 20:20:41 UTC. Scope: this test implementation through commit 2f55ad4124, before publication and Fabric PR-head validation. No inherited authoring activity or subagents.

Field Status Value Evidence or limit
Harness recorded Codex Desktop; CLI 0.166.0-alpha.5.1 Persisted session metadata and desktop instructions.
Skills recorded fabric-boundary-testing-reference; write-tests; test-driven-development; nemoclaw-contributor-create-pr; rust-lsp Complete file reads in the authoring record; rust-lsp guidance was read, with no semantic navigation call needed.
Instruction files recorded AGENTS.md; WRITING.md; PULL_REQUEST_TEMPLATE.md; SESSION_REPORT_TEMPLATE.md Startup instructions and complete reads from the isolated V1 checkout.
Tool calls recorded Wrapper calls: exec 70; request_user_input_async 1. Nested calls: shell/process results 143; apply_patch 10 Attempted wrapper calls include failures. Nested shell/process invocations counted separately from returned results; patch calls from executed wrapper inputs. Totals are not added together.
Tokens recorded input: 8453450; cached input: 8266112; output: 32752 Last actual cumulative token_count total_token_usage snapshot at the cutoff. No compactions or counter resets in the record; excludes subsequent publication and validation.
Session window recorded 2026-11-12T19:56:46Z to 2026-11-12T20:20:41Z; 23 min 55 s Whole authoring session through cutoff, including build waits.
Compactions recorded 0 Persisted record through cutoff.
Models recorded gpt-6-astra Recorded turn context; no changes.
Reasoning levels recorded medium Recorded turn context; no changes.

Local validation at opening:

  • V1 base: cb22d35ecd88286dabfda52ed1fded6ae346dea1; test commit: 2f55ad4124.
  • Unchanged Fabric pin: f33fa19a1aff584625e02af9007d8113e7b95dd8; Hermes source: bf0203a794a07aac3ff764972ab9087baa4e4f4e (release image v0.22.5).
  • cargo ci tools and cargo ci bundle passed on macOS ARM64. Verified bundle: 0.1.0-dev.87a5f7a9d6a56f42; built before final HCL formatting and live-case filesystem-policy alignment, which do not alter the exported-binding integration fixture.
  • Both modified E2E/integration binaries compiled. Targeted cargo clippy --locked -p nemoclaw-test-fixtures -p nemoclaw-e2e -p nemoclaw-build --all-targets -- -D warnings, Rust formatting, pinned OpenTofu formatting, and git diff --check passed.
  • deployment::authored_settings_survive_cli_export_and_drift_stops_plan_and_export passed against that bundle, failed at the CLI export equality when only the expected middleware name was deliberately changed, then passed after restoring the expectation. The temporary mutation is absent from the commit.
  • Seven HCL runner unit tests and seven live-Docker runner binding/cleanup tests passed. These are not deployed middleware-interception evidence.
  • Actual local Docker validation and the unpublished latest-Fabric-PR-head override follow this draft opening. The approved task-owned Linux VM has Landlock and Docker's containerd image store. The live test, its negative assertion checks, and updated-pin qualification are not yet claimed as passing.

The mandatory merge gate in the PR description remains outstanding. Testing Fabric's open PR head will not replace qualification after Fabric merges and V1 updates its pin.

@jyaunches

Copy link
Copy Markdown
Contributor Author
Local validation — Fabric PR #627 head

Both requested test extensions passed on Linux ARM64 with a temporary, unpublished override of Fabric PR #627. This does not satisfy the merge gate: keep this PR draft and unmerged until #627 lands, V1's pin includes it, and this PR is requalified against that V1 pin.

Test commit: 2f55ad412429572ea3ba43136d0d12abf0a3c87c, based on V1 cb22d35ecd88286dabfda52ed1fded6ae346dea1.
Fabric head: d787d90d3f619fb2b64731a9a10c6b5199750ec5, re-resolved before execution while #627 was open.

The disposable validation tree changed the Fabric source archive/checksum, regenerated its adapter catalog, and updated all three Rust nemo-fabric-core dependency revisions and Cargo.lock. Cargo also reselected existing Windows-only dependency edges; no package versions were upgraded. The published test branch and its pins remain unchanged. Hermes remains at bf0203a794a07aac3ff764972ab9087baa4e4f4e with its existing image pin.

Observed results:

  • Built the matched Linux bundle and test archives; qualified the matched dummy image.
  • Bundled CLI export/reapply integration: 1 passed (15.2 s).
  • Live docker-dummy lifecycle: all 16 HCL runs passed (26.4 s). Allowed traffic reached middleware and destination once; denied traffic reached middleware once and destination zero times; excluded traffic reached destination once and middleware zero times. Reapply reported no changes and preserved sandbox 670a5bd3-59d5-4c1c-8c7d-71ecd28a4e11.
  • Standard ci live-docker --archive-file .build/ci/live.tar.zst --hcl-cases docker-dummy: all four tests passed, including both gateway tests and the Fabric cache test. Test execution took 32.8 s; preparation and execution together took 389 s. The runner completed cleanup.
  • Negative checks in isolated copies: all six deliberately wrong effect counts failed; inverted no-change expectation failed; wrong baseline sandbox identity failed. Restoring the committed assertions produced the positive run above. Each run's deployment was destroyed and retained resources removed.

Artifact SHA-256 identities:

Artifact Identity
Fabric source archive 292e598c67dbe4af6047f1a04d5b9a33ca35ab0aac196e696453e9582529bafb
Regenerated catalog 0d29bb5823f282c4e75b412498ce7322c0821b2a0ae41ac86c06c6dd922288a8
Cargo.lock f68d90fe11b09cdcfc1ed24e92b3a413b669eb4aeda05cd8ca58c588a2a599fe
Bundle manifest (0.1.0-dev.1c1338fc9b79d282) fad41d21db180d7882934c0fd87df88b44f244348e73b766d63aaa036525528e
Integration test archive 1f601ca9e00b9428ad2a4eefe446746f7dd12912578949cc2304d096345d94b4
Live test archive 8dd9ca27abb1fe7caf0114b934466c5017c135152e761b4ef277bd97ffa45a60
Qualified dummy image 256366d3bba734f8f4c5b85cc89b1ef87c86a9d3b5d1af73a954c7b9f3f0a9f7
Derived dummy image used by focused runs a67fd74013a3fe062905c663e22a45d2e9c2d3fb45e0f4384346dce8a9e3da17

@wscurran wscurran added area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure area: policy Network policy, egress rules, presets, or sandbox policy area: security Security controls, permissions, secrets, or hardening platform: container Affects Docker, containerd, Podman, or images v1 NemoClaw v1 branch labels Nov 12, 2026
cv added 2 commits November 16, 2026 14:37
…ry-tests

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

# Conflicts:
#	crates/nemoclaw-e2e/tests/e2e/hcl_live.rs
#	crates/nemoclaw-e2e/tests/integration/rust/deployment.rs
- The case's fixture writes its request counts to the run's reports, so the
  docker-dummy run that sends the requests asserts on them directly; the
  middleware module and its observation probe go.
- One sandbox command sends the three requests.
- Drop the unchanged reapply and sandbox identity check: the case's
  unchanged plan already shows no changes.
- Bind the middleware ports and addresses inside dummy_case, before the
  UID and digest.
- The export journey's middleware policy keeps only the fields the binding
  needs.
- Document the middleware checks in the live test guide.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cv cv changed the title test: cover middleware effects and export (blocked on Fabric #627) test: cover middleware effects and export Nov 16, 2026
@cv
cv marked this pull request as ready for review November 16, 2026 21:55
@cv

cv commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

/ok to test b033521

The dummy case binding ran docker network inspect, so its unit test failed on
hosts without Docker, such as the macOS runner. prepare reads the address and
passes it in, and the test checks that the binding selects both recorder
addresses.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure area: policy Network policy, egress rules, presets, or sandbox policy area: security Security controls, permissions, secrets, or hardening platform: container Affects Docker, containerd, Podman, or images v1 NemoClaw v1 branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants