Skip to content

Run production dataplane tests and simplify harness architecture - #14

Open
lucarlig wants to merge 6 commits into
mainfrom
user/luca/production-dataplane-tests
Open

Run production dataplane tests and simplify harness architecture#14
lucarlig wants to merge 6 commits into
mainfrom
user/luca/production-dataplane-tests

Conversation

@lucarlig

@lucarlig lucarlig commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Standalone workflows run against production dataplane images without with_tools. The harness owns ephemeral RSA signing, loopback JWKS, fixture discovery and MessagePack configuration publishing. Official conformance and Inspector install their npm packages in a Docker image and run there; the CLI no longer needs host Node/npm or a local npm cache.

The architecture review covered CLI/configuration, checkouts and assets, Docker routing/lifecycle, MCP and Locust adapters, conformance, and CI/release orchestration. Changes:

  • Run auth, JWT signing, fixture discovery and Redis publishing in the existing Rust executable. Reuse the shared MCP metadata and JSON/SSE parser. Remove repository-owned JavaScript helpers, inline JavaScript tests, and the helper npm package. Python remains for Locust and upstream live-test integration.
  • Build helper and tooling images from the binary's embedded Rust sources and locked dependencies. The tooling image installs the pinned official conformance and Inspector packages, including the system CA certificates required by the Rust HTTP client. Installed CLI users need neither a repository checkout nor host Rust/Node/npm.
  • Run the upstream tools, authentication proxies and Rust client driver in containers. The client driver publishes directly to Redis, removing nested Compose calls, serialized Compose arguments, host executable/PATH discovery and host-gateway mappings. No Docker socket is mounted.
  • Share the fixture network namespace for direct conformance so upstream localhost/DNS-rebinding checks remain valid. Routed runners share nginx's namespace; authentication proxies preserve the public Host/Origin contract and deliberately mutated security-test headers. Reports use host-owned bind mounts.
  • Share runner image builds, command execution and explicit container removal after success, failure or interruption. Preserve Docker environment and working directory during removal so repo-configured connection settings select the same daemon. Apply fixture customization through a checked declarative patch against the pinned upstream revision; LF attributes preserve it on Windows.
  • Return CommandSpec directly instead of StackCommandPlan, remove unused MCP transport features, and consolidate conformance artifact validation, baseline evaluation and reporting. Share asynchronous process handling and the CI/release quality workflow.

Public routing, published-image defaults, managed cleanup and transactional baseline updates are preserved. Rust dependencies replace the removed npm helper dependencies; the cryptographic backend was already used by the HTTP stack.

Validation at 957eeb1:

  • Paired with dataplane PR #139 at 7f1961c: a fresh ARM64 production release image and the exact Linux x86-64 CI artifact both pass the production health/auth boundary checks and authenticated probe. Load smoke completes 42 and 27 requests respectively, with zero failures.
  • The CLI installed successfully from its pinned Git revision, then downloaded and packaged the PR-head CI artifact through ci prepare-image. Each image completes strict make conformance using dataplane-owned baselines: 44 passed, 40 expected failures, no unexpected failures/passes, skips or unknowns. All 311 checks match between images. Both runs leave host Node/npm untouched and preserve both repositories' baselines.
  • Linux, macOS, Windows, quality and DCO checks pass (CI run).
  • 398 local tests pass, including Docker connection settings on cleanup, runner exit classification, failure and cancellation. Clippy with all targets/features and -D warnings, formatting, package verification and Actionlint pass. Cargo audit reports zero vulnerabilities and no warnings.
  • A fresh installed binary outside the checkout passes standalone modern probe, modern load smoke (46 requests, zero failures), and legacy Inspector against a production dataplane image without with_tools.
  • With host node, npm and npx replaced by failing traps, all 84 modern conformance scenarios complete. No trap is invoked and no host npm cache is created.
  • All 311 check identifiers, names and statuses match the preceding PR head a44a505: 148 direct-fixture checks, 124 external server checks and 39 client checks. Server scenarios total 41 passed, 16 expected failures and 23 failures; client scenarios total 3 passed and 1 expected failure. The same existing local-image baseline mismatches cause exit 1. Baselines are unchanged.
  • SIGINT during the official runner removes its container and fixture, stops later matrix entries and preserves a baseline sentinel. An invalid signing key skips the workload and removes the partial stack.
  • Temporary validation containers, networks and volumes were removed; the original user containers remain running.
  • Earlier validation built a Linux helper image from a packaged-crate binary's embedded, normalized Cargo manifest and sources.

Inspector's pinned upstream release uses initialization; its documented invocation selects the legacy protocol. Containerization does not add modern Inspector support.

Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
@lucarlig lucarlig changed the title Run standalone tests without dataplane testing helpers Run production dataplane tests and simplify harness architecture Sep 7, 2026
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Signed-off-by: lucarlig <luca.carlig@ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant