[vyncint]

What is wrong

Every seed that produces a finding gets a replay.sh next to its artifacts. Its whole purpose is to re-run that variant, and it re-exported three environment variables by name:

for key in ["CUDA_VISIBLE_DEVICES", "CUDA_LAUNCH_BLOCKING", "GEMM_SOL_PHASE"]

The variant itself runs with the campaign's whole environment — run_binary builds a Command and never calls env_clear — so the script reproduces strictly less than the run had. The tell is that one member of a family was carried and its siblings were not:

variable	read at run time by	carried
GEMM_SOL_PHASE	gemm_sol	yes
GEMM_SOL_MODE	gemm_sol_final	no
GEMM_SOL_VARIANT	gemm_sol_final	no
CUDA_OXIDE_TARGET	device_ffi_test, mathdx_ffi_test, small_type_ffi_test	no
MATHDX_ROOT	mathdx_ffi_test	no

GEMM_SOL_MODE and GEMM_SOL_VARIANT are documented, user-facing knobs — gemm_sol_final/README.md shows GEMM_SOL_MODE=plan GEMM_SOL_VARIANT=m512xn256 cargo oxide run gemm_sol_final, and the book's kernel-families page documents the variant override. They select which mode runs and which kernel variant is chosen.

So GEMM_SOL_MODE=bench GEMM_SOL_VARIANT=m512xn256 cargo oxide fuzz-schedule gemm_sol_final writes replay scripts that run mode=both, variant=auto. The script reproduces a different execution than the one that found the bug — the worst failure available to a reproduction artifact, because it does not error, it just quietly disagrees.

The change

Carry by prefix instead of by name, so a new knob in a family that already matters needs no edit here: CUDA_, GEMM_SOL_, MATHDX_. Deliberately not the whole environment — this text is written to a file in the artifact directory, and only the CUDA knobs and the example-suite families belong there. (mathdx_ffi_test also reads HOME, which is not carried and does not need to be: any shell running the script already has it.)

Two details the rewrite fixes on the way:

• the exports are sorted. std::env::vars has no defined order and the script is an artifact, so the same environment now always produces the same file.
• a name a POSIX shell cannot assign is dropped. One export BAD-NAME=... would fail under the script's own set -eu and take every later export and the exec with it.

The selection is split into replay_exports, a pure function over (key, value) pairs, so all of this is testable without mutating the process environment — which set_var is unsafe for in edition 2024, and which races across test threads.

Verification

Five tests: every variable an in-tree example reads is carried; nothing outside those families is (PATH, HOME, LD_LIBRARY_PATH, RUSTFLAGS, AWS_SECRET_ACCESS_KEY, and a near-miss MYCUDA_THING); the order is stable under a shuffled input; an unassignable name is dropped; and a value containing a single quote still round-trips through shell_quote_value.

Reverting to the three hard-coded names fails three of them, dropping the name filter fails one, dropping the sort fails two.

19 tests pass, fmt clean, clippy --all-targets -- -D warnings clean, cargo doc --document-private-items clean.

One file. No open PR touches it; it shares campaign.rs with my #1134 and #1135, in a disjoint region, and all six of this batch merge to a byte-identical tree in either order.

[nihalpasham]

• Superseded by the formal changes-requested review on the current head.

[nihalpasham]

• Blocker: broad prefix capture persists unrelated secrets such as CUDA_API_TOKEN, while omitted variables are inherited from the replay shell. The recorded environment therefore is neither minimal nor reproducible.
• Replay scripts are also created permissively and changed to 0755 afterward; that leaves a disclosure window and makes captured values readable to other users.
• A complete fix needs one typed replay-environment schema used by both campaign and replay, env_clear, explicit captured/unset/required-external fields, and atomic creation with restrictive permissions.
• Add GPU-independent tests for secret exclusion, absent-variable replay, non-Unicode values, inherited-variable clearing, and file/directory modes. The current tests instead pin the unsafe prefix behavior.

[vyncint]

Rewritten against both blockers. Force-pushed as a single commit; the prefix rule and the tests that pinned it are gone.

Secrets. Carrying by prefix could not tell CUDA_VISIBLE_DEVICES from CUDA_API_TOKEN, and you were right that the old test made it look safe: it used AWS_SECRET_ACCESS_KEY, a name outside every captured prefix, so it asserted an exclusion the rule never provided. CAPTURED_ENV is now seven explicit names, each read at run time by something in-tree — the two driver knobs, CUDA_OXIDE_TARGET, the three GEMM_SOL_* knobs, MATHDX_ROOT. A token cannot be captured whatever it is called.

Reproducibility. One ReplayEnvironment with the three fields you asked for. required_external (HOME, LD_LIBRARY_PATH, PATH) is passed to the child and never recorded; the script guards each with : "${NAME:?required by this run and not recorded here}", so a replay in the wrong environment refuses rather than substituting an ambient value. unset NAME covers a captured name the campaign did not have, for the same reason. A non-UTF-8 value joins required_external — the child still gets it, but no POSIX script can carry it, so the script states the requirement instead of writing a lossy guess.

run_binary calls env_clear. Measured here, a child without it inherits 96 variables from the campaign, SSH_AUTH_SOCK among them.

Modes. Script and seed directory are created owner-only. OpenOptions::mode applies at creation, so the umask window is gone, and an explicit remove before create_new stops a stale file surviving with its old mode.

Two refactors the change forced: RunEnvironment bundles the captured environment with the process environment it came from, and SeedContext collects the six per-campaign constants that run_seed was taking one parameter at a time.

Seven tests, each shown to bite by reverting the thing it covers:

mutation	caught by
drop env_clear	96 parent variables reach the child
add CUDA_API_TOKEN to CAPTURED_ENV	secret exclusion, and the clear test
lossy-convert a non-UTF-8 value	the non-Unicode requirement test
fs::write then chmod	the file/directory mode test
a comment instead of ${NAME:?}	the refusal test
stop emitting unset	the absent-variable test
seed directory at 0755	the file/directory mode test

The refusal test executes the generated script under /bin/sh with a cleared environment and asserts it fails naming the missing variable, then succeeds once supplied. The clear test runs a real child through run_binary and asserts no name from this process's own environment reached it.

21 tests, fmt, strict clippy and rustdoc pass; cargo-oxide's 177 still pass. Merges clean over your commits on #1134 and #1135 in either order.

One thing I have not verified, and would not want merged without it: env_clear changes how every variant is launched, and I have no hardware evidence that a real CUDA example still runs under the reduced environment. If REQUIRED_EXTERNAL_ENV is short by something libcuda needs, every campaign breaks and no local test would show it. Happy to run a campaign on an A10G and post the evidence — say the word, or tell me if you would rather add a name to that list up front.

[vyncint]

Hardware evidence for env_clear

Ran on an A10G. This was the one claim I could not make locally: env_clear changes how every variant is launched, and if REQUIRED_EXTERNAL_ENV were short by something libcuda needs, every campaign would break and no unit test would show it.

NVIDIA A10G, 8.6, 595.71.05        # compute capability 8.6, driver 595.71.05

cargo-oxide was rebuilt from the branch on the box (Compiling ptx-schedule … Finished release profile in 1m 48s), so the binary under test carries the change. The crate's 21 unit tests also pass there.

Two campaigns, both with a secret in the environment

Each ran with CUDA_API_TOKEN, CUDA_OXIDE_LICENSE_KEY and SENTINEL_UNRELATED exported alongside CUDA_VISIBLE_DEVICES=0 and GEMM_SOL_MODE=bench.

vecadd — 0 static sites, so this isolates the launch path:

schedule-fuzz: static sites=0 kinds=
schedule-fuzz: environment captured=2 unset=5 required-external=3
schedule-fuzz: baseline Pass
seed=0 PASS   seed=1 PASS   seed=2 PASS

redux_sum — 2 WarpCollective sites, so variants are genuinely perturbed:

static sites: 2 {'WarpCollective': 2}
baseline: Pass
  seed 0: Pass  injected=1  ns_per_visit=91
  seed 1: Pass  injected=1  ns_per_visit=762
  seed 2: Pass  injected=2  ns_per_visit=47923
  seed 3: Pass  injected=2  ns_per_visit=2890
  seed 4: Pass  injected=1  ns_per_visit=48149

The injections are real — nanosleep.u32 91; // ptx_schedule after in seed-0/module.ptx, and the per-seed count matches the report (seed-4 shows one site with two sleeps, which is the before-and-after placement branch).

So nine CUDA launches across the two campaigns, every one through run_binary with env_clear, all Pass. HOME, LD_LIBRARY_PATH and PATH are sufficient; nothing needed to be added to the list.

The secret never reaches disk

Grepping every byte of both artifact trees for the three sentinel values and their variable names:

clean: sk-live-MUST-NOT-REACH-DISK absent from every artifact
clean: lic-MUST-NOT-REACH-DISK absent from every artifact
clean: sentinel-MUST-NOT-REACH-DISK absent from every artifact
clean: CUDA_API_TOKEN absent from every artifact
clean: CUDA_OXIDE_LICENSE_KEY absent from every artifact
clean: SENTINEL_UNRELATED absent from every artifact
secret-leak result: 0 (0 = clean)

The replay script, as generated on the box

#!/bin/sh
set -eu
: "${HOME:?required by this run and not recorded here}"
: "${LD_LIBRARY_PATH:?required by this run and not recorded here}"
: "${PATH:?required by this run and not recorded here}"
unset CUDA_LAUNCH_BLOCKING
unset CUDA_OXIDE_TARGET
unset GEMM_SOL_MODE
unset GEMM_SOL_PHASE
unset GEMM_SOL_VARIANT
unset MATHDX_ROOT
export CUDA_VISIBLE_DEVICES='0'
cd '.../examples/redux_sum'
exec '/tmp/campaign2/seed-0/redux_sum'

Executed in a minimal environment (env -i plus the three required names), replaying a perturbed seed:

✓ All 8 warp sums correct (each = 496)
SUCCESS: redux.sync.add produced correct warp sums
replay exit: 0

And with a required name withheld, it refuses rather than replaying in the wrong environment:

replay.sh: 3: HOME: required by this run and not recorded here
refusal exit: 2

Modes on a real filesystem

drwx------  /tmp/campaign/seed-0
-rwx------  /tmp/campaign/seed-0/replay.sh

All five seed directories and all five scripts came out 700.

[vyncint]

Amended the commit message only — no code change. It still carried "Not verified: … I have no hardware evidence", which stopped being true once the A10G run in the comment above landed. Leaving a false statement in the permanent record seemed worse than a force-push, so the paragraph now states what was actually measured: two campaigns, nine launches through run_binary with the environment cleared, all Pass; HOME, LD_LIBRARY_PATH and PATH sufficient; the sentinels absent from every artifact; a perturbed seed replayed and reproduced; the guard refusing when a required name is withheld; 0700 throughout.

Code is byte-identical to what you last saw. Ready for another look whenever it suits.

[nihalpasham]

Closing this PR because campaign execution and generated replay are still not faithful to the same environment and artifact contract, and the remaining repair is a redesign rather than a small follow-up.

• The latest 5059bb4d push changes commit evidence only; its code tree is byte-identical to the previously reviewed head.
• The campaign clears ambient state, but generated replay inherits it. An executable check sees CUDA_DEVICE_ORDER unset in the campaign and PCI_BUS_ID during replay, so CUDA behavior can differ and ambient secrets or LD_PRELOAD can leak in.
• A campaign accepts HOME="" while replay rejects it, and required external values do not preserve absence, emptiness, or identity.
• Reusing a 0755 seed directory leaves it 0755; relative/non-UTF-8 paths, stale contents, symlinks, partial writes, and concurrent writers are also not handled transactionally.

The stock 21 tests pass because they test the pieces separately; the executable contract counterexamples fail. A replacement needs one process specification shared by campaign and replay, exact environment-state semantics, ambient clearing, normalized/raw path policy, and exclusive atomic artifact creation with stale-path and concurrency tests.

Please align on that design first under the acceptance criteria shared in Discord.