Repository navigation
fix(ptx-schedule): a campaign reads one of the two skip spellings as a pass - #1112
Merged
nihalpasham merged 1 commit intoAug 21, 2026
Merged
Conversation
…a pass
`has_skip_marker` accepts `skipping:` and nothing else. The convention it is
implementing has two spellings, and `scripts/smoketest.sh` -- which owns it --
accepts both:
if grep -qiE '^[[:space:]]*(skipping:|pass \(skipped\))' "${log}"; then
Its own comment names the second form in so many words: "`PASS (skipped): ...`
form below sm_75". `generated_ldmatrix` prints exactly that:
println!(
"PASS (skipped): ldmatrix.m8n8.x4.b16 requires sm_75+; device is sm_{major}{minor}"
);
An example declining that way exits 0 and prints no mismatch marker, so
`run_binary` classifies the run `RunKind::Pass`. That matters because the
campaign gates on it:
if !matches!(baseline.kind, RunKind::Pass) {
... println!("schedule-fuzz: BASELINE FAILURE: {:?}; no schedule
variants were run", ...)
With the right kind the campaign stops there and says so. With `Pass` it
proceeds: it patches the PTX, sweeps every seed, and each variant declines
identically -- same exit code, same output, so not even `OutputChanged` fires.
The campaign reports a clean sweep of a kernel that never launched.
This is NVIDIA#665's bug in the second consumer of the same convention. That fix
put it as "a skip the function fails to recognise is not merely missing its
`(skipped)` annotation -- it is promoted to a full execution PASS", and taught
`verdict_standard` the second spelling. `ptx-schedule`'s campaign arrived later
(NVIDIA#1065) with only the first. A silent all-clear is the worst outcome available
to a bug-finding tool, and it is the one this produces on any device below an
example's floor.
Hoist both spellings into a `SKIP_MARKERS` constant and check the line against
each, with the coupling to the smoketest's pattern recorded next to it.
Case-insensitivity is unchanged and already correct: `run_binary` lowercases
before classifying, which is what makes this match the smoketest's `grep -i`.
Two tests: both spellings are recognised with and without leading whitespace,
and the near-misses are not -- a bare `pass`, `pass: 1024 elements verified`,
`success`, and a mid-line `no skipping: here`, since the marker is anchored to
the start of the line in the smoketest too. Both fail with the second marker
removed.
Verified: 11 tests pass, fmt clean, `clippy --all-targets -- -D warnings`
clean, and `cargo doc --document-private-items` clean.
Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
Collaborator
|
nihalpasham
approved these changes
Aug 21, 2026
This was referenced Aug 22, 2026
nihalpasham
added a commit
that referenced
this pull request
Aug 23, 2026
… does (#1134) * fix(ptx-schedule): the marker predicates fold case only if the caller does `has_skip_marker` and `has_mismatch_marker` implement a case-blind contract -- the smoketest greps the same markers with `grep -i` -- but neither folded case. They worked because their one caller lowercased first: let combined = format!("{stdout}\n{stderr}").to_ascii_lowercase(); That is a precondition living at a call site, in a crate whose one job is classifying process output. #1112's review named it as a residual. Called with a raw stream, both predicates silently disagree with their own names: has_skip_marker("Skipping: needs sm_90") -> false has_skip_marker("PASS (skipped): requires sm_75+") -> false has_mismatch_marker("MISMATCH at index 3") -> false Every one of those spellings is real. `Skipping:` and `PASS (skipped):` are what examples print -- `generated_ldmatrix` prints the second verbatim -- and `MISMATCH` in capitals is the conventional spelling for the finding, the one `crates/fuzzer` uses in its own status vocabulary. A caller that reached for these predicates without knowing the rule would classify a declined run as a pass and a failed comparison as a pass, which is the exact misclassification #1112 fixed. Fold case in the predicates instead of documenting the precondition: `has_skip_marker` compares the line's leading bytes with `eq_ignore_ascii_case`, and `has_mismatch_marker` lowercases once inside itself. `run_binary` then passes the raw combined stream, so the precondition is gone rather than moved -- and the one allocation moves from the call site into the predicate that needs it. Behaviour on the existing path is unchanged: the input was already lowercase there. One test over the spellings that matter, in both directions: five mixed-case skip declines and five mixed-case failure markers must match, and folding case must not widen what counts as a marker -- a bare `pass`/`PASS`, `pass: 1024 elements verified`, `SUCCESS`, a mid-line `no skipping: here`, and `all checks passed` all stay negative. It fails with the folding reverted. Verified: 15 tests pass, fmt clean, `clippy --all-targets -- -D warnings` clean, `cargo doc --document-private-items` clean. Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com> * fix(ptx-schedule): classify failures before skips Fold marker case at the predicates and classify timeout, process failure, and mismatch before accepting a graceful skip. Signed-off-by: nihalp <nihalp@nvidia.com> --------- Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com> Signed-off-by: nihalp <nihalp@nvidia.com> Co-authored-by: nihalp <nihalp@nvidia.com>
nihalpasham
pushed a commit
to vyncint/cuda-oxide
that referenced
this pull request
Aug 23, 2026
…ailure `RunKind::Skipped` became reachable in NVIDIA#1112, when `has_skip_marker` learned the second spelling examples use to decline. Nothing consumes it distinctly. Both places that read the baseline test for `Pass` and treat everything else alike: if !matches!(baseline.kind, RunKind::Pass) { ... println!("schedule-fuzz: BASELINE FAILURE: {:?}; ...") Ok(summary) if !matches!(summary.baseline.kind, RunKind::Pass) => { eprintln!("Error: baseline example did not pass; ..."); std::process::exit(1); } So `cargo oxide fuzz-schedule redux_f32` on anything below sm_100a prints `BASELINE FAILURE: Skipped`, then `Error: baseline example did not pass`, and exits 1. The example did not fail. It ran, declared it cannot run on this device, and exited 0 -- which is precisely what `scripts/smoketest.sh` reports as `PASS (skipped)`. NVIDIA#665 exists because reporting a decline as anything else made an arch-gated example indistinguishable from a broken one, and this is the same conflation in the campaign: sweeping a fleet, "below this example's arch floor" comes back red alongside a genuinely wedged device. Rather than repeat a `matches!` in two crates, the distinction becomes one typed definition: `RunKind::baseline_verdict()` returns `Usable`, `Declined` or `Broken`, and the campaign and the CLI both consult it. The match is exhaustive, so a new `RunKind` cannot join a class by accident -- adding one now fails with E0004 instead of silently reading as broken. Behaviour: `Declined` prints `BASELINE DECLINED` and, in the CLI, a `Note:` on stderr with exit **0**. Every other non-`Pass` baseline keeps `BASELINE FAILURE` and exit 1, now with the kind in the message so the reader knows whether it hung, crashed or wedged. `--fail-on-finding` is untouched and remains the switch for failing on real findings. If you would rather a decline carry a distinct nonzero code, that is a one-line change in the new arm. Two tests: the three classes pinned exhaustively over all eight `RunKind` variants, and that every *finding* kind is a `Broken` baseline -- a mismatch is a finding in a variant, but as the baseline it means there is no ground truth to compare against. The first fails with `Skipped` moved back into the broken class. Verified: 16 ptx-schedule tests and 177 cargo-oxide tests pass, fmt clean, `clippy --all-targets -- -D warnings` clean on both crates, and `cargo doc --document-private-items` clean. Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
nihalpasham
added a commit
that referenced
this pull request
Aug 23, 2026
…ailure (#1135) * fix(ptx-schedule): an example that declines to run is reported as a failure `RunKind::Skipped` became reachable in #1112, when `has_skip_marker` learned the second spelling examples use to decline. Nothing consumes it distinctly. Both places that read the baseline test for `Pass` and treat everything else alike: if !matches!(baseline.kind, RunKind::Pass) { ... println!("schedule-fuzz: BASELINE FAILURE: {:?}; ...") Ok(summary) if !matches!(summary.baseline.kind, RunKind::Pass) => { eprintln!("Error: baseline example did not pass; ..."); std::process::exit(1); } So `cargo oxide fuzz-schedule redux_f32` on anything below sm_100a prints `BASELINE FAILURE: Skipped`, then `Error: baseline example did not pass`, and exits 1. The example did not fail. It ran, declared it cannot run on this device, and exited 0 -- which is precisely what `scripts/smoketest.sh` reports as `PASS (skipped)`. #665 exists because reporting a decline as anything else made an arch-gated example indistinguishable from a broken one, and this is the same conflation in the campaign: sweeping a fleet, "below this example's arch floor" comes back red alongside a genuinely wedged device. Rather than repeat a `matches!` in two crates, the distinction becomes one typed definition: `RunKind::baseline_verdict()` returns `Usable`, `Declined` or `Broken`, and the campaign and the CLI both consult it. The match is exhaustive, so a new `RunKind` cannot join a class by accident -- adding one now fails with E0004 instead of silently reading as broken. Behaviour: `Declined` prints `BASELINE DECLINED` and, in the CLI, a `Note:` on stderr with exit **0**. Every other non-`Pass` baseline keeps `BASELINE FAILURE` and exit 1, now with the kind in the message so the reader knows whether it hung, crashed or wedged. `--fail-on-finding` is untouched and remains the switch for failing on real findings. If you would rather a decline carry a distinct nonzero code, that is a one-line change in the new arm. Two tests: the three classes pinned exhaustively over all eight `RunKind` variants, and that every *finding* kind is a `Broken` baseline -- a mismatch is a finding in a variant, but as the baseline it means there is no ground truth to compare against. The first fails with `Skipped` moved back into the broken class. Verified: 16 ptx-schedule tests and 177 cargo-oxide tests pass, fmt clean, `clippy --all-targets -- -D warnings` clean on both crates, and `cargo doc --document-private-items` clean. Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com> * fix(ptx-schedule): pin baseline and CLI verdict policy Keep stronger process failures ahead of skips and make the CLI distinction between declined, broken, and finding-bearing campaigns explicit and testable. Signed-off-by: nihalp <nihalp@nvidia.com> --------- Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com> Signed-off-by: nihalp <nihalp@nvidia.com> Co-authored-by: nihalp <nihalp@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is wrong
has_skip_markeracceptsskipping:and nothing else. The convention it is implementing has two spellings, andscripts/smoketest.sh— which owns it — accepts both:Its own comment names the second form in so many words: "
PASS (skipped): ...form below sm_75".generated_ldmatrixprints exactly that:Why it matters
An example declining that way exits 0 and prints no mismatch marker, so
run_binaryclassifies the runRunKind::Pass. The campaign gates on precisely that:With the right kind the campaign stops there and says so. With
Passit proceeds: it patches the PTX, sweeps every seed, and each variant declines identically — same exit code, same output, so not evenOutputChangedfires. The campaign reports a clean sweep of a kernel that never launched.This is #665's bug in the second consumer of the same convention. That fix put it as "a skip the function fails to recognise is not merely missing its
(skipped)annotation — it is promoted to a full execution PASS", and taughtverdict_standardthe second spelling.ptx-schedule's campaign arrived later (#1065) with only the first. A silent all-clear is the worst outcome available to a bug-finding tool, and it is the one this produces on any device below an example's floor.The change
Hoist both spellings into a
SKIP_MARKERSconstant and check the line against each, with the coupling to the smoketest's pattern recorded next to it.Case-insensitivity is unchanged and already correct:
run_binarylowercases the combined output before classifying, which is what makes this match the smoketest'sgrep -i.Tests
Two:
both_smoketest_skip_spellings_are_recognised— both forms, with and without leading whitespace; and the near-misses are rejected: a barepass,pass: 1024 elements verified,success, and a mid-lineno skipping: here, since the marker is anchored to the start of the line in the smoketest tooa_declined_run_is_skipped_not_passed— the two forms as they actually arrive, newline-terminatedBoth fail with the second marker removed.
One note, out of scope
smoketest.shitself is not uniform about this:verdict_standard(line 397) takes both spellings, but the ltoir verdict near line 609 greps only^[[:space:]]*skipping:. That one is inert today — everyLTOIR_EXAMPLESmember uses the first spelling, andgenerated_ldmatrixisstandard, so it goes throughverdict_standard. I have left it alone since #1102 is currently editing that script; happy to follow up once it lands if you would like the two verdicts aligned.Verification
All local, no GPU needed.
cargo test -p ptx-schedule --all-targets: 11 passed, 0 failedcargo fmt --check -p ptx-schedule: cleancargo clippy -p ptx-schedule --all-targets -- -D warnings: cleancargo doc --no-deps --document-private-items -p ptx-schedule: cleanOne file,
crates/ptx-schedule/src/campaign.rs. Checked against every open PR's file list viagh api --paginate— no overlap, including with my #1108/#1109/#1111 on this crate, which touchCargo.toml,README.md,main.rsandlib.rsrespectively.