Repository navigation
fix(ptx-schedule): an example that declines to run is reported as a failure - #1135
Merged
nihalpasham merged 2 commits intoAug 23, 2026
Merged
Conversation
Collaborator
|
…ailure `RunKind::Skipped` became reachable in NVIDIA#1112, when `has_skip_marker` learned the second spelling examples use to decline. Nothing consumes it distinctly. Both places that read the baseline test for `Pass` and treat everything else alike: if !matches!(baseline.kind, RunKind::Pass) { ... println!("schedule-fuzz: BASELINE FAILURE: {:?}; ...") Ok(summary) if !matches!(summary.baseline.kind, RunKind::Pass) => { eprintln!("Error: baseline example did not pass; ..."); std::process::exit(1); } So `cargo oxide fuzz-schedule redux_f32` on anything below sm_100a prints `BASELINE FAILURE: Skipped`, then `Error: baseline example did not pass`, and exits 1. The example did not fail. It ran, declared it cannot run on this device, and exited 0 -- which is precisely what `scripts/smoketest.sh` reports as `PASS (skipped)`. NVIDIA#665 exists because reporting a decline as anything else made an arch-gated example indistinguishable from a broken one, and this is the same conflation in the campaign: sweeping a fleet, "below this example's arch floor" comes back red alongside a genuinely wedged device. Rather than repeat a `matches!` in two crates, the distinction becomes one typed definition: `RunKind::baseline_verdict()` returns `Usable`, `Declined` or `Broken`, and the campaign and the CLI both consult it. The match is exhaustive, so a new `RunKind` cannot join a class by accident -- adding one now fails with E0004 instead of silently reading as broken. Behaviour: `Declined` prints `BASELINE DECLINED` and, in the CLI, a `Note:` on stderr with exit **0**. Every other non-`Pass` baseline keeps `BASELINE FAILURE` and exit 1, now with the kind in the message so the reader knows whether it hung, crashed or wedged. `--fail-on-finding` is untouched and remains the switch for failing on real findings. If you would rather a decline carry a distinct nonzero code, that is a one-line change in the new arm. Two tests: the three classes pinned exhaustively over all eight `RunKind` variants, and that every *finding* kind is a `Broken` baseline -- a mismatch is a finding in a variant, but as the baseline it means there is no ground truth to compare against. The first fails with `Skipped` moved back into the broken class. Verified: 16 ptx-schedule tests and 177 cargo-oxide tests pass, fmt clean, `clippy --all-targets -- -D warnings` clean on both crates, and `cargo doc --document-private-items` clean. Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
Keep stronger process failures ahead of skips and make the CLI distinction between declined, broken, and finding-bearing campaigns explicit and testable. Signed-off-by: nihalp <nihalp@nvidia.com>
nihalpasham
force-pushed
the
fix/declined-baseline-is-not-a-failure
branch
from
August 23, 2026 06:49
1630ce3 to
327e8d6
Compare
nihalpasham
approved these changes
Aug 23, 2026
nihalpasham
left a comment
Collaborator
There was a problem hiding this comment.
- Declined baselines now stay distinct from broken baselines and real findings.
- Failure precedence and CLI exit behavior are pinned by regression and control tests.
- All seven workflows pass on exact head
327e8d64; both commits have valid signatures and DCO trailers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is wrong
RunKind::Skippedbecame reachable in #1112, whenhas_skip_markerlearned the second spelling examples use to decline. Nothing consumes it distinctly. Both places that read the baseline test forPassand treat everything else alike:So
cargo oxide fuzz-schedule redux_f32on anything below sm_100a printsBASELINE FAILURE: Skipped, thenError: baseline example did not pass, and exits 1.The example did not fail. It ran, declared it cannot run on this device, and exited 0 — which is precisely what
scripts/smoketest.shreports asPASS (skipped). #665 exists because reporting a decline as anything else made an arch-gated example indistinguishable from a broken one, and this is the same conflation in the campaign: sweeping a fleet, "below this example's arch floor" comes back red alongside a genuinely wedged device.The change
Rather than repeat a
matches!in two crates, the distinction becomes one typed definition.RunKind::baseline_verdict()returnsUsable,DeclinedorBroken, and the campaign and the CLI both consult it. The match is exhaustive, so a newRunKindcannot join a class by accident — adding one now fails withE0004: non-exhaustive patternsinstead of silently reading as broken.Behaviour:
SkippedBASELINE FAILURE: Skipped, exit 1BASELINE DECLINED, aNote:on stderr, exit 0PassBASELINE FAILURE: {kind}, exit 1--fail-on-findingis untouched and remains the switch for failing on real findings. If you would rather a decline carry a distinct nonzero code, that is a one-line change in the new arm — I picked 0 to matchverdict_standard, but it is your call.Verification
Two tests: the three classes pinned exhaustively over all eight
RunKindvariants, and that every finding kind is aBrokenbaseline — a mismatch is a finding in a variant, but as the baseline it means there is no ground truth to compare against. The first fails withSkippedmoved back into the broken class; adding a ninth variant fails to compile.16 ptx-schedule tests and 177 cargo-oxide tests pass, fmt clean,
clippy --all-targets -- -D warningsclean on both crates,cargo doc --document-private-itemsclean.Two files. No open PR touches either. It shares
campaign.rswith my #1134 and #1137; the three touch disjoint regions and I verified all six of this batch merge to a byte-identical tree in either order.Maintainer repair
--fail-on-finding.