Skip to content

fix(ptx-schedule): an example that declines to run is reported as a failure - #1135

Merged
nihalpasham merged 2 commits into
NVIDIA:mainfrom
vyncint:fix/declined-baseline-is-not-a-failure
Aug 23, 2026
Merged

nihalpasham merged 2 commits into
NVIDIA:mainfrom
vyncint:fix/declined-baseline-is-not-a-failure

Conversation

@vyncint

@vyncint vyncint commented Aug 22, 2026 •

Copy link
Copy Markdown
Contributor

What is wrong

RunKind::Skipped became reachable in #1112, when has_skip_marker learned the second spelling examples use to decline. Nothing consumes it distinctly. Both places that read the baseline test for Pass and treat everything else alike:

// campaign.rs
if !matches!(baseline.kind, RunKind::Pass) {
    ... println!("schedule-fuzz: BASELINE FAILURE: {:?}; ...")

// cargo-oxide/src/main.rs
Ok(summary) if !matches!(summary.baseline.kind, RunKind::Pass) => {
    eprintln!("Error: baseline example did not pass; ...");
    std::process::exit(1);
}

So cargo oxide fuzz-schedule redux_f32 on anything below sm_100a prints BASELINE FAILURE: Skipped, then Error: baseline example did not pass, and exits 1.

The example did not fail. It ran, declared it cannot run on this device, and exited 0 — which is precisely what scripts/smoketest.sh reports as PASS (skipped). #665 exists because reporting a decline as anything else made an arch-gated example indistinguishable from a broken one, and this is the same conflation in the campaign: sweeping a fleet, "below this example's arch floor" comes back red alongside a genuinely wedged device.

The change

Rather than repeat a matches! in two crates, the distinction becomes one typed definition. RunKind::baseline_verdict() returns Usable, Declined or Broken, and the campaign and the CLI both consult it. The match is exhaustive, so a new RunKind cannot join a class by accident — adding one now fails with E0004: non-exhaustive patterns instead of silently reading as broken.

Behaviour:

baseline before after
Skipped BASELINE FAILURE: Skipped, exit 1 BASELINE DECLINED, a Note: on stderr, exit 0
any other non-Pass BASELINE FAILURE: {kind}, exit 1 unchanged, and the CLI message now names the kind

--fail-on-finding is untouched and remains the switch for failing on real findings. If you would rather a decline carry a distinct nonzero code, that is a one-line change in the new arm — I picked 0 to match verdict_standard, but it is your call.

Verification

Two tests: the three classes pinned exhaustively over all eight RunKind variants, and that every finding kind is a Broken baseline — a mismatch is a finding in a variant, but as the baseline it means there is no ground truth to compare against. The first fails with Skipped moved back into the broken class; adding a ninth variant fails to compile.

16 ptx-schedule tests and 177 cargo-oxide tests pass, fmt clean, clippy --all-targets -- -D warnings clean on both crates, cargo doc --document-private-items clean.

Two files. No open PR touches either. It shares campaign.rs with my #1134 and #1137; the three touch disjoint regions and I verified all six of this batch merge to a byte-identical tree in either order.

Maintainer repair

  • Strong failures now beat skip text.
  • The CLI policy is explicit: declined exits 0 with zero seeds, broken exits 1, and usable findings obey --fail-on-finding.
  • 17 ptx-schedule and 178 cargo-oxide tests pass, with fmt, strict clippy, rustdoc, and diff checks.

@nihalpasham nihalpasham added bug Something isn't working fuzzer Fuzzing infrastructure (rustlantis, difftest) cargo-oxide The `cargo oxide` subcommand and build automation labels Aug 23, 2026
@nihalpasham

nihalpasham commented Aug 23, 2026 •

Copy link
Copy Markdown
Collaborator
  • A skipped baseline had its own verdict, but skip text could still hide a nonzero exit or mismatch, and the CLI exit policy was not tested.
  • I layered one commit that fixes the shared classifier and pins the CLI behavior: declined runs execute zero seeds and exit 0; broken baselines exit 1; usable findings obey --fail-on-finding.
  • All 17 ptx-schedule tests, 178 cargo-oxide tests, fmt, strict clippy, and rustdoc pass. Hosted CI is queued and waiting for explicit fork-run approval.
  • This builds on fix(ptx-schedule): a campaign reads one of the two skip spellings as a pass #1112; no issue closes on merge.

vyncint and others added 2 commits August 23, 2026 12:15
…ailure

`RunKind::Skipped` became reachable in NVIDIA#1112, when `has_skip_marker` learned
the second spelling examples use to decline. Nothing consumes it distinctly.
Both places that read the baseline test for `Pass` and treat everything else
alike:

    if !matches!(baseline.kind, RunKind::Pass) {
        ... println!("schedule-fuzz: BASELINE FAILURE: {:?}; ...")

    Ok(summary) if !matches!(summary.baseline.kind, RunKind::Pass) => {
        eprintln!("Error: baseline example did not pass; ...");
        std::process::exit(1);
    }

So `cargo oxide fuzz-schedule redux_f32` on anything below sm_100a prints
`BASELINE FAILURE: Skipped`, then `Error: baseline example did not pass`, and
exits 1. The example did not fail. It ran, declared it cannot run on this
device, and exited 0 -- which is precisely what `scripts/smoketest.sh` reports
as `PASS (skipped)`. NVIDIA#665 exists because reporting a decline as anything else
made an arch-gated example indistinguishable from a broken one, and this is the
same conflation in the campaign: sweeping a fleet, "below this example's arch
floor" comes back red alongside a genuinely wedged device.

Rather than repeat a `matches!` in two crates, the distinction becomes one
typed definition: `RunKind::baseline_verdict()` returns `Usable`, `Declined` or
`Broken`, and the campaign and the CLI both consult it. The match is exhaustive,
so a new `RunKind` cannot join a class by accident -- adding one now fails with
E0004 instead of silently reading as broken.

Behaviour: `Declined` prints `BASELINE DECLINED` and, in the CLI, a `Note:` on
stderr with exit **0**. Every other non-`Pass` baseline keeps `BASELINE
FAILURE` and exit 1, now with the kind in the message so the reader knows
whether it hung, crashed or wedged. `--fail-on-finding` is untouched and
remains the switch for failing on real findings. If you would rather a decline
carry a distinct nonzero code, that is a one-line change in the new arm.

Two tests: the three classes pinned exhaustively over all eight `RunKind`
variants, and that every *finding* kind is a `Broken` baseline -- a mismatch is
a finding in a variant, but as the baseline it means there is no ground truth
to compare against. The first fails with `Skipped` moved back into the broken
class.

Verified: 16 ptx-schedule tests and 177 cargo-oxide tests pass, fmt clean,
`clippy --all-targets -- -D warnings` clean on both crates, and
`cargo doc --document-private-items` clean.

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
Keep stronger process failures ahead of skips and make the CLI distinction between declined, broken, and finding-bearing campaigns explicit and testable.

Signed-off-by: nihalp <nihalp@nvidia.com>
@nihalpasham
nihalpasham force-pushed the fix/declined-baseline-is-not-a-failure branch from 1630ce3 to 327e8d6 Compare August 23, 2026 06:49

@nihalpasham nihalpasham left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Declined baselines now stay distinct from broken baselines and real findings.
  • Failure precedence and CLI exit behavior are pinned by regression and control tests.
  • All seven workflows pass on exact head 327e8d64; both commits have valid signatures and DCO trailers.

@nihalpasham
nihalpasham merged commit 233beb7 into NVIDIA:main Aug 23, 2026
51 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cargo-oxide The `cargo oxide` subcommand and build automation fuzzer Fuzzing infrastructure (rustlantis, difftest)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants