Skip to content

fix(ptx-schedule): a campaign reads one of the two skip spellings as a pass - #1112

Merged
nihalpasham merged 1 commit into
NVIDIA:mainfrom
vyncint:fix/ptx-schedule-skip-marker-spelling
Aug 21, 2026
Merged

nihalpasham merged 1 commit into
NVIDIA:mainfrom
vyncint:fix/ptx-schedule-skip-marker-spelling

Conversation

@vyncint

@vyncint vyncint commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

What is wrong

has_skip_marker accepts skipping: and nothing else. The convention it is implementing has two spellings, and scripts/smoketest.sh — which owns it — accepts both:

if grep -qiE '^[[:space:]]*(skipping:|pass \(skipped\))' "${log}"; then

Its own comment names the second form in so many words: "PASS (skipped): ... form below sm_75". generated_ldmatrix prints exactly that:

if compute_capability < 75 {
    println!(
        "PASS (skipped): ldmatrix.m8n8.x4.b16 requires sm_75+; device is sm_{major}{minor}"
    );

Why it matters

An example declining that way exits 0 and prints no mismatch marker, so run_binary classifies the run RunKind::Pass. The campaign gates on precisely that:

if !matches!(baseline.kind, RunKind::Pass) {
    ...
    println!("schedule-fuzz: BASELINE FAILURE: {:?}; no schedule variants were run", ...)

With the right kind the campaign stops there and says so. With Pass it proceeds: it patches the PTX, sweeps every seed, and each variant declines identically — same exit code, same output, so not even OutputChanged fires. The campaign reports a clean sweep of a kernel that never launched.

This is #665's bug in the second consumer of the same convention. That fix put it as "a skip the function fails to recognise is not merely missing its (skipped) annotation — it is promoted to a full execution PASS", and taught verdict_standard the second spelling. ptx-schedule's campaign arrived later (#1065) with only the first. A silent all-clear is the worst outcome available to a bug-finding tool, and it is the one this produces on any device below an example's floor.

The change

Hoist both spellings into a SKIP_MARKERS constant and check the line against each, with the coupling to the smoketest's pattern recorded next to it.

Case-insensitivity is unchanged and already correct: run_binary lowercases the combined output before classifying, which is what makes this match the smoketest's grep -i.

Tests

Two:

  • both_smoketest_skip_spellings_are_recognised — both forms, with and without leading whitespace; and the near-misses are rejected: a bare pass, pass: 1024 elements verified, success, and a mid-line no skipping: here, since the marker is anchored to the start of the line in the smoketest too
  • a_declined_run_is_skipped_not_passed — the two forms as they actually arrive, newline-terminated

Both fail with the second marker removed.

One note, out of scope

smoketest.sh itself is not uniform about this: verdict_standard (line 397) takes both spellings, but the ltoir verdict near line 609 greps only ^[[:space:]]*skipping:. That one is inert today — every LTOIR_EXAMPLES member uses the first spelling, and generated_ldmatrix is standard, so it goes through verdict_standard. I have left it alone since #1102 is currently editing that script; happy to follow up once it lands if you would like the two verdicts aligned.

Verification

All local, no GPU needed.

  • cargo test -p ptx-schedule --all-targets: 11 passed, 0 failed
  • cargo fmt --check -p ptx-schedule: clean
  • cargo clippy -p ptx-schedule --all-targets -- -D warnings: clean
  • cargo doc --no-deps --document-private-items -p ptx-schedule: clean

One file, crates/ptx-schedule/src/campaign.rs. Checked against every open PR's file list via gh api --paginate — no overlap, including with my #1108/#1109/#1111 on this crate, which touch Cargo.toml, README.md, main.rs and lib.rs respectively.

…a pass

`has_skip_marker` accepts `skipping:` and nothing else. The convention it is
implementing has two spellings, and `scripts/smoketest.sh` -- which owns it --
accepts both:

    if grep -qiE '^[[:space:]]*(skipping:|pass \(skipped\))' "${log}"; then

Its own comment names the second form in so many words: "`PASS (skipped): ...`
form below sm_75". `generated_ldmatrix` prints exactly that:

    println!(
        "PASS (skipped): ldmatrix.m8n8.x4.b16 requires sm_75+; device is sm_{major}{minor}"
    );

An example declining that way exits 0 and prints no mismatch marker, so
`run_binary` classifies the run `RunKind::Pass`. That matters because the
campaign gates on it:

    if !matches!(baseline.kind, RunKind::Pass) {
        ... println!("schedule-fuzz: BASELINE FAILURE: {:?}; no schedule
        variants were run", ...)

With the right kind the campaign stops there and says so. With `Pass` it
proceeds: it patches the PTX, sweeps every seed, and each variant declines
identically -- same exit code, same output, so not even `OutputChanged` fires.
The campaign reports a clean sweep of a kernel that never launched.

This is NVIDIA#665's bug in the second consumer of the same convention. That fix
put it as "a skip the function fails to recognise is not merely missing its
`(skipped)` annotation -- it is promoted to a full execution PASS", and taught
`verdict_standard` the second spelling. `ptx-schedule`'s campaign arrived later
(NVIDIA#1065) with only the first. A silent all-clear is the worst outcome available
to a bug-finding tool, and it is the one this produces on any device below an
example's floor.

Hoist both spellings into a `SKIP_MARKERS` constant and check the line against
each, with the coupling to the smoketest's pattern recorded next to it.
Case-insensitivity is unchanged and already correct: `run_binary` lowercases
before classifying, which is what makes this match the smoketest's `grep -i`.

Two tests: both spellings are recognised with and without leading whitespace,
and the near-misses are not -- a bare `pass`, `pass: 1024 elements verified`,
`success`, and a mid-line `no skipping: here`, since the marker is anchored to
the start of the line in the smoketest too. Both fail with the second marker
removed.

Verified: 11 tests pass, fmt clean, `clippy --all-targets -- -D warnings`
clean, and `cargo doc --document-private-items` clean.

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
@nihalpasham

Copy link
Copy Markdown
Collaborator
  • Confirmed both halves: smoketest.sh accepts two skip spellings while has_skip_marker knew one, so generated_ldmatrix's "PASS (skipped)" decline exited 0 with no marker, classified as Pass, passed the baseline gate, and the campaign swept seeds against a kernel that never launched. Both tests fail without the fix; no false positive is possible since a bare "pass"/"pass:" line cannot start with "pass (skipped)".
  • The SKIP_MARKERS constant documents the smoketest coupling at the right place; the convention's owner stays smoketest.sh.
  • Two pre-existing residuals for a follow-up issue, not this PR: gemm_sol_final and tcgen05_matmul's arch-gated fallbacks print no marker at all, so their declines still read as Pass baselines; and has_skip_marker's case-insensitivity relies on callers pre-lowercasing.

@nihalpasham nihalpasham added bug Something isn't working fuzzer Fuzzing infrastructure (rustlantis, difftest) labels Aug 21, 2026
@nihalpasham
nihalpasham merged commit fddcb17 into NVIDIA:main Aug 21, 2026
50 checks passed
nihalpasham added a commit that referenced this pull request Aug 23, 2026
… does (#1134)

* fix(ptx-schedule): the marker predicates fold case only if the caller does

`has_skip_marker` and `has_mismatch_marker` implement a case-blind contract --
the smoketest greps the same markers with `grep -i` -- but neither folded case.
They worked because their one caller lowercased first:

    let combined = format!("{stdout}\n{stderr}").to_ascii_lowercase();

That is a precondition living at a call site, in a crate whose one job is
classifying process output. #1112's review named it as a residual. Called with
a raw stream, both predicates silently disagree with their own names:

    has_skip_marker("Skipping: needs sm_90")                 -> false
    has_skip_marker("PASS (skipped): requires sm_75+")        -> false
    has_mismatch_marker("MISMATCH at index 3")                -> false

Every one of those spellings is real. `Skipping:` and `PASS (skipped):` are
what examples print -- `generated_ldmatrix` prints the second verbatim -- and
`MISMATCH` in capitals is the conventional spelling for the finding, the one
`crates/fuzzer` uses in its own status vocabulary. A caller that reached for
these predicates without knowing the rule would classify a declined run as a
pass and a failed comparison as a pass, which is the exact misclassification
#1112 fixed.

Fold case in the predicates instead of documenting the precondition:
`has_skip_marker` compares the line's leading bytes with
`eq_ignore_ascii_case`, and `has_mismatch_marker` lowercases once inside
itself. `run_binary` then passes the raw combined stream, so the precondition
is gone rather than moved -- and the one allocation moves from the call site
into the predicate that needs it. Behaviour on the existing path is unchanged:
the input was already lowercase there.

One test over the spellings that matter, in both directions: five mixed-case
skip declines and five mixed-case failure markers must match, and folding case
must not widen what counts as a marker -- a bare `pass`/`PASS`,
`pass: 1024 elements verified`, `SUCCESS`, a mid-line `no skipping: here`, and
`all checks passed` all stay negative. It fails with the folding reverted.

Verified: 15 tests pass, fmt clean, `clippy --all-targets -- -D warnings`
clean, `cargo doc --document-private-items` clean.

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>

* fix(ptx-schedule): classify failures before skips

Fold marker case at the predicates and classify timeout, process failure, and mismatch before accepting a graceful skip.

Signed-off-by: nihalp <nihalp@nvidia.com>

---------

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
Signed-off-by: nihalp <nihalp@nvidia.com>
Co-authored-by: nihalp <nihalp@nvidia.com>
nihalpasham pushed a commit to vyncint/cuda-oxide that referenced this pull request Aug 23, 2026
…ailure

`RunKind::Skipped` became reachable in NVIDIA#1112, when `has_skip_marker` learned
the second spelling examples use to decline. Nothing consumes it distinctly.
Both places that read the baseline test for `Pass` and treat everything else
alike:

    if !matches!(baseline.kind, RunKind::Pass) {
        ... println!("schedule-fuzz: BASELINE FAILURE: {:?}; ...")

    Ok(summary) if !matches!(summary.baseline.kind, RunKind::Pass) => {
        eprintln!("Error: baseline example did not pass; ...");
        std::process::exit(1);
    }

So `cargo oxide fuzz-schedule redux_f32` on anything below sm_100a prints
`BASELINE FAILURE: Skipped`, then `Error: baseline example did not pass`, and
exits 1. The example did not fail. It ran, declared it cannot run on this
device, and exited 0 -- which is precisely what `scripts/smoketest.sh` reports
as `PASS (skipped)`. NVIDIA#665 exists because reporting a decline as anything else
made an arch-gated example indistinguishable from a broken one, and this is the
same conflation in the campaign: sweeping a fleet, "below this example's arch
floor" comes back red alongside a genuinely wedged device.

Rather than repeat a `matches!` in two crates, the distinction becomes one
typed definition: `RunKind::baseline_verdict()` returns `Usable`, `Declined` or
`Broken`, and the campaign and the CLI both consult it. The match is exhaustive,
so a new `RunKind` cannot join a class by accident -- adding one now fails with
E0004 instead of silently reading as broken.

Behaviour: `Declined` prints `BASELINE DECLINED` and, in the CLI, a `Note:` on
stderr with exit **0**. Every other non-`Pass` baseline keeps `BASELINE
FAILURE` and exit 1, now with the kind in the message so the reader knows
whether it hung, crashed or wedged. `--fail-on-finding` is untouched and
remains the switch for failing on real findings. If you would rather a decline
carry a distinct nonzero code, that is a one-line change in the new arm.

Two tests: the three classes pinned exhaustively over all eight `RunKind`
variants, and that every *finding* kind is a `Broken` baseline -- a mismatch is
a finding in a variant, but as the baseline it means there is no ground truth
to compare against. The first fails with `Skipped` moved back into the broken
class.

Verified: 16 ptx-schedule tests and 177 cargo-oxide tests pass, fmt clean,
`clippy --all-targets -- -D warnings` clean on both crates, and
`cargo doc --document-private-items` clean.

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
nihalpasham added a commit that referenced this pull request Aug 23, 2026
…ailure (#1135)

* fix(ptx-schedule): an example that declines to run is reported as a failure

`RunKind::Skipped` became reachable in #1112, when `has_skip_marker` learned
the second spelling examples use to decline. Nothing consumes it distinctly.
Both places that read the baseline test for `Pass` and treat everything else
alike:

    if !matches!(baseline.kind, RunKind::Pass) {
        ... println!("schedule-fuzz: BASELINE FAILURE: {:?}; ...")

    Ok(summary) if !matches!(summary.baseline.kind, RunKind::Pass) => {
        eprintln!("Error: baseline example did not pass; ...");
        std::process::exit(1);
    }

So `cargo oxide fuzz-schedule redux_f32` on anything below sm_100a prints
`BASELINE FAILURE: Skipped`, then `Error: baseline example did not pass`, and
exits 1. The example did not fail. It ran, declared it cannot run on this
device, and exited 0 -- which is precisely what `scripts/smoketest.sh` reports
as `PASS (skipped)`. #665 exists because reporting a decline as anything else
made an arch-gated example indistinguishable from a broken one, and this is the
same conflation in the campaign: sweeping a fleet, "below this example's arch
floor" comes back red alongside a genuinely wedged device.

Rather than repeat a `matches!` in two crates, the distinction becomes one
typed definition: `RunKind::baseline_verdict()` returns `Usable`, `Declined` or
`Broken`, and the campaign and the CLI both consult it. The match is exhaustive,
so a new `RunKind` cannot join a class by accident -- adding one now fails with
E0004 instead of silently reading as broken.

Behaviour: `Declined` prints `BASELINE DECLINED` and, in the CLI, a `Note:` on
stderr with exit **0**. Every other non-`Pass` baseline keeps `BASELINE
FAILURE` and exit 1, now with the kind in the message so the reader knows
whether it hung, crashed or wedged. `--fail-on-finding` is untouched and
remains the switch for failing on real findings. If you would rather a decline
carry a distinct nonzero code, that is a one-line change in the new arm.

Two tests: the three classes pinned exhaustively over all eight `RunKind`
variants, and that every *finding* kind is a `Broken` baseline -- a mismatch is
a finding in a variant, but as the baseline it means there is no ground truth
to compare against. The first fails with `Skipped` moved back into the broken
class.

Verified: 16 ptx-schedule tests and 177 cargo-oxide tests pass, fmt clean,
`clippy --all-targets -- -D warnings` clean on both crates, and
`cargo doc --document-private-items` clean.

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>

* fix(ptx-schedule): pin baseline and CLI verdict policy

Keep stronger process failures ahead of skips and make the CLI distinction between declined, broken, and finding-bearing campaigns explicit and testable.

Signed-off-by: nihalp <nihalp@nvidia.com>

---------

Signed-off-by: Vyncint Ng <115854244+vyncint@users.noreply.github.com>
Signed-off-by: nihalp <nihalp@nvidia.com>
Co-authored-by: nihalp <nihalp@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working fuzzer Fuzzing infrastructure (rustlantis, difftest)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants