Skip to content

ci: fail when the suite shrinks or a file's tests all skip; name skipped jobs (sable-5d3q, sable-wl3p) - #226

Merged
Rome-1 merged 2 commits into
mainfrom
ci/sable-5d3q-test-floors
Sep 2, 2026
Merged

ci: fail when the suite shrinks or a file's tests all skip; name skipped jobs (sable-5d3q, sable-wl3p)#226
Rome-1 merged 2 commits into
mainfrom
ci/sable-5d3q-test-floors

Conversation

@Rome-1

@Rome-1 Rome-1 commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Two of the detection rules from sable-d2x2 (vacuous checks), as CI changes. Independent of #224/#225.

Rule C — a green run must have run something (sable-5d3q). sable-cazq was 40 parity tests describe.skip'd on a missing interpreter, and the release path exited 0. A total-count floor would not have caught it (2112 − 40 is still a big number); a per-file rule does. .github/scripts/test_floor.py (stdlib only) reads the vitest JSON report or the pytest JUnit XML and fails the job when:

  • fewer tests executed (passed + failed) than the floor — set at ~95% of the 2026-09-02 counts (Node 1990 of 2099, Python 1530 of 1614), to be lowered in the same PR that deliberately removes tests, so a shrink is a reviewed decision; or
  • any file has tests and every one of them was skipped, unless it is named in --allow-all-skipped (a visible exception). No file is in that state today, so the list is empty.

Every skipped test is listed by name in the step summary whether or not the job fails, so a skip is never invisible. Wired into test-node and test-python on both the PR path (test-comprehensive.yml) and the release path (publish.yaml), which is where cazq's silent skip actually shipped. The runners' own output is unchanged (--reporter=default is kept alongside --reporter=json; pytest gains -o junit_family=xunit1 --junitxml, the family that records file per test case).

Rule B — a skipped job must say so, by name (sable-wl3p). The gate job already skipped the extended matrix on internal PRs with one summary sentence. A skipped job renders as a grey check and satisfies a required status check (sable-d3n6), so the skip now emits a ::warning:: annotation and a step-summary section that names each job that did not run, says that grey means skipped, and points at workflow_dispatch for the full matrix. The list is read from the workflow file itself (every job whose if: is gated on needs.gate.outputs.run), not typed by hand: the first hand-typed version of this very PR omitted e2e-node within the hour. Today that derives e2e-node, secret-detection-accuracy, sarif-validation, backend-api, package-integrity, cross-platform; an empty derivation fails the gate rather than printing an empty notice.

Verified

Script exercised locally against real reports (a one-file vitest JSON and a one-file pytest XML) and crafted fixtures, exit codes checked:

Case Exit
Real one-file reports, floor 50 (both runtimes) 0
Floor above the executed count 1
A vitest file whose 40 tests are all pending (the cazq shape) 1
Same file named in --allow-all-skipped 0, and the summary says so
A pytest file whose tests all carry <skipped/> 1
A report listing no files 1

Step summary confirmed written (the fully-skipped file and all 40 test names appear). Both workflow files parse; py_compile and ruff clean on the script. Reporter flags confirmed on this repo's vitest 4.1 (--outputFile.json=) and pytest (file attribute present under xunit1).

The floors themselves are proved by this PR's own CI run: test-node and test-python must pass the new step against the real suite.

…ped jobs (sable-5d3q, sable-wl3p)

A green run must have run something. sable-cazq was 40 parity tests
describe.skip'd on a missing interpreter and the release path exited 0; a
total-count floor would not have caught it, a per-file rule does.
.github/scripts/test_floor.py reads the vitest JSON or pytest JUnit report
and fails the job when executed tests fall below a floor (~95% of today's
counts, lowered in the same PR that removes tests) or any file's tests all
skipped, and lists every skipped test by name in the step summary. Wired
into test-node and test-python on the PR path and the release path.

The gate job's skip of the extended matrix on internal PRs now emits a
::warning:: and a summary section naming each job that did not run, so a
grey check reads as skipped rather than passed.
…a hand-typed list (it omitted e2e-node within the hour)
@Rome-1
Rome-1 merged commit 3ac4b0e into main Sep 2, 2026
9 checks passed
@Rome-1
Rome-1 deleted the ci/sable-5d3q-test-floors branch September 2, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant