ci: fail when the suite shrinks or a file's tests all skip; name skipped jobs (sable-5d3q, sable-wl3p) - #226
Merged
Merged
Conversation
…ped jobs (sable-5d3q, sable-wl3p) A green run must have run something. sable-cazq was 40 parity tests describe.skip'd on a missing interpreter and the release path exited 0; a total-count floor would not have caught it, a per-file rule does. .github/scripts/test_floor.py reads the vitest JSON or pytest JUnit report and fails the job when executed tests fall below a floor (~95% of today's counts, lowered in the same PR that removes tests) or any file's tests all skipped, and lists every skipped test by name in the step summary. Wired into test-node and test-python on the PR path and the release path. The gate job's skip of the extended matrix on internal PRs now emits a ::warning:: and a summary section naming each job that did not run, so a grey check reads as skipped rather than passed.
…a hand-typed list (it omitted e2e-node within the hour)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two of the detection rules from sable-d2x2 (vacuous checks), as CI changes. Independent of #224/#225.
Rule C — a green run must have run something (sable-5d3q). sable-cazq was 40 parity tests
describe.skip'd on a missing interpreter, and the release path exited 0. A total-count floor would not have caught it (2112 − 40 is still a big number); a per-file rule does..github/scripts/test_floor.py(stdlib only) reads the vitest JSON report or the pytest JUnit XML and fails the job when:--allow-all-skipped(a visible exception). No file is in that state today, so the list is empty.Every skipped test is listed by name in the step summary whether or not the job fails, so a skip is never invisible. Wired into
test-nodeandtest-pythonon both the PR path (test-comprehensive.yml) and the release path (publish.yaml), which is where cazq's silent skip actually shipped. The runners' own output is unchanged (--reporter=defaultis kept alongside--reporter=json; pytest gains-o junit_family=xunit1 --junitxml, the family that recordsfileper test case).Rule B — a skipped job must say so, by name (sable-wl3p). The
gatejob already skipped the extended matrix on internal PRs with one summary sentence. A skipped job renders as a grey check and satisfies a required status check (sable-d3n6), so the skip now emits a::warning::annotation and a step-summary section that names each job that did not run, says that grey means skipped, and points atworkflow_dispatchfor the full matrix. The list is read from the workflow file itself (every job whoseif:is gated onneeds.gate.outputs.run), not typed by hand: the first hand-typed version of this very PR omittede2e-nodewithin the hour. Today that derivese2e-node, secret-detection-accuracy, sarif-validation, backend-api, package-integrity, cross-platform; an empty derivation fails the gate rather than printing an empty notice.Verified
Script exercised locally against real reports (a one-file vitest JSON and a one-file pytest XML) and crafted fixtures, exit codes checked:
pending(the cazq shape)--allow-all-skipped<skipped/>Step summary confirmed written (the fully-skipped file and all 40 test names appear). Both workflow files parse;
py_compileandruffclean on the script. Reporter flags confirmed on this repo's vitest 4.1 (--outputFile.json=) and pytest (fileattribute present under xunit1).The floors themselves are proved by this PR's own CI run:
test-nodeandtest-pythonmust pass the new step against the real suite.