ci: run the Python tests on the release path (sable-cazq) - #222
Merged
Conversation
python/tests/ executed in exactly one workflow in this repo — test-comprehensive.yml — and that workflow is not on the release path. publish.yaml had no Python test job at all, and validate-release.yml's test-build ran `python -m build` without pytest, so the pre-release gate asserted the Python package compiles and never that it works. A Python-only regression reached PyPI green, and PyPI is not somewhere you can quietly unship from. - publish.yaml: add test-python, mirroring test-comprehensive.yml's job. It runs in parallel with test-node, so it costs no wall clock the release was not already spending. - publish.yaml: gate BOTH publish jobs on it. The two registries publish from one push and version parity is enforced elsewhere, so a suite that goes red after npm has already published leaves the two runtimes at different versions on the two indexes. The divergence is the failure mode whichever half breaks. - publish.yaml: give test-node a Python toolchain. The node release path looked safe because publish-node has a `needs:` clause, but the job it needed was quietly weaker than the same job in test-comprehensive.yml: tests/cross-runtime-parity.test.ts gates its whole describe block on `python3 -c "import typer"` and describe.skip is silent, so all 40 parity assertions were skipped and the suite reported green. Verified by stubbing python3: "40 skipped", exit 0. Those tests are what enforce the dual-implementation contract — the ones a release least wants to skip. - validate-release.yml: run pytest in test-build. ~80s. Note for PR #221, which touches the same job: it adds `needs: [test-node, test-package]` to publish-python to stop a red NODE suite shipping the PyPI package. That is a different hole. Keep both sets of needs if the two land together.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes sable-cazq.
The hole
python/tests/runs in exactly one workflow in this repo —test-comprehensive.yml— and that workflow is not on the release path.publish.yamlhad no Python test job at all.validate-release.yml'stest-buildranpython -m buildand no pytest, so the pre-release gate asserted that the Python package compiles, never that it works.A Python-only regression reached PyPI green. PyPI is not somewhere you can quietly unship from.
The part that was not in the bead
The Node release path looked safe because
publish-nodehas aneeds:clause. It is weaker than it looks.publish.yaml'stest-nodehad no Python toolchain, andtests/cross-runtime-parity.test.tsgates its entire describe block onpython3 -c "import typer"succeeding:describe.skipis silent. Verified by stubbingpython3toexit 1and running the file:Exit 0, suite green, zero parity assertions executed. The same job in
test-comprehensive.ymlinstalls the Python package first, so those 40 tests do run there — the release path was running a quietly weaker version of the job it depends on. These are the tests that enforce the dual-implementation contract, which makes them the ones a release least wants to skip.Changes
publish.yamltest-pythonjob, mirroringtest-comprehensive.yml'spublish.yamlpublish-nodeandpublish-pythonbothneeds:itpublish.yamltest-nodegetssetup-python+pip install -e ./python, so the 40 parity tests actually runvalidate-release.ymltest-buildruns pytest after the wheel buildWhy both publish jobs and not just
publish-python: the two registries publish from one push and version parity is enforced. A suite that goes red after npm has already published leaves the two runtimes at different versions on the two indexes. The divergence is the failure mode whichever half breaks.Cost.
test-pythonruns in parallel withtest-node, so it adds no wall clock the release was not already spending. The pytest addition tovalidate-release.yml'stest-buildis ~80s in a job that already spends ~4 minutes onpnpm test. CI slow enough that people route around it is worse than CI with holes; this stays well short of that.Verification
pytest tests/ -qlocally: 1481 passed, 1 skipped. The 7 failures on my box are a dangling~/.local/bin/skill-scannersymlink makingshutil.whichreport a binary that cannot execute; with that path removed,20 passed, 7 skipped. Not reproducible in CI, whereskill-scanneris simply absent and theskipifguard fires. Filed separately as sable-of2h.cross-runtime-parity.test.tswith Python available: 39/40 pass. The one failure is local-only — the globally installed editablerafter-clidist-info on this box points at another worktree at 0.9.0, soimportlib.metadata.version()returns a stale string. CI doespip install -e .from the checkout immediately before running, so the metadata is fresh there. This is exactly the test the release path was skipping.Relationship to #221
#221 adds
needs: [test-node, test-package]topublish-python, which stops a red Node suite from shipping the Python package. It does not make the Python tests run. Different hole, adjacent lines. If both land, keep both sets of needs — the merged clause is[test-node, test-python, test-package], which is what this branch already has.Not verified
GitHub Actions minutes and artifact storage are both exhausted on this account (sable-9nmu), so CI may not execute on this PR. The workflow changes are structural and were validated by parsing and by reproducing each skip locally, but they have not been observed running green on a runner.