Skip to content

ci: run the Python tests on the release path (sable-cazq) - #222

Merged
Rome-1 merged 2 commits into
mainfrom
fix/sable-cazq-python-tests-gate-release
Sep 1, 2026
Merged

ci: run the Python tests on the release path (sable-cazq)#222
Rome-1 merged 2 commits into
mainfrom
fix/sable-cazq-python-tests-gate-release

Conversation

@Rome-1

@Rome-1 Rome-1 commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Closes sable-cazq.

The hole

python/tests/ runs in exactly one workflow in this repo — test-comprehensive.yml — and that workflow is not on the release path.

  • publish.yaml had no Python test job at all.
  • validate-release.yml's test-build ran python -m build and no pytest, so the pre-release gate asserted that the Python package compiles, never that it works.

A Python-only regression reached PyPI green. PyPI is not somewhere you can quietly unship from.

The part that was not in the bead

The Node release path looked safe because publish-node has a needs: clause. It is weaker than it looks. publish.yaml's test-node had no Python toolchain, and tests/cross-runtime-parity.test.ts gates its entire describe block on python3 -c "import typer" succeeding:

const describeIfPython = PYTHON_AVAILABLE ? describe : describe.skip;

describe.skip is silent. Verified by stubbing python3 to exit 1 and running the file:

Test Files  1 skipped (1)
     Tests  40 skipped (40)

Exit 0, suite green, zero parity assertions executed. The same job in test-comprehensive.yml installs the Python package first, so those 40 tests do run there — the release path was running a quietly weaker version of the job it depends on. These are the tests that enforce the dual-implementation contract, which makes them the ones a release least wants to skip.

Changes

File Change
publish.yaml New test-python job, mirroring test-comprehensive.yml's
publish.yaml publish-node and publish-python both needs: it
publish.yaml test-node gets setup-python + pip install -e ./python, so the 40 parity tests actually run
validate-release.yml test-build runs pytest after the wheel build

Why both publish jobs and not just publish-python: the two registries publish from one push and version parity is enforced. A suite that goes red after npm has already published leaves the two runtimes at different versions on the two indexes. The divergence is the failure mode whichever half breaks.

Cost. test-python runs in parallel with test-node, so it adds no wall clock the release was not already spending. The pytest addition to validate-release.yml's test-build is ~80s in a job that already spends ~4 minutes on pnpm test. CI slow enough that people route around it is worse than CI with holes; this stays well short of that.

Verification

  • Reproduced the claim before fixing it — read both workflows, confirmed no pytest on the release path.
  • pytest tests/ -q locally: 1481 passed, 1 skipped. The 7 failures on my box are a dangling ~/.local/bin/skill-scanner symlink making shutil.which report a binary that cannot execute; with that path removed, 20 passed, 7 skipped. Not reproducible in CI, where skill-scanner is simply absent and the skipif guard fires. Filed separately as sable-of2h.
  • cross-runtime-parity.test.ts with Python available: 39/40 pass. The one failure is local-only — the globally installed editable rafter-cli dist-info on this box points at another worktree at 0.9.0, so importlib.metadata.version() returns a stale string. CI does pip install -e . from the checkout immediately before running, so the metadata is fresh there. This is exactly the test the release path was skipping.
  • Both workflow files parse; job graph checked.

Relationship to #221

#221 adds needs: [test-node, test-package] to publish-python, which stops a red Node suite from shipping the Python package. It does not make the Python tests run. Different hole, adjacent lines. If both land, keep both sets of needs — the merged clause is [test-node, test-python, test-package], which is what this branch already has.

Not verified

GitHub Actions minutes and artifact storage are both exhausted on this account (sable-9nmu), so CI may not execute on this PR. The workflow changes are structural and were validated by parsing and by reproducing each skip locally, but they have not been observed running green on a runner.

Rome-1 and others added 2 commits August 31, 2026 19:25
python/tests/ executed in exactly one workflow in this repo —
test-comprehensive.yml — and that workflow is not on the release path.
publish.yaml had no Python test job at all, and validate-release.yml's
test-build ran `python -m build` without pytest, so the pre-release gate
asserted the Python package compiles and never that it works. A
Python-only regression reached PyPI green, and PyPI is not somewhere you
can quietly unship from.

- publish.yaml: add test-python, mirroring test-comprehensive.yml's job.
  It runs in parallel with test-node, so it costs no wall clock the
  release was not already spending.
- publish.yaml: gate BOTH publish jobs on it. The two registries publish
  from one push and version parity is enforced elsewhere, so a suite that
  goes red after npm has already published leaves the two runtimes at
  different versions on the two indexes. The divergence is the failure
  mode whichever half breaks.
- publish.yaml: give test-node a Python toolchain. The node release path
  looked safe because publish-node has a `needs:` clause, but the job it
  needed was quietly weaker than the same job in test-comprehensive.yml:
  tests/cross-runtime-parity.test.ts gates its whole describe block on
  `python3 -c "import typer"` and describe.skip is silent, so all 40
  parity assertions were skipped and the suite reported green. Verified
  by stubbing python3: "40 skipped", exit 0. Those tests are what enforce
  the dual-implementation contract — the ones a release least wants to
  skip.
- validate-release.yml: run pytest in test-build. ~80s.

Note for PR #221, which touches the same job: it adds
`needs: [test-node, test-package]` to publish-python to stop a red NODE
suite shipping the PyPI package. That is a different hole. Keep both
sets of needs if the two land together.
#221 added [test-node, test-package] to stop a red Node suite shipping the
PyPI package. #222 adds test-python so Python tests run on the release path
at all. Both are wanted; the union is [test-node, test-python, test-package].
@Rome-1
Rome-1 merged commit 0a2ae5f into main Sep 1, 2026
1 check passed
@Rome-1
Rome-1 deleted the fix/sable-cazq-python-tests-gate-release branch September 1, 2026 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants