Repository navigation
test(release): model gh and curl failure exits in the release fakes - #1488
Merged
Merged
Conversation
Two release fakes modeled a failure by its output while the real tools report it through their exit status: - gh attestation verify 2.101.0 reports a missing attestation by exiting 1 with empty stdout and "Error: HTTP 404: Not Found (...)" on stderr, not by printing []. A fake gh with that captured reply now drives collect_attestations through _run_verifier, whose verifier-rejected exit check no test exercised. The [] case stays as the guard for a clean exit that carries an empty list. - curl exits 6, 7 or 28 on a transport failure after --write-out prints 000, so set -euo pipefail ends the PyPI reconciliation step before it can print "HTTP 000". The curl stub now exits with the code it is given, and the 000 case is a transport-failure test that expects the step to end with curl's code and its error line.
doublewhy
marked this pull request as ready for review
October 11, 2026 08:52
This was referenced Oct 11, 2026
Open
doublewhy
added a commit
that referenced
this pull request
Oct 12, 2026
…ter entry Each open entry in the divergence register now names its issue: #1525 to #1557 for the production changes, and #1521 for the smaller corrections. The entries that open pull requests now address move to "Addressed by pull requests" with their issues: GH-7 (#1491, #1490), NS-1 (#1476, #1475), NS-2 (#1480, #1479), MF-2 (#1483, #1482), CL-2 (#1474, #1473), ST-2 (#1472, #1559) and NS-7 and NS-8 (#1488, #1560 and #1561). ST-1 names #1558, GH-3 records that #1415 merged, and "What remains" names #1522 for the unverified and unexamined boundaries.
Closed
10 tasks done
Dev builds formal evidence live since #1949 and keeps no retained captures. No tool on dev reads this capture.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of #1344, the fixture-provenance audit: findings NS-7 and NS-8, which #1461 recorded before it closed without a merge. Two release-test fakes model a failure through the tool's output, but the real tools report it through their exit status. Both fakes now fail the way the tools do; only tests change. Dev already refuses the release in both cases; the tests now run the path production takes.
gh attestation verify2.101.0, run with the production argv against a file with no attestation, exited 1 with empty stdout and 224 bytes of stderr (Error: HTTP 404: Not Found …).run_verifier(tools/release_evidence_verifier.py) turns any non-zero exit intoverifier-rejectedbefore reading stdout, but no test reached it.000and exited 6, 7 or 28 (unresolvable host, refused connection, timeout), creating no output file. Underset -euo pipefailthat ends the "Reconcile the PyPI destination" step atstatus="$(curl …)", before theHTTP ${status}refusal. The old stub exited 0, so the000case expectedHTTP 000.#1461 closed without a merge, so dev has no record of NS-7 or NS-8 to update. #1344 keeps the audit open.
Requirement UIDs
Related Issues
Closes #1560
Closes #1561
ADR Impact
Changes
gh(shell builtins only) alone onPATH; it replays the captured reply byte for byte and the test drivescollect_attestationsthrough its default runner,run_verifier.test_empty_attestation_list_is_refusedstays, documented as covering a clean exit with[].curl_exitandcurl_error: on a non-zero exit it writes no body, printscurl: (N) <error>and exits N.000leaves the uncertain-destination parametrization; 403 and 500 stay.Done-when matrix. "Fails without" comes from temporary mutations, restored with
git checkout; dev's copies of the tests pass under both._run_verifieragainst a stub that exits 1 with an empty stdout, assertsverifier-rejected, and fails if the non-zero exit is ignored.test_issue_1226_attestation_verifier.py:test_missing_attestation_reply_is_refusedassert 'verifier-output-unparsable' == 'verifier-rejected'canonical / test-shard000case uses a stub that exits non-zero as curl does, and the test asserts the step's failure and the absence of a pending output instead of theHTTP 000message.test_release_workflows.py:_run_pypi_reconciliation,test_pypi_reconciliation_stops_at_a_transport_failure|| trueon the curl:assert 1 == 6,== 7,== 28canonical / integrationdocs/research/fixture-provenance/index.mdand its row ininventory.mdrecord the fix.gh pr view 1461: CLOSED, not mergeddocs/research/fixture-provenance/index.mdand its row ininventory.mdrecord the fix.Test Plan
On
e15ee28a,uv run --project implementations/python --frozen --all-extras python -m pyteston both modules: 32 passed (dev: 31); with-m integration, 39 passed (dev: 37). Evidence was collected by running the real gh and curl on macOS on 2026-10-11, and the real step in bash with real curl andhttps_proxyat a closed port: it exited 7, printed only curl's error line and wrote nothing toGITHUB_OUTPUT.Gates on
e15ee28a:nox -s verify-fast-feedback -- --base-rev origin/dev,nox -s lintandmake policy(15 stages) passed. No requirement UID, so requirement governance is skipped, as in CI.tools/check_specification_coverage.py --jsonprinted[]; the tests sit outside the evidence digest. No package ortools/code changes.CI on
e15ee28a: run 38124105948 passed (28 checks pass,deployskipped); CodeQL run 38124104196 passed; SonarCloud gate OK.Head
d4c1cbd7merges deve64ef303and removes the retained runtime-mapping capture thata9e89b64added. Dev builds that evidence live since #1949. The net diff against dev is the two test modules. Ond4c1cbd7, the 11 verifier and PyPI reconciliation cases pass locally.CI on
d4c1cbd7: run 42683813146 passed, withverify,canonical / integration, all test shards andsonar. CodeQL run 42683804507 passed. GitGuardian returned neutral: the PR commit range, with the dev merge, exceeds its size limit, so it did not scan.gitleakson the three PR commits and on the net diff against dev found no leaks.Ground Control Checks
Traceability
implementations/python/tests/test_issue_1226_attestation_verifier.py,implementations/python/tests/test_release_workflows.pytools/release_evidence.py(collect_attestations),tools/release_evidence_verifier.py(run_verifier), the PyPI reconciliation step ofrelease-please.yml. GOV-913 already lists both modules as TESTS.Checklist
testdoes not cut a release.Documentation
Verified unchanged: the T19 note in
docs/decisions/package-artifacts/operations.md. The audit record stays with #1344.