Skip to content

test: cover pin-watch and tooling-audit - #151

Merged
Jaro-c merged 1 commit into
mainfrom
test/cover-pin-watch-and-tooling-audit
Aug 25, 2026
Merged

Jaro-c merged 1 commit into
mainfrom
test/cover-pin-watch-and-tooling-audit

Conversation

@Jaro-c

@Jaro-c Jaro-c commented Aug 25, 2026

Copy link
Copy Markdown
Member

pin-watch.py and tooling-audit.py shipped without tests, while pin-policy.py and verify-release-assets.py have them. Worse: neither runs on a pull request. They are weekly crons, so a regression in either was invisible for up to seven days — and silent when it arrived, because a broken watcher does not shout, it stops reporting.

The rule this follows is being added to standards/testing in Glyndor/ai-context#126.

Sixteen tests, none of which touches the network

Verified rather than intended: run in a container with --network=none, all sixteen pass. A test that needs crates.io to pass is one that fails on a Tuesday for somebody else's reason.

They cover what I verified by hand while building the two scripts, now permanent:

  • reading a pin from each of the three source kinds, and returning None rather than something falsy-but-wrong when absent
  • requirements.txt pins are == only — a range is not a pin, and reporting drift against one is noise
  • a held pin stays silent while the pin matches its hold, and is reported stale when it stops. Both directions, because that suppression is one edit from suppressing everything and nothing turns red when it does
  • the exit-code asymmetry: an advisory must not fail the run, a failure of the script itself must
  • a .crate with no bundled lockfile is reported as not audited rather than counted clean
  • yanked crates land in their own section
  • deduplication against an already-open issue

Both expensive controls watched failing

inverting the advisory exit path
  → An advisory found by cargo audit must not fail the run; got exit 1

emptying HOLDS
  → FAILED test_held_pin_reported_stale_when_pin_no_longer_matches_hold

A method failure worth recording, because it nearly produced the wrong conclusion

My first two sabotage attempts were no-ops. HOLDS = {} or {...} returns the second operand, so HOLDS stayed full. And a regex missed the exit path, so the fallback edited an unrelated return 0 further up the file.

Both tests passed. That reads as "these tests are weak" when the truth was "the sabotage did nothing" — and acting on the first reading would have meant rewriting working tests.

Check that the edit changed behaviour before concluding anything from a green run. The deletion test in standards/testing says to delete the control and watch it go red; it is worth adding that you must also confirm you deleted the control.

Verification

ruff check and ruff format --check clean. Sixteen tests pass with no network. The two sabotage runs above were reverted and git status confirms only the two new test files are added.

Both shipped without tests while pin-policy and verify-release-assets have
them, and neither runs on a pull request — they are weekly crons, so a
regression in either was invisible for up to seven days and silent when it
arrived. A broken watcher does not shout, it stops reporting.

Sixteen tests, none of which touches the network: verified by running them in
a container with `--network=none`. A test that needs crates.io to pass is one
that fails on a Tuesday for somebody else's reason.

What they cover is what I verified by hand while building the two scripts, now
permanent:

- reading a pin from each of the three source kinds, and returning None rather
  than something falsy-but-wrong when it is absent
- `requirements.txt` pins are `==` only; a range is not a pin and reporting
  drift against one is noise
- a held pin stays silent while the pin matches its hold, and is reported
  stale when it stops. Both directions, because that suppression is one edit
  away from suppressing everything and nothing turns red when it does
- the exit-code asymmetry in the auditor: an advisory must not fail the run, a
  failure of the script itself must
- a `.crate` with no bundled lockfile is reported as not audited rather than
  counted clean
- yanked crates land in their own section
- deduplication against an already-open issue

Each of the two expensive controls was watched failing: inverting the
advisory exit path produces "An advisory found by cargo audit must not fail
the run; got exit 1", and emptying HOLDS reddens the stale-hold test.

Recording a method failure with it, because it nearly produced the wrong
conclusion. My first two sabotage attempts were no-ops — `{} or {...}` returns
the second operand, so HOLDS stayed full, and a regex missed the exit path and
the fallback edited an unrelated `return 0`. The tests passed, which read as
"these tests are weak" when the truth was "the sabotage did nothing". Check
that the edit changed behaviour before concluding anything from a green run.

Refs Glyndor/apt#121, Glyndor/podup#1526

Signed-off-by: Jaro-c <75870284+Jaro-c@users.noreply.github.com>
@Jaro-c
Jaro-c merged commit 329aa31 into main Aug 25, 2026
9 checks passed
@Jaro-c
Jaro-c deleted the test/cover-pin-watch-and-tooling-audit branch August 25, 2026 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant