Repository navigation
test: cover pin-watch and tooling-audit - #151
Merged
Merged
Conversation
Both shipped without tests while pin-policy and verify-release-assets have
them, and neither runs on a pull request — they are weekly crons, so a
regression in either was invisible for up to seven days and silent when it
arrived. A broken watcher does not shout, it stops reporting.
Sixteen tests, none of which touches the network: verified by running them in
a container with `--network=none`. A test that needs crates.io to pass is one
that fails on a Tuesday for somebody else's reason.
What they cover is what I verified by hand while building the two scripts, now
permanent:
- reading a pin from each of the three source kinds, and returning None rather
than something falsy-but-wrong when it is absent
- `requirements.txt` pins are `==` only; a range is not a pin and reporting
drift against one is noise
- a held pin stays silent while the pin matches its hold, and is reported
stale when it stops. Both directions, because that suppression is one edit
away from suppressing everything and nothing turns red when it does
- the exit-code asymmetry in the auditor: an advisory must not fail the run, a
failure of the script itself must
- a `.crate` with no bundled lockfile is reported as not audited rather than
counted clean
- yanked crates land in their own section
- deduplication against an already-open issue
Each of the two expensive controls was watched failing: inverting the
advisory exit path produces "An advisory found by cargo audit must not fail
the run; got exit 1", and emptying HOLDS reddens the stale-hold test.
Recording a method failure with it, because it nearly produced the wrong
conclusion. My first two sabotage attempts were no-ops — `{} or {...}` returns
the second operand, so HOLDS stayed full, and a regex missed the exit path and
the fallback edited an unrelated `return 0`. The tests passed, which read as
"these tests are weak" when the truth was "the sabotage did nothing". Check
that the edit changed behaviour before concluding anything from a green run.
Refs Glyndor/apt#121, Glyndor/podup#1526
Signed-off-by: Jaro-c <75870284+Jaro-c@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pin-watch.pyandtooling-audit.pyshipped without tests, whilepin-policy.pyandverify-release-assets.pyhave them. Worse: neither runs on a pull request. They are weekly crons, so a regression in either was invisible for up to seven days — and silent when it arrived, because a broken watcher does not shout, it stops reporting.The rule this follows is being added to
standards/testingin Glyndor/ai-context#126.Sixteen tests, none of which touches the network
Verified rather than intended: run in a container with
--network=none, all sixteen pass. A test that needs crates.io to pass is one that fails on a Tuesday for somebody else's reason.They cover what I verified by hand while building the two scripts, now permanent:
Nonerather than something falsy-but-wrong when absentrequirements.txtpins are==only — a range is not a pin, and reporting drift against one is noise.cratewith no bundled lockfile is reported as not audited rather than counted cleanBoth expensive controls watched failing
A method failure worth recording, because it nearly produced the wrong conclusion
My first two sabotage attempts were no-ops.
HOLDS = {} or {...}returns the second operand, so HOLDS stayed full. And a regex missed the exit path, so the fallback edited an unrelatedreturn 0further up the file.Both tests passed. That reads as "these tests are weak" when the truth was "the sabotage did nothing" — and acting on the first reading would have meant rewriting working tests.
Check that the edit changed behaviour before concluding anything from a green run. The deletion test in
standards/testingsays to delete the control and watch it go red; it is worth adding that you must also confirm you deleted the control.Verification
ruff checkandruff format --checkclean. Sixteen tests pass with no network. The two sabotage runs above were reverted andgit statusconfirms only the two new test files are added.