Repository navigation
ci: watch the pins, and test the ones that must agree - #149
Merged
Merged
Conversation
Three reusables declare a Rust `toolchain`, and rust-supply-chain sat on 1.97 while rust-ci and rust-audit were on 1.98 — a bump raised two of them and not the third, and nothing compared them. The consequence was not cosmetic: an SBOM was produced by a compiler the test suite never ran, which is the defect Glyndor/podup#1487 recorded. That was fixed with a comment. A comment does not fail, so this replaces it with four tests that do, on every pull request rather than weekly: - the three Rust `toolchain` defaults agree - that shared value is MAJOR.MINOR, not `stable` and not MAJOR.MINOR.PATCH. `rustup toolchain install` accepts all three and they mean different things: `stable` floats, `1.98.0` pins one patch, `1.98` pins the stable release on that minor - SHELLCHECK_VERSION agrees across the three workflows that install it - ACTIONLINT_VERSION agrees across the two that install it None of them hardcodes a version. They assert agreement, so they keep passing across bumps and fail only on divergence — which is the failure mode that actually happened. Each was watched failing before it was kept: the divergence was introduced, the test went red, the file was restored. The messages name the file that drifted rather than only reporting that one did. Refs Glyndor/podup#1526 Signed-off-by: Jaro-c <75870284+Jaro-c@users.noreply.github.com>
Consumers each got a watcher; this repository, which owns the pins, had none. A hand survey on 2026-08-24 found eight of nineteen had drifted, one of them a real defect — and a hand survey is what podup#1526 says is not good enough. Every value is read from the workflow file rather than hardcoded here. A watcher carrying its own copy of the pin can disagree with reality, which is the bug it exists to prevent. The part that decides whether this gets read or muted: four pins are held below upstream on purpose — ruff, bun, gosec and cargo-llvm-cov, each for a reason recorded next to it. Reporting them every Tuesday would train the reader to close the issue unread. A held pin is reported only when the pin stops matching its hold version, meaning somebody moved it and the note is now stale. That inversion is commented so nobody "fixes" the watcher by deleting the list. It also does not hide what it could not check. PowerShell Gallery returns Atom XML and is not wired, so PSScriptAnalyzer appears in a separate table marked as not queried rather than being silently counted as current. Verified by running it against this repository with only `gh` stubbed, so crates.io, PyPI, static.rust-lang.org and endoflife.date were real: one drift reported (python 3.13 against 3.14), the four held pins silent, and the already-open case declining to open a duplicate. Moving a held pin was exercised too — ruff at 0.14.0 against a hold of 0.12.0 reports STALE HOLD. Not registered with schedule-freshness. That guard needs a successful run with event=schedule, crons fire only from the default branch, and workflow_dispatch does not count, so an entry added here would fail on day one. It gets one after the first scheduled run. Refs Glyndor/podup#1526 Signed-off-by: Jaro-c <75870284+Jaro-c@users.noreply.github.com>
Two failures on the previous push, both legitimate. `tests / Lint & test` died with `ModuleNotFoundError: No module named 'yaml'`. python-ci installs ruff and pytest and nothing else, and the new pin-agreement tests parse the reusables as YAML. It passed locally because my container had PyYAML and the runner does not — the harness carried a dependency the real job never had. python-ci already installs `requirements.txt` when the file exists, so that is the fix rather than widening the reusable for every consumer. The file pins pyyaml, which makes it a hand-typed pin like the nineteen in the workflows — so pin-watch reads it too. A watcher that skips the pin its own repository just added is the failure it exists to prevent. Reading it needed a new source kind and a path that resolves at the repository root rather than under .github/workflows. `pin-policy` reported that this repository's own `python-ci` pin points at a surface that differs from v1.21.0 — true, because v1.21.0 is where python-ci changed. The three self-pins move to v1.21.0, which also means the test job now runs the pytest and ruff versions that release ships. Refs Glyndor/podup#1526 Signed-off-by: Jaro-c <75870284+Jaro-c@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the watcher half of Glyndor/podup#1526, and replaces a comment with a test.
Two changes, and the test matters more
tests/test_workflow_pins.py— four assertions that run on every pull request.Three reusables declare a Rust
toolchain, andrust-supply-chainsat on 1.97 whilerust-ciandrust-auditwere on 1.98: a bump raised two and not the third, and nothing compared them. The consequence was not cosmetic — an SBOM was produced by a compiler the test suite never ran, the defectGlyndor/podup#1487recorded. That was fixed with a comment, and a comment does not fail.toolchaindefaults agreerustupacceptsstable,1.98and1.98.0and they mean different things — floating, a minor's stable release, and one pinned patchSHELLCHECK_VERSIONagrees across three workflowsACTIONLINT_VERSIONagrees across twoNone hardcodes a version. They assert agreement, so they survive every bump and fail only on divergence.
Each was watched failing before being kept — the divergence introduced, the red observed, the file restored:
Each message names the file that drifted, not merely that one did.
The watcher
pin-watch.ymlplusscripts/pin-watch.py, weekly on Tuesday at 06:00 so it does not stack on podup's Monday 05:00.Every value is read from the workflow file, never hardcoded in the watcher. A watcher holding its own copy of a pin can disagree with reality, which is the bug it exists to prevent.
The part that decides whether this is read or muted. Four pins are held below upstream on purpose —
ruff,bun,gosec,cargo-llvm-cov— each with its reason beside it. Reporting them every Tuesday would train the reader to close the issue unread. So a held pin is reported only when the pin stops matching its hold version, meaning somebody moved it and the note is now stale. The inversion is commented, so nobody "fixes" the watcher by deleting the list.It also does not hide what it could not check: PowerShell Gallery returns Atom XML and is not wired, so PSScriptAnalyzer appears in a separate table marked as not queried rather than being counted as current.
Verified by running it
Against this repository with only
ghstubbed — crates.io, PyPI, static.rust-lang.org and endoflife.date were real:One real drift, four held pins silent. Two more paths exercised: moving
ruffto 0.14.0 against a hold of 0.12.0 reportsSTALE HOLD, and with the issue already open it declines to open a duplicate.One correction worth recording: my first run reported actionlint, shellcheck and govulncheck as drifted. That was my stub returning raw JSON while ignoring
--jq, not the script. The comparison wasv1.7.12against[{"tagName":"v1.7.12"}].Not registered with schedule-freshness
The guard needs a successful run with
event=schedule; crons fire only from the default branch andworkflow_dispatchdoes not count, so an entry added in the pull request that adds the workflow fails on day one. It gets one after the first scheduled run — the same ordering podup hit with its own watcher.