Skip to content

ci: watch the pins, and test the ones that must agree - #149

Merged
Jaro-c merged 3 commits into
mainfrom
ci/1526-pin-consistency-and-watch
Aug 25, 2026
Merged

Jaro-c merged 3 commits into
mainfrom
ci/1526-pin-consistency-and-watch

Conversation

@Jaro-c

@Jaro-c Jaro-c commented Aug 25, 2026

Copy link
Copy Markdown
Member

Closes the watcher half of Glyndor/podup#1526, and replaces a comment with a test.

Two changes, and the test matters more

tests/test_workflow_pins.py — four assertions that run on every pull request.

Three reusables declare a Rust toolchain, and rust-supply-chain sat on 1.97 while rust-ci and rust-audit were on 1.98: a bump raised two and not the third, and nothing compared them. The consequence was not cosmetic — an SBOM was produced by a compiler the test suite never ran, the defect Glyndor/podup#1487 recorded. That was fixed with a comment, and a comment does not fail.

test catches
the three toolchain defaults agree the divergence that actually happened
that value is MAJOR.MINOR rustup accepts stable, 1.98 and 1.98.0 and they mean different things — floating, a minor's stable release, and one pinned patch
SHELLCHECK_VERSION agrees across three workflows same failure mode
ACTIONLINT_VERSION agrees across two same

None hardcodes a version. They assert agreement, so they survive every bump and fail only on divergence.

Each was watched failing before being kept — the divergence introduced, the red observed, the file restored:

Rust toolchain defaults disagree: rust-ci.yml='1.98', rust-audit.yml='1.98',
  rust-supply-chain.yml='1.97'
Rust toolchain default is not a two-component version: rust-audit.yml='stable'
Shellcheck version disagrees: actionlint.yml='v0.10.0', shell-ci.yml='v0.11.0', ...
Actionlint version disagrees: actionlint.yml='v1.7.12', workflow-lint.yml='v1.7.11'

Each message names the file that drifted, not merely that one did.

The watcher

pin-watch.yml plus scripts/pin-watch.py, weekly on Tuesday at 06:00 so it does not stack on podup's Monday 05:00.

Every value is read from the workflow file, never hardcoded in the watcher. A watcher holding its own copy of a pin can disagree with reality, which is the bug it exists to prevent.

The part that decides whether this is read or muted. Four pins are held below upstream on purpose — ruff, bun, gosec, cargo-llvm-cov — each with its reason beside it. Reporting them every Tuesday would train the reader to close the issue unread. So a held pin is reported only when the pin stops matching its hold version, meaning somebody moved it and the note is now stale. The inversion is commented, so nobody "fixes" the watcher by deleting the list.

It also does not hide what it could not check: PowerShell Gallery returns Atom XML and is not wired, so PSScriptAnalyzer appears in a separate table marked as not queried rather than being counted as current.

Verified by running it

Against this repository with only gh stubbed — crates.io, PyPI, static.rust-lang.org and endoflife.date were real:

python            pin='3.13'     upstream='3.14'   DRIFT
rust              pin='1.98'     <match>
shellcheck        pin='v0.11.0'  <match>
bun               pin='1.3.14'   hold='1.3.14'     SILENT
cargo-llvm-cov    pin='0.8.7'    hold='0.8.7'      SILENT
gosec             pin='v2.27.1'  hold='v2.27.1'    SILENT
ruff              pin='0.12.0'   hold='0.12.0'     SILENT

One real drift, four held pins silent. Two more paths exercised: moving ruff to 0.14.0 against a hold of 0.12.0 reports STALE HOLD, and with the issue already open it declines to open a duplicate.

One correction worth recording: my first run reported actionlint, shellcheck and govulncheck as drifted. That was my stub returning raw JSON while ignoring --jq, not the script. The comparison was v1.7.12 against [{"tagName":"v1.7.12"}].

Not registered with schedule-freshness

The guard needs a successful run with event=schedule; crons fire only from the default branch and workflow_dispatch does not count, so an entry added in the pull request that adds the workflow fails on day one. It gets one after the first scheduled run — the same ordering podup hit with its own watcher.

Jaro-c added 3 commits August 25, 2026 01:38
Three reusables declare a Rust `toolchain`, and rust-supply-chain sat on 1.97
while rust-ci and rust-audit were on 1.98 — a bump raised two of them and not
the third, and nothing compared them. The consequence was not cosmetic: an
SBOM was produced by a compiler the test suite never ran, which is the defect
Glyndor/podup#1487 recorded.

That was fixed with a comment. A comment does not fail, so this replaces it
with four tests that do, on every pull request rather than weekly:

- the three Rust `toolchain` defaults agree
- that shared value is MAJOR.MINOR, not `stable` and not MAJOR.MINOR.PATCH.
  `rustup toolchain install` accepts all three and they mean different things:
  `stable` floats, `1.98.0` pins one patch, `1.98` pins the stable release on
  that minor
- SHELLCHECK_VERSION agrees across the three workflows that install it
- ACTIONLINT_VERSION agrees across the two that install it

None of them hardcodes a version. They assert agreement, so they keep passing
across bumps and fail only on divergence — which is the failure mode that
actually happened.

Each was watched failing before it was kept: the divergence was introduced,
the test went red, the file was restored. The messages name the file that
drifted rather than only reporting that one did.

Refs Glyndor/podup#1526

Signed-off-by: Jaro-c <75870284+Jaro-c@users.noreply.github.com>
Consumers each got a watcher; this repository, which owns the pins, had none.
A hand survey on 2026-08-24 found eight of nineteen had drifted, one of them a
real defect — and a hand survey is what podup#1526 says is not good enough.

Every value is read from the workflow file rather than hardcoded here. A
watcher carrying its own copy of the pin can disagree with reality, which is
the bug it exists to prevent.

The part that decides whether this gets read or muted: four pins are held
below upstream on purpose — ruff, bun, gosec and cargo-llvm-cov, each for a
reason recorded next to it. Reporting them every Tuesday would train the
reader to close the issue unread. A held pin is reported only when the pin
stops matching its hold version, meaning somebody moved it and the note is now
stale. That inversion is commented so nobody "fixes" the watcher by deleting
the list.

It also does not hide what it could not check. PowerShell Gallery returns Atom
XML and is not wired, so PSScriptAnalyzer appears in a separate table marked
as not queried rather than being silently counted as current.

Verified by running it against this repository with only `gh` stubbed, so
crates.io, PyPI, static.rust-lang.org and endoflife.date were real: one drift
reported (python 3.13 against 3.14), the four held pins silent, and the
already-open case declining to open a duplicate. Moving a held pin was
exercised too — ruff at 0.14.0 against a hold of 0.12.0 reports STALE HOLD.

Not registered with schedule-freshness. That guard needs a successful run with
event=schedule, crons fire only from the default branch, and
workflow_dispatch does not count, so an entry added here would fail on day
one. It gets one after the first scheduled run.

Refs Glyndor/podup#1526

Signed-off-by: Jaro-c <75870284+Jaro-c@users.noreply.github.com>
Two failures on the previous push, both legitimate.

`tests / Lint & test` died with `ModuleNotFoundError: No module named 'yaml'`.
python-ci installs ruff and pytest and nothing else, and the new pin-agreement
tests parse the reusables as YAML. It passed locally because my container had
PyYAML and the runner does not — the harness carried a dependency the real job
never had.

python-ci already installs `requirements.txt` when the file exists, so that is
the fix rather than widening the reusable for every consumer. The file pins
pyyaml, which makes it a hand-typed pin like the nineteen in the workflows —
so pin-watch reads it too. A watcher that skips the pin its own repository
just added is the failure it exists to prevent. Reading it needed a new source
kind and a path that resolves at the repository root rather than under
.github/workflows.

`pin-policy` reported that this repository's own `python-ci` pin points at a
surface that differs from v1.21.0 — true, because v1.21.0 is where python-ci
changed. The three self-pins move to v1.21.0, which also means the test job
now runs the pytest and ruff versions that release ships.

Refs Glyndor/podup#1526

Signed-off-by: Jaro-c <75870284+Jaro-c@users.noreply.github.com>
@Jaro-c
Jaro-c merged commit fd9b469 into main Aug 25, 2026
9 checks passed
@Jaro-c
Jaro-c deleted the ci/1526-pin-consistency-and-watch branch August 25, 2026 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant