Fix/staleness gate honest verdicts - #551
Merged
Merged
Conversation
MEASURED 2026-07-21: 294 of 350 consumers were red on the single pin d7c2271 (2026-06-26) — every one of them green a fortnight earlier, with nothing changed in any consumer. aspasia is the clean natural experiment: same pin, 2026-07-09 run at 32 commits / 12d = passing notice; 2026-07-21 at 63 commits / 24d = hard error. standards moves ~2.6 commits/day, so a consumer exhausts both window budgets ~14 days after any propagation. Holding the fleet green under that rule means re-pinning ~300 repos every fortnight (order 7,800 PRs/year). A gate that fails everyone on a timer is not a guard: it trains the estate to ignore it, and it buries the findings that matter — the same sweep shows 148 real Workflow security linter failures and 74 anti-pattern failures that read as noise once the fleet is uniformly red. 1. Age outside the window is now a ::notice, not an error. The window is still computed and still reported, so propagate-workflow-pins.sh and the Hypatia sha_bump_propagation rule keep their signal. 2. A named deny-list (KNOWN_BAD_BEFORE) replaces age as the hard failure. Each entry is <reusable>:<fix-sha>; a pin that is a strict ancestor of the fix carries that defect and is rejected at any age. This is what the window was only ever a proxy for, and it is strictly better in both directions: a RECENT pin carrying the defect is now caught, and an old pin carrying none is no longer punished by the calendar. First entry is e9c8888 (#441). Before it, hypatia-scan-reusable and the validate-hypatia-baseline job cached the built Hypatia scanner under a keyless key while the build steps were guarded by `if [ ! -d ]`, so the first scanner build ever cached was reused forever and scanner fixes never took effect. A pin older than this reports a FALSE GREEN. All 294 consumers on d7c2271 predate it by one day and stay red — now for a true and actionable reason. 3. Integrity no longer rests on the runner's clone. The UNKNOWN / "may be forged" verdict was firing on legitimate pins: awesome-haskell pins governance-reusable@5a93d9d5 and was accused on four consecutive runs over 17 days, while that commit verifies as a true ancestor of main locally in both treeless and --depth 200 clones and via the compare API (behind=0, ahead=90). The mechanism was never reproduced off-runner. A hard FORGED verdict now requires either a COMPLETE local clone (not shallow, not partial) or confirmation from GET /repos/{nwo}/compare/{pin}...{branch}. Where neither is available the gate warns and passes: "cannot verify" is not "compromised". Zero API calls on the happy path — the server is consulted only when about to accuse. The deny-list gets the same fallback, so a degraded runner cannot silently skip it; "could not check" is reported as SKIPPED, never as passed. Tests: the hermetic fixture suite goes 12 -> 16 cases. Out-of-window now asserts exit 0 AND the advisory notice (a new run_case_out helper — an exit code alone cannot distinguish "passed silently" from "passed with the notice the propagation path depends on"). New coverage: deny-list rejects at any age, rejects even when in-window, does not reject the fixing commit itself, and is scoped per reusable. 16/16 pass, and the suite stays hermetic because a complete fixture clone resolves the forged case without the network. Also verified against live standards history and in degraded-clone mode; both give identical verdicts for fresh / denied / old-but-clean / forged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…low probe Follow-up to the previous commit, closing two holes found by adding a test for the path production actually takes. 1. The deny-list's NEGATIVE was still trusting local `merge-base` on a partial clone — which is ALWAYS the case in CI (`--filter=tree:0`) and is the one operation measured unreliable there. A treeless clone resolves every commit, so the API fallback was gated on "commits do not resolve" and was therefore dead code in production. Trace for awesome-haskell (pin 5a93d9d, which predates the Hypatia fix and must be denied): local branch taken -> a wrong negative -> `continue` -> not denied -> classify_pin also mis-fires -> API says ahead=90 -> OUT_OF_WINDOW -> notice -> PASS. That is the exact false green this list exists to prevent, on the very cohort that proved the bug. Now: a local POSITIVE is still trusted (no false positives observed, and a deny must not require the network), but a local NEGATIVE is accepted only from a COMPLETE clone; otherwise it is confirmed server-side. 2. `clone_is_complete` used `rev-parse --git-dir`, which returns a path relative to the CWD — and the CWD here is the CONSUMER's checkout, which actions/checkout makes shallow by default. So `.git/shallow` would almost always exist and every standards clone would be called incomplete. Uses `--absolute-git-dir` now. Caught by test 13c, which failed for exactly this reason when run from a shallow clone. 3. governance-reusable.yml passes GITHUB_TOKEN to the staleness step. The server-side confirmation is unauthenticated (60/hr per runner IP) and exhausting it degrades the gate to ::warning. That fail-open is deliberate and loud, and the common path never calls the API (a genuine deny is a local positive) — but a token removes the cliff. It reaches consumers only as they re-pin past this commit, which is precisely the propagation the deny-listed pins need anyway. Tests: 16 -> 19 cases, all passing, still hermetic (the API is pointed at a closed port). The new cases cover the production path that cases 1-12 could not reach, because they all run against a complete `git init` fixture: * partial clone: local positive still denies without the network * partial clone: unverifiable deny-list check is reported SKIPPED, not passed * complete clone: local negative is trusted silently (no SKIPPED warning) run_case_out gained a leading-'!' inversion so a test can assert the gate did NOT say something. Suite verified from two different working directories, since the bug in (2) was cwd-dependent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #
Type of change
How has this been verified?
Checklist
git commit -S).SPDX-License-Identifier(code/configMPL-2.0,prose
CC-BY-SA-4.0); I did not relicense existing files.Notes for reviewers
Summary by Gitar
cicheckscwd-relative shallow probe in CI scriptsThis will update automatically on new commits.