ci: stop three metadata-only jobs from cloning every blob - #6756
Merged
Conversation
`detect CodeQL scope` failed at 5m05s against a 5-minute budget, which is the same shape #6739 fixed for `check-trailer`: clone all history, then read only `git diff --name-only`. The blob download is the whole job. When it overruns, the job reports `cancelled` -- indistinguishable from a supersede -- and the dependent `codeql-pr` fails four seconds later having done nothing. #6739 fixed that job in the shared workflow. It did not reach this repo's own workflows, and none of aletheia's thirty checkouts carried the filter. Two qualify: the CodeQL scope detector reads path names, and the attribution scanner reads commit messages through `git log --format`. Neither opens a file. Deliberately not changed: codeql-pr's second checkout feeds CodeQL's analyzer, which reads sources, and gate-attestation both reads trailers and builds, so its three checkouts need judging individually rather than as a class.
Caught it failing live on #6749: cancelled at 5m04s against a 5-minute budget, and the gate's own message says so plainly -- "attestation-verify did not run to a verdict (result: cancelled). This is NOT a check failure: re-run the job." The job reads a commit message, one kanon.toml, and a tree hash. It never builds. Its 5-minute budget was being spent entirely on blobs it does not open. I had deferred all of gate-attestation.yml from this PR because that workflow both reads trailers and builds. That was too coarse: the file has three checkouts and they are not alike. This one is metadata-only. The other two stay untouched -- gate-coverage-scripts runs cargo metadata and scripts over the tree.
forkwright
changed the title
|
forkwright
added a commit
that referenced
this pull request
Aug 14, 2026
…imeouts (#6767) Closes #6757 ## The result is mostly a negative, and that is the finding All **32** `actions/checkout` steps across 20 workflows were enumerated and classified by *what the job actually reads*. The 27 not already fixed were re-derived independently rather than taken from #6756's summary. **There is almost nothing left to filter.** `filter: blob:none` only does anything alongside `fetch-depth: 0`. Most steps here never set it, so a filter would be a **no-op** — including several that looked like obvious candidates (`yaml-validate`, `docs-lint`, `llm-freshness`, `pii-scan`, both `security.yml` cargo jobs). Of the steps that *do* set `fetch-depth: 0`, nearly all genuinely need the objects: | job | why the full tree | |---|---| | `desktop`, `integration-server-smoke`, `online-tests` ×2, `release/test`, `gate-coverage-compile-checks` | they compile | | `secret-scan` (trufflehog), `gitleaks` | they scan **history content** by design — `security.yml`'s own `WHY(#6609)` says so | For those a blob-filtered clone is a **loss**, not a win — it fetches blobs lazily and they need all of them. So **#6756 already fixed the entire qualifying population**: `check-trailer`, `detect CodeQL scope`, `attestation-verify`. The class was three, not twenty-seven. ## What actually needed building The three were each found only by reading a job log *after something else looked broken*. That is the durable problem: an overrun reports **`cancelled`**, and GitHub uses that same status for a superseded run, so a real timeout reads as "an older run got replaced" and nobody investigates. `scripts/check-checkout-filter-risk.py` surfaces the signature — `fetch-depth: 0` + a short `timeout-minutes` + no `filter:` — so the next instance announces itself instead of waiting to be noticed sideways. ## Deliberately report-only, not gating One job carries that exact shape **on purpose**: `gate-coverage-scripts` reads nearly every tracked file at HEAD via `check-conflict-markers.py`, so `blob:none` buys it little. A gating check would nag a documented, reasoned exception until someone silenced it — and a silenced check is worse than none. So the script names the pattern and states plainly that not every flag wants a filter; a human still applies the same "what does this job read" judgement the triage did. It runs inside `yaml-validate`, which already walks the workflow directory. Against current `main` it reports exactly one flag, `gate-coverage-scripts` — the known exception, correctly identified. Co-authored-by: Cody Kickertz <admin@forkwright>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Finding
detect CodeQL scopefailed at 5m05s againsttimeout-minutes: 5on PR #6740. The dependentcodeql-prthen failed four seconds later, having done nothing.This is the same defect #6739 fixed for
check-trailer, in a workflow that fix could not reach.The job checks out with
fetch-depth: 0— every commit, every tree, every blob — and then does exactly this:Path names. It never opens a file. The blob download is the entire cost of the job.
Why it is worse than a slow job
When the clone overruns the budget, the job reports
cancelled, notfailed. GitHub uses that same status for a superseded run, so the two are indistinguishable in the checks list — a real timeout reads as "an older run got replaced." That is what made thecheck-trailerversion of this survive so long, and it is whycodeql-prlooks like a CodeQL problem when nothing about CodeQL is involved.Why #6739 did not cover it
#6739 bumped the pin to
forkwright/.github's fixed reusable workflow, which carries the filter forcheck-trailer. aletheia's own workflows were never touched, and an audit shows none of its 30actions/checkoutsteps across 20 workflows carriedfilter: blob:none.What changed
Two jobs qualify — both clone everything and read only git metadata:
codeql-pr.yml(detect CodeQL scope)git diff --name-onlyno-ai-attribution.ymlgit log --formatBoth get
filter: blob:none, keeping the commits and trees they do need. Each carries aWHY(filter)comment so a later reader does not "tidy" it away as a stray option.Deliberately not changed
The filter is not free for every job, so this is not a sweep:
codeql-pr.yml's second checkout feeds CodeQL's analyzer, which reads sources. It needs blobs.gate-attestation.ymlreads commit trailers and builds, across three separate checkouts. Those need judging one at a time, not as a class.Filed as #6757 so the remaining 28 get triaged deliberately rather than blanket-patched.
Verification
Both files parse as valid YAML (
yaml.safe_load). The real proof is the next run ofdetect CodeQL scope: it should drop from ~5 minutes to seconds, the waycheck-trailerwent from a 5m02s timeout-cancel to 7 seconds after the equivalent fix.Third instance, caught live
attestation-verifywas cancelled at 5m04s on PR #6749 while this PR was open. The gate's own evaluation says it outright:So a PR with nothing wrong with it reads as gate-failed, and the only remedy is knowing to re-run.
That job reads a commit message, one
kanon.toml, and a tree hash. It never builds. Its entire 5-minute budget was going to blobs it does not open.I had deferred all of
gate-attestation.ymlfrom this PR on the grounds that the workflow "reads trailers and builds". That was too coarse a judgment: the file has three checkouts and they are not alike.attestation-verifyis metadata-only and is now filtered. The other two are untouched —gate-coverage-scriptsrunscargo metadataand scripts across the tree, and genuinely wants blobs.Three demonstrated instances now —
check-trailer,detect CodeQL scope,attestation-verify— each found only by reading a job log after something else looked broken. That is the argument for #6757 doing the remaining triage systematically rather than waiting for each one to bite.