ci: bound and retry every apt-get on a CI critical path - #437
Conversation
…ocked queue NO BACKLOG NUMBER IS CITED, DELIBERATELY. None has been allocated for this, and citing a number I have not allocated is the trap that arms itself the day someone legitimately issues it -- the citation would begin resolving, to unrelated work, with nothing reporting a problem. Naming the subject costs nothing and cannot arm. The Lander found this and explicitly left it unfiled; whoever files it should allocate then. WHAT IT FIXES, measured by the Lander on PR #427 on 2026-08-18 and relayed as content: the Qt system-library install runs `sudo apt-get update && sudo apt-get install -y ...` with NO retry and NO timeout on every ubuntu leg. THREE HANGS ACROSS TWO ATTEMPTS ON THREE DIFFERENT JOBS, the worst 27+ minutes and still hanging when reported. One of those jobs is `test (ubuntu-latest, py3.14)`, a REQUIRED context -- so an external apt mirror stalling blocks the merge queue outright. I MEASURED THE SURFACE MYSELF RATHER THAN TAKING THE COUNT: 8 unguarded apt steps across three workflow files, not the 3+1 reported. Five are this Qt step (ci.yml x3, quality-advisory.yml x2) and are guarded here. DELIBERATELY NOT TOUCHED, and this is scope rather than oversight: the three ODBC steps (ci.yml x2, benchmark.yml x1). They are a DIFFERENT SHAPE -- a larger block that also curls Microsoft's signing key and repo config before apt -- so the retry above is not transplantable without thinking about what re-running those curls means. They also sit on path-gated server-DB legs rather than on a required context, so they are not the measured blocker. They remain unguarded and someone should price them. TWO LEVERS, BECAUSE THEY ANSWER DIFFERENT FAILURES: per-command `timeout 120/180` kills a HANG so the retry can run at all step `timeout-minutes: 8` backstop -- this step can never eat a job budget again, even if the loop is later edited wrong A retry alone would still hang forever on attempt 1; a timeout alone would fail the leg on a blip that a second attempt clears. The Lander measured one job passing on attempt 2, which is what makes the retry worth having and also why it is not called a fix for the mirror. THE FAILURE TEXT NAMES THE CAUSE. After three attempts it prints, as a GitHub error annotation, that this is the ubuntu runner mirror and NOT the change under test. That is the same defect class as the margin gate: a check whose label points a reader at the wrong subject costs diagnostic time on every future occurrence, and the Lander had to broadcast "do not go hunting in your change" precisely because nothing in the failure said so. WHAT THIS DOES NOT CLAIM: any root cause beyond the step. The Lander did not establish one, did not measure other repos or other times of day, and neither did I. This bounds the blast radius of an external failure; it does not diagnose it. Verification, with its scope: both workflow files still parse as YAML (12 and 5 jobs); tests/test_workflow_shell_syntax.py 3 passed, which is the harness that runs every `run:` block through `bash -n` -- the new loop is real shell and is checked as such; tests/test_ci_step_margin.py + test_required_contexts.py + test_ci_docs_only_detector.py + test_ci_engine_step_excludes_webconsole.py, 68 passed, so the job-cap nesting invariant and the pinned required contexts are unchanged. actionlint runs in pre-commit over these files. Not run here: the full suite. NOT VERIFIED ON A RUNNER -- no local run can exercise a GitHub apt mirror, so the retry's real behaviour is unproven until a leg fires. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… all THE PRIORITY INVERTED AFTER MY PREVIOUS COMMIT, AND THAT COMMIT GUARDED THE SAFE HALF. a177702 bounded the five Qt sites because those were the ones visibly blocking the queue. Re-measured here, independently, after the Lander re-measured it: ci.yml test / webconsole / tooling job timeout-minutes present -> a hang dies at the cap quality-advisory coverage / mutation job timeout-minutes present -> same ci.yml sqlserver-store NO job-level timeout -> GitHub default, 360 min ci.yml load-test-sqlserver NO job-level timeout -> 360 min benchmark.yml baseline-sqlserver NO job-level timeout -> 360 min ALL FIVE SITES I FIXED FIRST SIT IN BOUNDED JOBS. ALL THREE I DEFERRED SIT IN UNBOUNDED ONES. So the sites that hurt were the ones that could not hurt much, and the ones nobody was watching are where the same mirror hiccup burns a six-hour runner budget -- on path-gated legs where a stalled job is least likely to be noticed. THE GENERAL SHAPE, worth more than either commit: VISIBILITY AND SEVERITY WERE ANTICORRELATED HERE. The queue-blocking symptom pointed at the bounded half precisely BECAUSE it was bounded -- it failed fast enough to be seen. Scoping a fix to "the one that is hurting us" would have left the worse half in place and looked finished. WHAT THIS ADDS to those three steps: timeout-minutes: 10 bounds EVERY command in the step, including the two curls, which are equally unguarded network calls in a job with no cap of its own --max-time 60 on both curls, so the bound is not solely the step cap a 3-attempt retry around the apt pair ONLY. Re-running apt is idempotent; re-fetching the signing key and the repo list is not the part that hangs, so wrapping those in a retry would add churn without adding resilience. THE DEEPER FIX IS A JOB-LEVEL TIMEOUT ON THOSE THREE JOBS and it is deliberately NOT here. That would bound every step rather than the one that bit -- correct altitude -- but choosing the number needs a measurement of how long each job legitimately runs, and a guessed cap on a server-DB or benchmark leg kills real work. The step comments say so at each site rather than leaving the next reader to wonder whether the omission was considered. COMPLETENESS, re-derived rather than asserted: a walk over all three workflow files now reports 8 apt steps, 8 with both a step cap and a retry, 0 unguarded. That is the same instrument that found the original 8, so the before and after are comparable. Verification, with its scope: all three workflow files parse as YAML (12, 5 and 3 jobs); tests/test_workflow_shell_syntax.py + test_ci_step_margin.py + test_required_contexts.py, 43 passed, so every new `run:` block is real shell under `bash -n` and the pinned required contexts and job-cap nesting invariant are unchanged; actionlint green in pre-commit. NOT VERIFIED ON A RUNNER -- no local run can exercise a GitHub apt mirror or a 360-minute hang. No BACKLOG number cited: none is allocated, and citing an unallocated number arms a reference that begins resolving to unrelated work the day someone issues it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Do not merge without rebasing first: this would silently DELETE content from
|
Correction to my previous comment: I quoted the wrong side of the statMy earlier comment said 168 deletions. That number is what this branch ADDS, not what merging would lose. I read The corrected figure: merging would lose 22 line(s) that The finding itself stands and so does the remedy -- rebase before merging, do not clear BEHIND and merge on green -- because a clean merge still drops those 22 lines with no marker. But the magnitude was overstated and the number in my first comment should not be quoted. |
RETRACTION: my "would silently delete content" comments on this PR were wrongBoth of my earlier comments are withdrawn. The finding does not survive a discriminator, and the remedy I attached to it was justified by a defect that is not there. What I did wrongI counted removed lines in a merge result and reported them as lost content. Those are different questions. A line-for-line REPLACEMENT scores identically to a DELETION under a line count -- and this branch's changes are overwhelmingly rewrites. The discriminator, run on this PRFor every line Zero -- and this PR is where the error was caught. The The concrete case that broke my own claimI said this branch deletes Not deleted -- expanded, plus a new mechanism (ADR 0164, What still stands, and what does notDoes not stand: any claim that merging this drops content, and the "never update-branch-and-merge-on-green" remedy insofar as it rested on that. Still true, but unremarkable: this branch is BEHIND, and branch protection is Credit where due: a peer session caught this by verifying one file in #437 -- |
1 participant
An unguarded
apt-getsits on the critical path of every ubuntu leg, with no retry and no steptimeout. Tonight it converted an external mirror hiccup into a blocked merge queue: it killed the
same PR twice.
Measured on #427, three hangs across two attempts, on three different jobs, all on the same step:
diff-coverage (advisory)web console tests (ubuntu-latest)web console tests (ubuntu-latest)test (ubuntu-latest, py3.14)That last one is a required context, so the queue could not drain.
The priority is the opposite of what the symptom suggested
All five Qt-libs sites sit in jobs with a job-level timeout (ubuntu 43, windows 66, webconsole
20), so they die at a cap. The three ODBC sites --
sqlserver-store,load-test-sqlserver,benchmark.yml'sbaseline-sqlserver-- have no job-level timeout at all, so the same hiccupthere runs to the platform default.
Visibility and severity were anti-correlated, structurally: the bound is what makes a site loud,
so triaging by "what is blocking me" surfaces the survivable sites first. A fix scoped to the step
that was visibly hurting the queue would have hardened the safe half and left the dangerous half
alone.
What this changes
Two levers, because they answer different failures. A per-command timeout kills a hang so the retry
can run at all; a step-level cap is the backstop so the step can never eat a job budget again. After
three attempts it prints an annotation saying the cause is the runner mirror and not the change
under test -- which had to be broadcast by hand tonight because nothing in the failure said so.
Completeness, same instrument before and after: 8 apt steps across three workflow files, 8 now
carrying both a step cap and a retry, 0 unguarded.
Provenance and verification limits
Both commits are cherry-picked unchanged from
claude/builder-2bee12, which authored them; theyare split out here because that branch also carries ledger commits that are not ready. Content
equivalence verified by
git patch-id --stable: identical on both.Verified in this worktree: all three files parse as YAML (12 / 5 / 3 jobs), and the completeness
walk above.
NOT verified here, stated rather than implied: pytest,
actionlintand the shell-syntax gatecannot run in this worktree --
pydanticis absent from its venv andactionlintis not on PATH.The authoring session reports them green on identical content. CI on this PR is the authority.
NOT established by anyone: no hang has been reproduced on an ODBC leg, so that half is a claim
about the configuration, not an observed outage. The retry itself is unproven until a leg fires --
this PR's own CI run is its first real test.
Filed as BACKLOG #1286 by a third session, with three-way attribution.