From 67eeab50d2ab3ab0bcc29e12b836794c6f57f09d Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Mon, 17 Aug 2026 20:38:16 -0500 Subject: [PATCH 1/2] ci: bound and retry the apt step that turns a mirror hiccup into a blocked queue NO BACKLOG NUMBER IS CITED, DELIBERATELY. None has been allocated for this, and citing a number I have not allocated is the trap that arms itself the day someone legitimately issues it -- the citation would begin resolving, to unrelated work, with nothing reporting a problem. Naming the subject costs nothing and cannot arm. The Lander found this and explicitly left it unfiled; whoever files it should allocate then. WHAT IT FIXES, measured by the Lander on PR #427 on 2026-08-18 and relayed as content: the Qt system-library install runs `sudo apt-get update && sudo apt-get install -y ...` with NO retry and NO timeout on every ubuntu leg. THREE HANGS ACROSS TWO ATTEMPTS ON THREE DIFFERENT JOBS, the worst 27+ minutes and still hanging when reported. One of those jobs is `test (ubuntu-latest, py3.14)`, a REQUIRED context -- so an external apt mirror stalling blocks the merge queue outright. I MEASURED THE SURFACE MYSELF RATHER THAN TAKING THE COUNT: 8 unguarded apt steps across three workflow files, not the 3+1 reported. Five are this Qt step (ci.yml x3, quality-advisory.yml x2) and are guarded here. DELIBERATELY NOT TOUCHED, and this is scope rather than oversight: the three ODBC steps (ci.yml x2, benchmark.yml x1). They are a DIFFERENT SHAPE -- a larger block that also curls Microsoft's signing key and repo config before apt -- so the retry above is not transplantable without thinking about what re-running those curls means. They also sit on path-gated server-DB legs rather than on a required context, so they are not the measured blocker. They remain unguarded and someone should price them. TWO LEVERS, BECAUSE THEY ANSWER DIFFERENT FAILURES: per-command `timeout 120/180` kills a HANG so the retry can run at all step `timeout-minutes: 8` backstop -- this step can never eat a job budget again, even if the loop is later edited wrong A retry alone would still hang forever on attempt 1; a timeout alone would fail the leg on a blip that a second attempt clears. The Lander measured one job passing on attempt 2, which is what makes the retry worth having and also why it is not called a fix for the mirror. THE FAILURE TEXT NAMES THE CAUSE. After three attempts it prints, as a GitHub error annotation, that this is the ubuntu runner mirror and NOT the change under test. That is the same defect class as the margin gate: a check whose label points a reader at the wrong subject costs diagnostic time on every future occurrence, and the Lander had to broadcast "do not go hunting in your change" precisely because nothing in the failure said so. WHAT THIS DOES NOT CLAIM: any root cause beyond the step. The Lander did not establish one, did not measure other repos or other times of day, and neither did I. This bounds the blast radius of an external failure; it does not diagnose it. Verification, with its scope: both workflow files still parse as YAML (12 and 5 jobs); tests/test_workflow_shell_syntax.py 3 passed, which is the harness that runs every `run:` block through `bash -n` -- the new loop is real shell and is checked as such; tests/test_ci_step_margin.py + test_required_contexts.py + test_ci_docs_only_detector.py + test_ci_engine_step_excludes_webconsole.py, 68 passed, so the job-cap nesting invariant and the pinned required contexts are unchanged. actionlint runs in pre-commit over these files. Not run here: the full suite. NOT VERIFIED ON A RUNNER -- no local run can exercise a GitHub apt mirror, so the retry's real behaviour is unproven until a leg fires. Co-Authored-By: Claude Opus 5 --- .github/workflows/ci.yml | 60 +++++++++++++++++++++++--- .github/workflows/quality-advisory.yml | 40 +++++++++++++++-- 2 files changed, 90 insertions(+), 10 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 48934ec0..1750db2c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -230,9 +230,25 @@ jobs: # headless. Linux-only; Windows runners need no equivalent. - name: Install Qt offscreen system libraries if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux' + # BOUNDED AND RETRIED. Unguarded, this step turns an external apt-mirror hiccup into a + # blocked merge queue: measured 2026-08-18, three hangs across two attempts on three + # different jobs, one of them 27+ minutes on `test (ubuntu-latest, py3.14)` -- a REQUIRED + # context, so the queue stops. Two levers, because they answer different failures. The + # per-command `timeout` kills a HANG and lets the retry run; `timeout-minutes` is the + # backstop that stops this step ever eating a job budget again if the loop is edited wrong. + # The failure text names the cause, so the next reader is not sent hunting in their diff -- + # a check whose label points at the wrong subject is what BACKLOG #1254 is about. + timeout-minutes: 8 run: | - sudo apt-get update - sudo apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3 + for attempt in 1 2 3; do + if sudo timeout 120 apt-get update && sudo timeout 180 apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3; then + exit 0 + fi + echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying" + sleep $((attempt * 5)) + done + echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test." + exit 1 # uv is the installer for every leg (was pip): it resolves + installs the pyproject extras far # faster than pip, into the same setup-python interpreter via `uv pip install --system`. Pinned @@ -818,9 +834,25 @@ jobs: - name: Install Qt offscreen system libraries if: (needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch') && runner.os == 'Linux' + # BOUNDED AND RETRIED. Unguarded, this step turns an external apt-mirror hiccup into a + # blocked merge queue: measured 2026-08-18, three hangs across two attempts on three + # different jobs, one of them 27+ minutes on `test (ubuntu-latest, py3.14)` -- a REQUIRED + # context, so the queue stops. Two levers, because they answer different failures. The + # per-command `timeout` kills a HANG and lets the retry run; `timeout-minutes` is the + # backstop that stops this step ever eating a job budget again if the loop is edited wrong. + # The failure text names the cause, so the next reader is not sent hunting in their diff -- + # a check whose label points at the wrong subject is what BACKLOG #1254 is about. + timeout-minutes: 8 run: | - sudo apt-get update - sudo apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3 + for attempt in 1 2 3; do + if sudo timeout 120 apt-get update && sudo timeout 180 apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3; then + exit 0 + fi + echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying" + sleep $((attempt * 5)) + done + echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test." + exit 1 - name: Set up uv if: needs.changes.outputs.code == 'true' || github.event_name == 'push' || github.event_name == 'workflow_dispatch' @@ -963,9 +995,25 @@ jobs: # imports the whole suite regardless of what this job intends to execute. - name: Install Qt offscreen system libraries if: runner.os == 'Linux' + # BOUNDED AND RETRIED. Unguarded, this step turns an external apt-mirror hiccup into a + # blocked merge queue: measured 2026-08-18, three hangs across two attempts on three + # different jobs, one of them 27+ minutes on `test (ubuntu-latest, py3.14)` -- a REQUIRED + # context, so the queue stops. Two levers, because they answer different failures. The + # per-command `timeout` kills a HANG and lets the retry run; `timeout-minutes` is the + # backstop that stops this step ever eating a job budget again if the loop is edited wrong. + # The failure text names the cause, so the next reader is not sent hunting in their diff -- + # a check whose label points at the wrong subject is what BACKLOG #1254 is about. + timeout-minutes: 8 run: | - sudo apt-get update - sudo apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3 + for attempt in 1 2 3; do + if sudo timeout 120 apt-get update && sudo timeout 180 apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3; then + exit 0 + fi + echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying" + sleep $((attempt * 5)) + done + echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test." + exit 1 - name: Set up uv uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 diff --git a/.github/workflows/quality-advisory.yml b/.github/workflows/quality-advisory.yml index cd96c88b..acc66274 100644 --- a/.github/workflows/quality-advisory.yml +++ b/.github/workflows/quality-advisory.yml @@ -292,9 +292,25 @@ jobs: - name: Set up uv uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 - name: Qt headless system deps + # BOUNDED AND RETRIED. Unguarded, this step turns an external apt-mirror hiccup into a + # blocked merge queue: measured 2026-08-18, three hangs across two attempts on three + # different jobs, one of them 27+ minutes on `test (ubuntu-latest, py3.14)` -- a REQUIRED + # context, so the queue stops. Two levers, because they answer different failures. The + # per-command `timeout` kills a HANG and lets the retry run; `timeout-minutes` is the + # backstop that stops this step ever eating a job budget again if the loop is edited wrong. + # The failure text names the cause, so the next reader is not sent hunting in their diff -- + # a check whose label points at the wrong subject is what BACKLOG #1254 is about. + timeout-minutes: 8 run: | - sudo apt-get update - sudo apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3 + for attempt in 1 2 3; do + if sudo timeout 120 apt-get update && sudo timeout 180 apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3; then + exit 0 + fi + echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying" + sleep $((attempt * 5)) + done + echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test." + exit 1 - name: Install project + coverage tools run: | uv pip install --system --constraint constraints.lock -e ".[dev,harness,fhir,dicom,x12,xml,webauthn]" -e packaging/messagefoundry-webconsole @@ -436,9 +452,25 @@ jobs: - name: Set up uv uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0 - name: Qt headless system deps + # BOUNDED AND RETRIED. Unguarded, this step turns an external apt-mirror hiccup into a + # blocked merge queue: measured 2026-08-18, three hangs across two attempts on three + # different jobs, one of them 27+ minutes on `test (ubuntu-latest, py3.14)` -- a REQUIRED + # context, so the queue stops. Two levers, because they answer different failures. The + # per-command `timeout` kills a HANG and lets the retry run; `timeout-minutes` is the + # backstop that stops this step ever eating a job budget again if the loop is edited wrong. + # The failure text names the cause, so the next reader is not sent hunting in their diff -- + # a check whose label points at the wrong subject is what BACKLOG #1254 is about. + timeout-minutes: 8 run: | - sudo apt-get update - sudo apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3 + for attempt in 1 2 3; do + if sudo timeout 120 apt-get update && sudo timeout 180 apt-get install -y libegl1 libgl1 libxkbcommon0 libdbus-1-3; then + exit 0 + fi + echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying" + sleep $((attempt * 5)) + done + echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test." + exit 1 - name: Install project + mutmut run: | uv pip install --system --constraint constraints.lock -e ".[dev,harness,fhir,dicom,x12,xml,webauthn]" -e packaging/messagefoundry-webconsole From 7e8dab376179a9d689dab819542f65883168739d Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Mon, 17 Aug 2026 20:41:45 -0500 Subject: [PATCH 2/2] ci: guard the three apt steps whose jobs have NO job-level timeout at all THE PRIORITY INVERTED AFTER MY PREVIOUS COMMIT, AND THAT COMMIT GUARDED THE SAFE HALF. a177702d1 bounded the five Qt sites because those were the ones visibly blocking the queue. Re-measured here, independently, after the Lander re-measured it: ci.yml test / webconsole / tooling job timeout-minutes present -> a hang dies at the cap quality-advisory coverage / mutation job timeout-minutes present -> same ci.yml sqlserver-store NO job-level timeout -> GitHub default, 360 min ci.yml load-test-sqlserver NO job-level timeout -> 360 min benchmark.yml baseline-sqlserver NO job-level timeout -> 360 min ALL FIVE SITES I FIXED FIRST SIT IN BOUNDED JOBS. ALL THREE I DEFERRED SIT IN UNBOUNDED ONES. So the sites that hurt were the ones that could not hurt much, and the ones nobody was watching are where the same mirror hiccup burns a six-hour runner budget -- on path-gated legs where a stalled job is least likely to be noticed. THE GENERAL SHAPE, worth more than either commit: VISIBILITY AND SEVERITY WERE ANTICORRELATED HERE. The queue-blocking symptom pointed at the bounded half precisely BECAUSE it was bounded -- it failed fast enough to be seen. Scoping a fix to "the one that is hurting us" would have left the worse half in place and looked finished. WHAT THIS ADDS to those three steps: timeout-minutes: 10 bounds EVERY command in the step, including the two curls, which are equally unguarded network calls in a job with no cap of its own --max-time 60 on both curls, so the bound is not solely the step cap a 3-attempt retry around the apt pair ONLY. Re-running apt is idempotent; re-fetching the signing key and the repo list is not the part that hangs, so wrapping those in a retry would add churn without adding resilience. THE DEEPER FIX IS A JOB-LEVEL TIMEOUT ON THOSE THREE JOBS and it is deliberately NOT here. That would bound every step rather than the one that bit -- correct altitude -- but choosing the number needs a measurement of how long each job legitimately runs, and a guessed cap on a server-DB or benchmark leg kills real work. The step comments say so at each site rather than leaving the next reader to wonder whether the omission was considered. COMPLETENESS, re-derived rather than asserted: a walk over all three workflow files now reports 8 apt steps, 8 with both a step cap and a retry, 0 unguarded. That is the same instrument that found the original 8, so the before and after are comparable. Verification, with its scope: all three workflow files parse as YAML (12, 5 and 3 jobs); tests/test_workflow_shell_syntax.py + test_ci_step_margin.py + test_required_contexts.py, 43 passed, so every new `run:` block is real shell under `bash -n` and the pinned required contexts and job-cap nesting invariant are unchanged; actionlint green in pre-commit. NOT VERIFIED ON A RUNNER -- no local run can exercise a GitHub apt mirror or a 360-minute hang. No BACKLOG number cited: none is allocated, and citing an unallocated number arms a reference that begins resolving to unrelated work the day someone issues it. Co-Authored-By: Claude Opus 5 --- .github/workflows/benchmark.yml | 30 ++++++++++++++--- .github/workflows/ci.yml | 60 ++++++++++++++++++++++++++++----- 2 files changed, 78 insertions(+), 12 deletions(-) diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml index e04a09ee..04deef96 100644 --- a/.github/workflows/benchmark.yml +++ b/.github/workflows/benchmark.yml @@ -166,13 +166,35 @@ jobs: with: python-version: "3.14" - name: Install Microsoft ODBC Driver 18 + sqlcmd + # THIS IS THE DANGEROUS HALF OF THE apt CLASS, and it is the opposite of the one that was + # visibly hurting. The Qt sites sit in jobs carrying a job-level `timeout-minutes`, so a + # mirror hang there dies at the job cap (20-40 min, measured). THIS JOB HAS NO JOB-LEVEL + # TIMEOUT, so the same hang runs to GitHub's 360-minute default -- a whole runner-hour + # budget burned on a stalled mirror, on a leg nobody is watching because it is path-gated. + # + # The step cap bounds EVERY command here, including the two curls, which are equally + # unguarded network calls. The retry wraps only the apt pair: re-running that is idempotent, + # whereas re-fetching the signing key and repo list is not the part that hangs. + # + # THE DEEPER FIX IS A JOB-LEVEL `timeout-minutes` ON THIS JOB, which would bound every step + # rather than the one that bit. Deliberately NOT done here: choosing that number needs a + # measurement of how long this job legitimately runs, and a guessed cap on a server-DB leg + # would kill real work. Recorded as a direction, not applied as a guess. + timeout-minutes: 10 run: | - curl -fsSL https://packages.microsoft.com/keys/microsoft.asc \ + curl -fsSL --max-time 60 https://packages.microsoft.com/keys/microsoft.asc \ | sudo tee /etc/apt/trusted.gpg.d/microsoft.asc > /dev/null - curl -fsSL "https://packages.microsoft.com/config/ubuntu/$(. /etc/os-release; echo "$VERSION_ID")/prod.list" \ + curl -fsSL --max-time 60 "https://packages.microsoft.com/config/ubuntu/$(. /etc/os-release; echo "$VERSION_ID")/prod.list" \ | sudo tee /etc/apt/sources.list.d/mssql-release.list > /dev/null - sudo apt-get update - sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18 mssql-tools18 unixodbc-dev + for attempt in 1 2 3; do + if sudo timeout 120 apt-get update && sudo ACCEPT_EULA=Y timeout 240 apt-get install -y msodbcsql18 mssql-tools18 unixodbc-dev; then + exit 0 + fi + echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying" + sleep $((attempt * 5)) + done + echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test." + exit 1 - name: Wait for SQL Server and create the database (RCSI on) run: | sqlcmd=/opt/mssql-tools18/bin/sqlcmd diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1750db2c..10c45537 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1501,13 +1501,35 @@ jobs: python-version: "3.14" - name: Install Microsoft ODBC Driver 18 + sqlcmd + # THIS IS THE DANGEROUS HALF OF THE apt CLASS, and it is the opposite of the one that was + # visibly hurting. The Qt sites sit in jobs carrying a job-level `timeout-minutes`, so a + # mirror hang there dies at the job cap (20-40 min, measured). THIS JOB HAS NO JOB-LEVEL + # TIMEOUT, so the same hang runs to GitHub's 360-minute default -- a whole runner-hour + # budget burned on a stalled mirror, on a leg nobody is watching because it is path-gated. + # + # The step cap bounds EVERY command here, including the two curls, which are equally + # unguarded network calls. The retry wraps only the apt pair: re-running that is idempotent, + # whereas re-fetching the signing key and repo list is not the part that hangs. + # + # THE DEEPER FIX IS A JOB-LEVEL `timeout-minutes` ON THIS JOB, which would bound every step + # rather than the one that bit. Deliberately NOT done here: choosing that number needs a + # measurement of how long this job legitimately runs, and a guessed cap on a server-DB leg + # would kill real work. Recorded as a direction, not applied as a guess. + timeout-minutes: 10 run: | - curl -fsSL https://packages.microsoft.com/keys/microsoft.asc \ + curl -fsSL --max-time 60 https://packages.microsoft.com/keys/microsoft.asc \ | sudo tee /etc/apt/trusted.gpg.d/microsoft.asc > /dev/null - curl -fsSL "https://packages.microsoft.com/config/ubuntu/$(. /etc/os-release; echo "$VERSION_ID")/prod.list" \ + curl -fsSL --max-time 60 "https://packages.microsoft.com/config/ubuntu/$(. /etc/os-release; echo "$VERSION_ID")/prod.list" \ | sudo tee /etc/apt/sources.list.d/mssql-release.list > /dev/null - sudo apt-get update - sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18 mssql-tools18 unixodbc-dev + for attempt in 1 2 3; do + if sudo timeout 120 apt-get update && sudo ACCEPT_EULA=Y timeout 240 apt-get install -y msodbcsql18 mssql-tools18 unixodbc-dev; then + exit 0 + fi + echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying" + sleep $((attempt * 5)) + done + echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test." + exit 1 - name: Wait for SQL Server and create the database run: | @@ -2080,13 +2102,35 @@ jobs: python-version: "3.14" - name: Install Microsoft ODBC Driver 18 + sqlcmd + # THIS IS THE DANGEROUS HALF OF THE apt CLASS, and it is the opposite of the one that was + # visibly hurting. The Qt sites sit in jobs carrying a job-level `timeout-minutes`, so a + # mirror hang there dies at the job cap (20-40 min, measured). THIS JOB HAS NO JOB-LEVEL + # TIMEOUT, so the same hang runs to GitHub's 360-minute default -- a whole runner-hour + # budget burned on a stalled mirror, on a leg nobody is watching because it is path-gated. + # + # The step cap bounds EVERY command here, including the two curls, which are equally + # unguarded network calls. The retry wraps only the apt pair: re-running that is idempotent, + # whereas re-fetching the signing key and repo list is not the part that hangs. + # + # THE DEEPER FIX IS A JOB-LEVEL `timeout-minutes` ON THIS JOB, which would bound every step + # rather than the one that bit. Deliberately NOT done here: choosing that number needs a + # measurement of how long this job legitimately runs, and a guessed cap on a server-DB leg + # would kill real work. Recorded as a direction, not applied as a guess. + timeout-minutes: 10 run: | - curl -fsSL https://packages.microsoft.com/keys/microsoft.asc \ + curl -fsSL --max-time 60 https://packages.microsoft.com/keys/microsoft.asc \ | sudo tee /etc/apt/trusted.gpg.d/microsoft.asc > /dev/null - curl -fsSL "https://packages.microsoft.com/config/ubuntu/$(. /etc/os-release; echo "$VERSION_ID")/prod.list" \ + curl -fsSL --max-time 60 "https://packages.microsoft.com/config/ubuntu/$(. /etc/os-release; echo "$VERSION_ID")/prod.list" \ | sudo tee /etc/apt/sources.list.d/mssql-release.list > /dev/null - sudo apt-get update - sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18 mssql-tools18 unixodbc-dev + for attempt in 1 2 3; do + if sudo timeout 120 apt-get update && sudo ACCEPT_EULA=Y timeout 240 apt-get install -y msodbcsql18 mssql-tools18 unixodbc-dev; then + exit 0 + fi + echo "::warning::apt attempt ${attempt}/3 failed or timed out; retrying" + sleep $((attempt * 5)) + done + echo "::error::apt-get failed 3 times. This is the UBUNTU RUNNER MIRROR, not the change under test." + exit 1 - name: Wait for SQL Server and create the database (RCSI on, like the store suite) run: |