From f7241bd31519d970bf2ce0e88969782ad6041013 Mon Sep 17 00:00:00 2001 From: Hugo Montenegro Date: Wed, 29 Jul 2026 17:49:39 +0100 Subject: [PATCH] fix(watchdog): don't treat a mid-run mirror as an emergency MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The watchdog alerted 21 seconds after a content push, 12 seconds before the mirror finished, and was green again on the next tick: 16:03:52 push f3008839 16:04:13 watchdog: "the mirror is behind mono ... Stuck for 5m" 16:04:25 mirror completes 16:43:38 green A mirror-behind state skipped the 45-minute grace period, on the reasoning that "a failed mirror will not fix itself". That reasoning was wrong. A mirror that is merely mid-run fixes itself in about 30 seconds, and the two states look identical from outside. So any push landing shortly before a */15 tick produced a false alarm — roughly one in twenty pushes, indefinitely. Skipping the grace period is now reserved for states that genuinely cannot recover: red CI, and a watchdog that cannot evaluate itself. A dead mirror still alerts; it waits out the window first, which is what the window is for. Verified the full decision table: mirror mid-flight, 5m -> SILENT (was: ALERT — the bug) mirror stuck, 60m -> ALERT prod behind, 5m -> SILENT prod behind, 60m -> ALERT prod behind, CI RED -> ALERT immediately watchdog degraded -> ALERT immediately healthy -> SILENT Also ran the real step under `bash -e` against live APIs: reports current, exits 0. --- .../workflows/content-pipeline-watchdog.yml | 21 +++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/.github/workflows/content-pipeline-watchdog.yml b/.github/workflows/content-pipeline-watchdog.yml index b3aad9a5e..7aa2be1d2 100644 --- a/.github/workflows/content-pipeline-watchdog.yml +++ b/.github/workflows/content-pipeline-watchdog.yml @@ -116,13 +116,20 @@ jobs: fi STALE="" - MIRROR_BROKEN=false + # URGENT = skip the grace period. Reserve it for states that cannot + # resolve on their own. A mirror that is merely mid-run is NOT one of + # them: it finishes in ~30s, and treating it as urgent turned every + # push landing shortly before a tick into a false alarm. That fired + # for real (16:04:13 run, 21s after a push, 12s before the mirror + # completed — "Stuck for 5m", and green again on the next tick). + # A genuinely dead mirror still alerts; it just waits out the grace, + # which is exactly what the grace period is for. + URGENT=false if [ "$DEGRADED" = true ]; then STALE="**the watchdog itself could not evaluate the pipeline** — check its permissions before assuming content is stuck" - MIRROR_BROKEN=true + URGENT=true elif [ "${EXPECTED:0:7}" != "${MIRRORED:0:7}" ]; then STALE="the mirror is behind mono (expected \`${EXPECTED:0:7}\`, published \`${MIRRORED:0:7}\`)" - MIRROR_BROKEN=true elif [ "$LIVE" != "$TIP" ]; then STALE="production is behind peanut-content (live \`${LIVE:0:7}\`, latest \`${TIP:0:7}\`)" fi @@ -139,8 +146,7 @@ jobs: --json number,headRefName \ --jq '[.[] | select((.headRefName | startswith("auto/update-content-")) or (.headRefName | startswith("content/publish-to-main")))] | first | .number // empty' || true) - # A failed mirror will not fix itself, so skip the grace period. - DOOMED=$MIRROR_BROKEN + DOOMED=$URGENT if [ -n "$NUM" ]; then PR_SHA=$(GH_TOKEN=$UI_TOKEN gh api "repos/$REPO/pulls/$NUM" --jq '.head.sha' || true) CI=$(GH_TOKEN=$UI_TOKEN gh api "repos/$REPO/commits/$PR_SHA/check-runs?per_page=100" \ @@ -151,7 +157,10 @@ jobs: WHERE="no publish PR is open — the mirror, the dispatch, or update-content.yml failed" fi - # Red CI never self-resolves, so don't wait out the grace period. + # Only genuinely unrecoverable states skip the grace period: red CI, + # and a watchdog that cannot evaluate itself. Everything else gets + # the full window, because most staleness at this point is just the + # pipeline mid-flight. if [ "$DOOMED" = false ] && [ "$AGE" -lt "$GRACE_MINUTES" ]; then echo "$STALE — ${AGE}m old, within ${GRACE_MINUTES}m grace ($WHERE). Not alerting yet." exit 0