From dad34d272f7afe3e0c61e6da5943a73e0b20e7a4 Mon Sep 17 00:00:00 2001 From: Hugo Montenegro Date: Tue, 28 Jul 2026 11:59:08 +0100 Subject: [PATCH] fix(watchdog): stop the script dying silently under bash -e MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third fix, same root cause each time: I verified in an environment that was not the one it runs in. GitHub runs the step as `bash -e {0}`. Combined with `set -o pipefail` that means any non-zero anywhere kills the script at that line with NO output — a red run and an empty log, which is the least debuggable possible failure. It hit twice: the Actions-API 403, then `grep` returning 1 because a commit touched no mirrored paths. grep exiting 1 on no-match is a normal answer, not an error, and it was the very first loop iteration — so the last run produced literally nothing between the env block and `exit 1`. `set +e` explicitly. Every branch here decides for itself and reports why; `-e` adds nothing but the ability to die mid-check without saying so. Also filter LIVE/TIP/EXPECTED through a 40-hex check. On failure `gh` prints its error body to stdout, which was landing in those vars and being compared as if it were a sha; now a broken call resolves to empty and trips the labelled DEGRADED branch instead. Verified by extracting the step and running it under `bash -e` against the real APIs, both paths: healthy -> "production content is current", exit 0, silent; broken token -> one WATCHDOG DEGRADED line, exit 1. That is the check I should have been running from the first commit. --- .../workflows/content-pipeline-watchdog.yml | 23 +++++++++++++++---- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/.github/workflows/content-pipeline-watchdog.yml b/.github/workflows/content-pipeline-watchdog.yml index c230a0c79..b3aad9a5e 100644 --- a/.github/workflows/content-pipeline-watchdog.yml +++ b/.github/workflows/content-pipeline-watchdog.yml @@ -49,11 +49,15 @@ jobs: RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} run: | set -uo pipefail - - # Every API call is `|| true`: the step shell is `bash -e`, so an - # unguarded 403 aborts mid-script and surfaces as a bare red run - # indistinguishable from a real content alert. That happened once - # (actions:read 403). Collect what we can, then decide explicitly. + # `set +e` is deliberate, and `-e` here is actively harmful. + # GitHub runs this step as `bash -e {0}`, which combined with + # pipefail means ANY non-zero anywhere kills the script at that + # line with no output at all — a red run and an empty log. It bit + # twice: a 403 from the Actions API, then a `grep` with no match + # inside the mirrored-paths filter (grep exits 1 on no-match, which + # is a normal answer, not an error). This script decides everything + # explicitly below; it must never die silently mid-check. + set +e # ---- hop 3: what production is serving ----------------------- LIVE=$(GH_TOKEN=$UI_TOKEN gh api "repos/$REPO/contents/src/content?ref=main" --jq '.sha' || true) @@ -87,6 +91,15 @@ jobs: if [ -n "$PUBLISHED" ]; then EXPECTED="$SHA"; break; fi done + # On failure `gh` prints its error body to stdout, which would land + # in these vars and read as a bogus sha. Keep only real ones, so a + # broken call resolves to empty and trips the DEGRADED branch below + # instead of being compared as if it were data. + only_sha() { printf '%s' "${1:-}" | grep -Eox '[0-9a-f]{40}' || true; } + LIVE=$(only_sha "$LIVE") + TIP=$(only_sha "$TIP") + EXPECTED=$(only_sha "$EXPECTED") + MIRRORED=$(printf '%s' "$TIP_MSG" | sed -nE 's/.*mono@([0-9a-f]+).*/\1/p') echo "expected mono@${EXPECTED:0:7} | mirrored @${MIRRORED:-none} | peanut-content ${TIP:0:7} | live ${LIVE:0:7}"