Skip to content

fix(watchdog): stop the script dying silently under bash -e #2544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Hugo0 merged 1 commit into from
Jul 28, 2026
+18 −5
Merged

fix(watchdog): stop the script dying silently under bash -e #2544

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 18 additions & 5 deletions .github/workflows/content-pipeline-watchdog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,15 @@ jobs:
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
set -uo pipefail

# Every API call is `|| true`: the step shell is `bash -e`, so an
# unguarded 403 aborts mid-script and surfaces as a bare red run
# indistinguishable from a real content alert. That happened once
# (actions:read 403). Collect what we can, then decide explicitly.
# `set +e` is deliberate, and `-e` here is actively harmful.
# GitHub runs this step as `bash -e {0}`, which combined with
# pipefail means ANY non-zero anywhere kills the script at that
# line with no output at all — a red run and an empty log. It bit
# twice: a 403 from the Actions API, then a `grep` with no match
# inside the mirrored-paths filter (grep exits 1 on no-match, which
# is a normal answer, not an error). This script decides everything
# explicitly below; it must never die silently mid-check.
set +e
Comment on lines +52 to +60

@coderabbitai coderabbitai Bot Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Handle every failed gh result before downstream use.

set +e allows the script to continue, but only LIVE, TIP, and EXPECTED are sanitized. A failed gh call can still place error text in TIP_AT; line 135 then passes it to date, potentially leaving AGE invalid or unset, and the later $AGE expansion can abort under set -u before WATCHDOG DEGRADED is reported. Preserve each command’s exit status and clear or validate outputs such as TIP_AT, NUM, PR_SHA, and CI; invalid timestamps should also mark the watchdog degraded.

Also applies to: 94-102

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/content-pipeline-watchdog.yml around lines 52 - 60, Update
the watchdog script’s `gh` result handling so every call validates its exit
status before downstream use, including `TIP_AT`, `NUM`, `PR_SHA`, and `CI`;
clear invalid outputs rather than treating error text as data. Validate the
timestamp before computing `AGE`, ensure invalid or unset values cannot trigger
`set -u`, and route all such failures to the existing `WATCHDOG DEGRADED`
outcome.


# ---- hop 3: what production is serving -----------------------
LIVE=$(GH_TOKEN=$UI_TOKEN gh api "repos/$REPO/contents/src/content?ref=main" --jq '.sha' || true)
Expand Down Expand Up @@ -87,6 +91,15 @@ jobs:
if [ -n "$PUBLISHED" ]; then EXPECTED="$SHA"; break; fi
done

# On failure `gh` prints its error body to stdout, which would land
# in these vars and read as a bogus sha. Keep only real ones, so a
# broken call resolves to empty and trips the DEGRADED branch below
# instead of being compared as if it were data.
only_sha() { printf '%s' "${1:-}" | grep -Eox '[0-9a-f]{40}' || true; }
LIVE=$(only_sha "$LIVE")
TIP=$(only_sha "$TIP")
EXPECTED=$(only_sha "$EXPECTED")

MIRRORED=$(printf '%s' "$TIP_MSG" | sed -nE 's/.*mono@([0-9a-f]+).*/\1/p')

echo "expected mono@${EXPECTED:0:7} | mirrored @${MIRRORED:-none} | peanut-content ${TIP:0:7} | live ${LIVE:0:7}"
Expand Down
Loading