fix(watchdog): stop the script dying silently under bash -e - #2544
Merged
Conversation
Third fix, same root cause each time: I verified in an environment that was not
the one it runs in.
GitHub runs the step as `bash -e {0}`. Combined with `set -o pipefail` that
means any non-zero anywhere kills the script at that line with NO output — a red
run and an empty log, which is the least debuggable possible failure. It hit
twice: the Actions-API 403, then `grep` returning 1 because a commit touched no
mirrored paths. grep exiting 1 on no-match is a normal answer, not an error, and
it was the very first loop iteration — so the last run produced literally
nothing between the env block and `exit 1`.
`set +e` explicitly. Every branch here decides for itself and reports why; `-e`
adds nothing but the ability to die mid-check without saying so.
Also filter LIVE/TIP/EXPECTED through a 40-hex check. On failure `gh` prints its
error body to stdout, which was landing in those vars and being compared as if
it were a sha; now a broken call resolves to empty and trips the labelled
DEGRADED branch instead.
Verified by extracting the step and running it under `bash -e` against the real
APIs, both paths: healthy -> "production content is current", exit 0, silent;
broken token -> one WATCHDOG DEGRADED line, exit 1. That is the check I should
have been running from the first commit.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Contributor
📝 WalkthroughWalkthroughThe watchdog now continues through expected command failures and validates API-derived commit SHAs before deciding whether content is stuck or degraded. ChangesWatchdog fault tolerance
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
Code-analysis diffPainscore total: 6309.47 → 6309.47 (0) |
Contributor
🧪 UI test report — ✅ all greenSuites
📊 Coverage (unit)
⏱ 10 slowest test cases
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/content-pipeline-watchdog.yml:
- Around line 52-60: Update the watchdog script’s `gh` result handling so every
call validates its exit status before downstream use, including `TIP_AT`, `NUM`,
`PR_SHA`, and `CI`; clear invalid outputs rather than treating error text as
data. Validate the timestamp before computing `AGE`, ensure invalid or unset
values cannot trigger `set -u`, and route all such failures to the existing
`WATCHDOG DEGRADED` outcome.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 0d3898dc-25d9-403a-995a-b97536b76aee
📒 Files selected for processing (1)
.github/workflows/content-pipeline-watchdog.yml
Comment on lines
+52
to
+60
| # `set +e` is deliberate, and `-e` here is actively harmful. | ||
| # GitHub runs this step as `bash -e {0}`, which combined with | ||
| # pipefail means ANY non-zero anywhere kills the script at that | ||
| # line with no output at all — a red run and an empty log. It bit | ||
| # twice: a 403 from the Actions API, then a `grep` with no match | ||
| # inside the mirrored-paths filter (grep exits 1 on no-match, which | ||
| # is a normal answer, not an error). This script decides everything | ||
| # explicitly below; it must never die silently mid-check. | ||
| set +e |
Contributor
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Handle every failed gh result before downstream use.
set +e allows the script to continue, but only LIVE, TIP, and EXPECTED are sanitized. A failed gh call can still place error text in TIP_AT; line 135 then passes it to date, potentially leaving AGE invalid or unset, and the later $AGE expansion can abort under set -u before WATCHDOG DEGRADED is reported. Preserve each command’s exit status and clear or validate outputs such as TIP_AT, NUM, PR_SHA, and CI; invalid timestamps should also mark the watchdog degraded.
Also applies to: 94-102
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/content-pipeline-watchdog.yml around lines 52 - 60, Update
the watchdog script’s `gh` result handling so every call validates its exit
status before downstream use, including `TIP_AT`, `NUM`, `PR_SHA`, and `CI`;
clear invalid outputs rather than treating error text as data. Validate the
timestamp before computing `AGE`, ensure invalid or unset values cannot trigger
`set -u`, and route all such failures to the existing `WATCHDOG DEGRADED`
outcome.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third fix for this watchdog, and the root cause was the same all three times: I verified in an environment that wasn't the one it runs in.
The bug
The last run produced no output at all — nothing between the env block and
exit 1:GitHub runs the step as
bash -e {0}. Withset -o pipefail, any non-zero anywhere kills the script at that line with no message. It died on the first loop iteration:grepreturns 1 when a commit touched no mirrored paths — a normal answer, not an error — andd9a33deis exactly that case.Reproduced in isolation:
The fix
set +e, explicitly and with the reasoning in a comment. Every branch in this script already decides for itself and reports why;-econtributes nothing except the ability to die mid-check without saying so.Also filter
LIVE/TIP/EXPECTEDthrough a 40-hex check — on failureghprints its error body to stdout, which was landing in those vars and being compared as though it were a sha (tip={ "m). Now a broken call resolves to empty and trips the labelled DEGRADED branch.Verification — the check I should have run from the start
Extracted the step from the YAML and executed it under
bash -eagainst the real APIs, both paths:bash -nand dry-running the logic in my own shell both passed every time and caught none of this, because neither has-eset.Summary by CodeRabbit