Skip to content

fix(watchdog): use a scope the token actually has, and label watchdog faults #2542

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Hugo0 merged 1 commit into from
Jul 28, 2026
Merged

fix(watchdog): use a scope the token actually has, and label watchdog faults #2542

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 47 additions & 17 deletions .github/workflows/content-pipeline-watchdog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,18 @@ jobs:
run: |
set -uo pipefail

# Every API call is `|| true`: the step shell is `bash -e`, so an
# unguarded 403 aborts mid-script and surfaces as a bare red run
# indistinguishable from a real content alert. That happened once
# (actions:read 403). Collect what we can, then decide explicitly.

# ---- hop 3: what production is serving -----------------------
LIVE=$(GH_TOKEN=$UI_TOKEN gh api "repos/$REPO/contents/src/content?ref=main" --jq '.sha')
LIVE=$(GH_TOKEN=$UI_TOKEN gh api "repos/$REPO/contents/src/content?ref=main" --jq '.sha' || true)

# ---- hop 2: what the mirror has published --------------------
TIP=$(GH_TOKEN=$CONTENT_TOKEN gh api repos/peanutprotocol/peanut-content/commits/main --jq '.sha')
TIP_AT=$(GH_TOKEN=$CONTENT_TOKEN gh api repos/peanutprotocol/peanut-content/commits/main --jq '.commit.committer.date')
TIP=$(GH_TOKEN=$CONTENT_TOKEN gh api repos/peanutprotocol/peanut-content/commits/main --jq '.sha' || true)
TIP_MSG=$(GH_TOKEN=$CONTENT_TOKEN gh api repos/peanutprotocol/peanut-content/commits/main --jq '.commit.message' 2>/dev/null | head -1 || true)
TIP_AT=$(GH_TOKEN=$CONTENT_TOKEN gh api repos/peanutprotocol/peanut-content/commits/main --jq '.commit.committer.date' || true)

# ---- hop 1: is the mirror itself healthy? --------------------
# Deliberately NOT a SHA comparison against the "mirror: sync from
Expand All @@ -66,19 +72,43 @@ jobs:
# minutes forever. That is exactly what happened on the first live
# run (d9a33de, a docs edit to content/_system/ARCHITECTURE.md).
#
# Ask the question that has a real yes/no answer instead: did the
# mirror's last completed run succeed? A mirror that ran and
# published nothing is healthy; one that errored is not.
MIRROR=$(GH_TOKEN=$MONO_TOKEN gh api \
"repos/peanutprotocol/mono/actions/workflows/mirror-to-peanut-content.yml/runs?per_page=1&status=completed" \
--jq '.workflow_runs[0].conclusion // "none"')

echo "mirror last run: $MIRROR | peanut-content ${TIP:0:7} | live ${LIVE:0:7}"
# Instead, find the newest mono commit that touched a path the
# mirror actually publishes, and compare THAT against the stamp.
# Reading the Actions API for the mirror's run status would be
# simpler but MONO_READ_TOKEN has contents:read only — it 403s on
# actions:read, which is how the first attempt at this fix failed.
EXPECTED=""
for SHA in $(GH_TOKEN=$MONO_TOKEN gh api \
"repos/peanutprotocol/mono/commits?path=content&per_page=15" --jq '.[].sha' || true); do
FILES=$(GH_TOKEN=$MONO_TOKEN gh api "repos/peanutprotocol/mono/commits/$SHA" --jq '.files[].filename' || true)
# Published: content/** except content/_system/**, plus content/_system/generated/**
PUBLISHED=$( { printf '%s\n' "$FILES" | grep '^content/' | grep -v '^content/_system/'
printf '%s\n' "$FILES" | grep '^content/_system/generated/'; } | head -1 )
if [ -n "$PUBLISHED" ]; then EXPECTED="$SHA"; break; fi
done
Comment on lines +81 to +88

@coderabbitai coderabbitai Bot Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Per-commit fetch failure inside the loop is misclassified as "not published".

If gh api .../commits/$SHA --jq '.files[].filename' fails transiently for the actual newest publish-touching commit, FILES comes back empty via || true and the loop just moves on to an older SHA, silently setting EXPECTED to a stale commit. If the mirror already synced past that point, this produces exactly the false "mirror is behind mono" alert this PR is trying to eliminate — just one layer deeper. The top-level DEGRADED check (line 98) can't catch this because EXPECTED is non-empty, just wrong.

🐛 Proposed fix — distinguish fetch failure from "no published paths" 
                   EXPECTED=""
+                  FETCH_FAILED=false
                   for SHA in $(GH_TOKEN=$MONO_TOKEN gh api \
                         "repos/peanutprotocol/mono/commits?path=content&per_page=15" --jq '.[].sha' || true); do
-                    FILES=$(GH_TOKEN=$MONO_TOKEN gh api "repos/peanutprotocol/mono/commits/$SHA" --jq '.files[].filename' || true)
+                    if ! FILES=$(GH_TOKEN=$MONO_TOKEN gh api "repos/peanutprotocol/mono/commits/$SHA" --jq '.files[].filename'); then
+                      FETCH_FAILED=true
+                      break
+                    fi
                     # Published: content/** except content/_system/**, plus content/_system/generated/**
                     PUBLISHED=$( { printf '%s\n' "$FILES" | grep '^content/' | grep -v '^content/_system/'
                                    printf '%s\n' "$FILES" | grep '^content/_system/generated/'; } | head -1 )
                     if [ -n "$PUBLISHED" ]; then EXPECTED="$SHA"; break; fi
                   done
+                  [ "$FETCH_FAILED" = true ] && EXPECTED=""
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for SHA in $(GH_TOKEN=$MONO_TOKEN gh api \
"repos/peanutprotocol/mono/commits?path=content&per_page=15" --jq '.[].sha' || true); do
FILES=$(GH_TOKEN=$MONO_TOKEN gh api "repos/peanutprotocol/mono/commits/$SHA" --jq '.files[].filename' || true)
# Published: content/** except content/_system/**, plus content/_system/generated/**
PUBLISHED=$( { printf '%s\n' "$FILES" | grep '^content/' | grep -v '^content/_system/'
printf '%s\n' "$FILES" | grep '^content/_system/generated/'; } | head -1 )
if [ -n "$PUBLISHED" ]; then EXPECTED="$SHA"; break; fi
done
EXPECTED=""
FETCH_FAILED=false
for SHA in $(GH_TOKEN=$MONO_TOKEN gh api \
"repos/peanutprotocol/mono/commits?path=content&per_page=15" --jq '.[].sha' || true); do
if ! FILES=$(GH_TOKEN=$MONO_TOKEN gh api "repos/peanutprotocol/mono/commits/$SHA" --jq '.files[].filename'); then
FETCH_FAILED=true
break
fi
# Published: content/** except content/_system/**, plus content/_system/generated/**
PUBLISHED=$( { printf '%s\n' "$FILES" | grep '^content/' | grep -v '^content/_system/'
printf '%s\n' "$FILES" | grep '^content/_system/generated/'; } | head -1 )
if [ -n "$PUBLISHED" ]; then EXPECTED="$SHA"; break; fi
done
[ "$FETCH_FAILED" = true ] && EXPECTED=""
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/content-pipeline-watchdog.yml around lines 81 - 88, Update
the per-commit fetch logic in the SHA loop so a failed `gh api` request for
commit file names is distinguished from a successful response containing no
published paths. Track the request status separately, and mark the watchdog as
degraded or otherwise stop normal expected-SHA selection when fetching fails
instead of continuing to an older SHA; preserve the existing behavior for
successful commits with no published paths.


MIRRORED=$(printf '%s' "$TIP_MSG" | sed -nE 's/.*mono@([0-9a-f]+).*/\1/p')

echo "expected mono@${EXPECTED:0:7} | mirrored @${MIRRORED:-none} | peanut-content ${TIP:0:7} | live ${LIVE:0:7}"

# Cannot evaluate == broken watchdog, which must NOT look like a
# broken pipeline. Label it distinctly so nobody debugs the wrong
# thing, but still be loud — a watchdog that can't watch is an
# outage of the thing that tells us about outages.
if [ -z "$EXPECTED" ] || [ -z "$MIRRORED" ] || [ -z "$LIVE" ] || [ -z "$TIP" ]; then
echo "::error::WATCHDOG DEGRADED — could not evaluate the pipeline (expected=${EXPECTED:0:7} mirrored=${MIRRORED:-none} live=${LIVE:0:7} tip=${TIP:0:7}). This is a watchdog fault, not necessarily a content fault."
DEGRADED=true
else
DEGRADED=false
fi

STALE=""
MIRROR_BROKEN=false
if [ "$MIRROR" != "success" ] && [ "$MIRROR" != "none" ]; then
STALE="the mirror workflow's last run concluded \`$MIRROR\` — content is not leaving mono"
if [ "$DEGRADED" = true ]; then
STALE="**the watchdog itself could not evaluate the pipeline** — check its permissions before assuming content is stuck"
MIRROR_BROKEN=true
elif [ "${EXPECTED:0:7}" != "${MIRRORED:0:7}" ]; then
STALE="the mirror is behind mono (expected \`${EXPECTED:0:7}\`, published \`${MIRRORED:0:7}\`)"
MIRROR_BROKEN=true
elif [ "$LIVE" != "$TIP" ]; then
STALE="production is behind peanut-content (live \`${LIVE:0:7}\`, latest \`${TIP:0:7}\`)"
Expand All @@ -89,18 +119,18 @@ jobs:
exit 0
fi

AGE=$(( ( $(date -u +%s) - $(date -u -d "$TIP_AT" +%s) ) / 60 ))
AGE=$(( ( $(date -u +%s) - $(date -u -d "${TIP_AT:-now}" +%s) ) / 60 ))

# ---- which hop, and is it already doomed? --------------------
NUM=$(GH_TOKEN=$UI_TOKEN gh pr list --repo "$REPO" --state open --base main \
--json number,headRefName \
--jq '[.[] | select((.headRefName | startswith("auto/update-content-")) or (.headRefName | startswith("content/publish-to-main")))] | first | .number // empty')
--jq '[.[] | select((.headRefName | startswith("auto/update-content-")) or (.headRefName | startswith("content/publish-to-main")))] | first | .number // empty' || true)

# A failed mirror will not fix itself, so skip the grace period.
DOOMED=$MIRROR_BROKEN
if [ -n "$NUM" ]; then
SHA=$(GH_TOKEN=$UI_TOKEN gh api "repos/$REPO/pulls/$NUM" --jq '.head.sha')
CI=$(GH_TOKEN=$UI_TOKEN gh api "repos/$REPO/commits/$SHA/check-runs?per_page=100" \
PR_SHA=$(GH_TOKEN=$UI_TOKEN gh api "repos/$REPO/pulls/$NUM" --jq '.head.sha' || true)
CI=$(GH_TOKEN=$UI_TOKEN gh api "repos/$REPO/commits/$PR_SHA/check-runs?per_page=100" \
--jq '[.check_runs[] | select(.name=="ci-success")] | if length == 0 then "pending" elif all(.conclusion == "success") then "green" else "RED" end')
Comment on lines 131 to 134

@coderabbitai coderabbitai Bot Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

set -euo pipefail

printf '--- workflow excerpt ---\n'
nl -ba .github/workflows/content-pipeline-watchdog.yml | sed -n '1,170p'

printf '\n--- search for checks permission / check-runs usage ---\n'
rg -n 'checks:|check-runs|ci-success|permissions:' .github/workflows/content-pipeline-watchdog.yml

Repository: peanutprotocol/peanut-ui

Length of output: 253

🏁 Script executed:

set -euo pipefail

echo '--- workflow excerpt ---'
nl -ba .github/workflows/content-pipeline-watchdog.yml | sed -n '1,170p'

echo
echo '--- search for checks permission / check-runs usage ---'
rg -n 'checks:|check-runs|ci-success|permissions:' .github/workflows/content-pipeline-watchdog.yml

Repository: peanutprotocol/peanut-ui

Length of output: 227

🏁 Script executed:

set -euo pipefail

echo '--- workflow excerpt with line numbers ---'
awk 'NR>=1 && NR<=170 { printf "%4d  %s\n", NR, $0 }' .github/workflows/content-pipeline-watchdog.yml

echo
echo '--- search for checks permission / check-runs usage ---'
rg -n 'checks:|check-runs|ci-success|permissions:' .github/workflows/content-pipeline-watchdog.yml

Repository: peanutprotocol/peanut-ui

Length of output: 11401

🏁 Script executed:

set -euo pipefail

python3 - <<'PY'
from pathlib import Path
p = Path('.github/workflows/content-pipeline-watchdog.yml')
for i, line in enumerate(p.read_text().splitlines(), 1):
    if 1 <= i <= 170:
        print(f"{i:4d}  {line}")
PY

Repository: peanutprotocol/peanut-ui

Length of output: 10921

🏁 Script executed:

set -euo pipefail

echo '--- workflow excerpt with line numbers ---'
python3 - <<'PY'
from pathlib import Path
p = Path('.github/workflows/content-pipeline-watchdog.yml')
for i, line in enumerate(p.read_text().splitlines(), 1):
    if 1 <= i <= 170:
        print(f"{i:4d}  {line}")
PY

echo
echo '--- search for checks permission / check-runs usage ---'
rg -n 'checks:|check-runs|ci-success|permissions:' .github/workflows/content-pipeline-watchdog.yml

Repository: peanutprotocol/peanut-ui

Length of output: 11401

Guard the check-runs lookup and add checks: read.
UI_TOKEN only has contents: read and pull-requests: read, so this call needs checks: read; without it, the lookup can fail and, because it isn’t wrapped in || true, abort the watchdog mid-step.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/content-pipeline-watchdog.yml around lines 131 - 134,
Update the workflow permissions to grant checks: read alongside the existing
UI_TOKEN permissions, and guard the check-runs gh api invocation in the CI
assignment with || true so lookup failures do not abort the watchdog step.

WHERE="publish PR #$NUM is open, ci-success=$CI"
[ "$CI" = "RED" ] && DOOMED=true
Expand Down
Loading