fix(watchdog): don't treat a mid-run mirror as an emergency by Hugo0 · Pull Request #2578 · peanutprotocol/peanut-ui

Hugo0 · 2026-07-29T16:49:57Z

The watchdog false-alarmed in Discord today. Content was fine — it caught the pipeline mid-flight:

16:03:52  push f3008839
16:04:13  watchdog: "the mirror is behind mono ... Stuck for 5m"   ← 21s after the push
16:04:25  mirror completes                                          ← 12s after the watchdog looked
16:43:38  green

Cause

Mirror-behind skipped the 45-minute grace period, on my reasoning that "a failed mirror will not fix itself." That reasoning was wrong. A mirror that is merely mid-run fixes itself in ~30 seconds, and from outside it is indistinguishable from a dead one.

So any content push landing shortly before a */15 tick produces a false alarm — roughly 1 in 20 pushes, indefinitely. That is exactly the alert-fatigue pattern that makes a watchdog worthless.

Fix

Skipping the grace period is now reserved for states that genuinely cannot recover: red CI, and a watchdog that cannot evaluate itself. A dead mirror still alerts — it just waits out the window first, which is what the window is for.

Decision table, verified

Case	Before	After
mirror mid-flight, 5m	ALERT ❌	SILENT ✅
mirror genuinely stuck, 60m	ALERT	ALERT ✅
prod behind, 5m	SILENT	SILENT ✅
prod behind, 60m	ALERT	ALERT ✅
prod behind, CI red	ALERT now	ALERT now ✅
watchdog degraded	ALERT now	ALERT now ✅
healthy	silent	silent ✅

Also ran the real step under bash -e against live APIs — reports production content is current, exit 0.

Note

This is a false-alarm fix, not a pipeline fix. Content publishing itself is working: three consecutive autonomous publishes today (#2561, #2575, #2576), all merged by the automation, production in sync.

Summary by CodeRabbit

Bug Fixes
- Improved pipeline monitoring during degraded conditions.
- Watchdog evaluation failures are now escalated immediately.
- Mirror delays receive the full grace period before being marked as failed, reducing premature failure alerts.

The watchdog alerted 21 seconds after a content push, 12 seconds before the mirror finished, and was green again on the next tick: 16:03:52 push f3008839 16:04:13 watchdog: "the mirror is behind mono ... Stuck for 5m" 16:04:25 mirror completes 16:43:38 green A mirror-behind state skipped the 45-minute grace period, on the reasoning that "a failed mirror will not fix itself". That reasoning was wrong. A mirror that is merely mid-run fixes itself in about 30 seconds, and the two states look identical from outside. So any push landing shortly before a */15 tick produced a false alarm — roughly one in twenty pushes, indefinitely. Skipping the grace period is now reserved for states that genuinely cannot recover: red CI, and a watchdog that cannot evaluate itself. A dead mirror still alerts; it waits out the window first, which is what the window is for. Verified the full decision table: mirror mid-flight, 5m -> SILENT (was: ALERT — the bug) mirror stuck, 60m -> ALERT prod behind, 5m -> SILENT prod behind, 60m -> ALERT prod behind, CI RED -> ALERT immediately watchdog degraded -> ALERT immediately healthy -> SILENT Also ran the real step under `bash -e` against live APIs: reports current, exits 0.

cursor · 2026-07-29T16:50:01Z

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

vercel · 2026-07-29T16:50:04Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
peanut-wallet	Ready	Preview, Comment	Jul 29, 2026 4:56pm

coderabbitai · 2026-07-29T16:50:18Z

📝 Walkthrough

Walkthrough

The watchdog workflow replaces MIRROR_BROKEN with an URGENT flag. Degraded watchdog evaluation marks runs urgent and skips the grace period, while mirror-behind staleness remains non-urgent and waits through the full period.

Changes

Watchdog grace-period classification

Layer / File(s)	Summary
Classify urgent watchdog states `.github/workflows/content-pipeline-watchdog.yml`	The workflow distinguishes degraded watchdog evaluation from mirror-behind staleness, uses `URGENT` for `DOOMED` assignment, and updates the grace-period comment.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

peanutprotocol/peanut-ui#2542: Updates the same watchdog’s degraded-state and stale-computation logic.
peanutprotocol/peanut-ui#2544: Changes watchdog degraded-path behavior and SHA input handling in the same workflow.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the watchdog grace-period fix and the change to treat mid-run mirror states as non-emergencies.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/watchdog-mirror-inflight

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-07-29T16:51:27Z

Code-analysis diff

Painscore total: 6308.27 → 6308.27 (0)
Findings: 0 net (+0 new, -0 resolved)

github-actions · 2026-07-29T16:51:54Z

🧪 UI test report — ✅ all green

Suites

✅ unit: 2244 ran, 0 failed, 0 skipped, 32.8s

📊 Coverage (unit)

metric	%
statements	61.9%
branches	45.4%
functions	51.3%
lines	62.3%

⏱ 10 slowest test cases

time	test
2.8s	`src/components/Card/share-asset/__tests__/shareAssetLayout.test.ts` › never places two stickers in heavy overlap (broad seed sweep)
1.0s	`src/utils/__tests__/demo-api.test.ts` › isDemoMode() is false when not running under Capacitor
0.5s	`src/lib/url-parser/parse-token-amount.test.ts` › should throw for invalid amount formats
0.5s	`src/components/Card/share-asset/__tests__/shareAssetLayout.test.ts` › every sticker stays within canvas at any count
0.3s	`src/app/(mobile-ui)/withdraw/__tests__/withdraw-states.test.tsx` › Bank withdrawal keeps the $1 minimum for sub-$1 amounts
0.3s	`src/app/actions/__tests__/api-headers-extended.test.ts` › should not include apiKey in validateInviteCode body
0.2s	`src/app/actions/__tests__/api-headers.test.ts` › should include Content-Type in validateInviteCode
0.2s	`src/components/Card/share-asset/__tests__/shareAssetLayout.test.ts` › keeps stickers off the username pill (final pass respects the keep-out)
0.2s	`src/utils/__tests__/demo-balance.test.ts` › debits and floors at zero
0.1s	`src/utils/__tests__/demo-balance.test.ts` › starts at the full balance on a fresh install and stamps a timestamp

_{📍 Inline annotations are in the **Unit test report** check above. Coverage artifact: `coverage-unit`. Generated by `.github/workflows/tests.yml`.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/content-pipeline-watchdog.yml:
- Line 149: Update the watchdog’s mirror-behind grace calculation around DOOMED,
EXPECTED, and TIP_AT to track the timestamp of the selected EXPECTED commit and
use it when computing mirror-behind AGE; continue using TIP_AT for
production-behind age. Preserve the full grace window for a newly detected mono
change and revise the nearby comment to match the resulting behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 67c5ecea-8ce3-4fac-9f74-e2d0a71cf738

📥 Commits

Reviewing files that changed from the base of the PR and between ede6c7a and f7241bd.

📒 Files selected for processing (1)

.github/workflows/content-pipeline-watchdog.yml

coderabbitai · 2026-07-29T16:52:10Z


-                  # A failed mirror will not fix itself, so skip the grace period.
-                  DOOMED=$MIRROR_BROKEN
+                  DOOMED=$URGENT


🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Anchor mirror grace to the newly detected mono change.

DOOMED=$URGENT removes the explicit bypass, but the grace check still uses AGE from TIP_AT. If the mirror’s previous publish is already older than 45 minutes, a fresh mono change makes the watchdog alert immediately on the first tick, so the false alarm remains possible. Track the selected EXPECTED commit timestamp and use it for mirror-behind age; keep TIP_AT for production-behind age. The updated comment is otherwise misleading because this state does not always receive the full window.

Also applies to: 160-163

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/content-pipeline-watchdog.yml at line 149, Update the watchdog’s mirror-behind grace calculation around DOOMED, EXPECTED, and TIP_AT to track the timestamp of the selected EXPECTED commit and use it when computing mirror-behind AGE; continue using TIP_AT for production-behind age. Preserve the full grace window for a newly detected mono change and revise the nearby comment to match the resulting behavior.

coderabbitai Bot reviewed Jul 29, 2026

View reviewed changes

vercel Bot deployed to Preview July 29, 2026 16:56 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(watchdog): don't treat a mid-run mirror as an emergency - #2578

fix(watchdog): don't treat a mid-run mirror as an emergency#2578
Hugo0 wants to merge 1 commit into
mainfrom
fix/watchdog-mirror-inflight

Hugo0 commented Jul 29, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

cursor Bot commented Jul 29, 2026

Uh oh!

vercel Bot commented Jul 29, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 29, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

github-actions Bot commented Jul 29, 2026

Uh oh!

github-actions Bot commented Jul 29, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Jul 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Hugo0 commented Jul 29, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Cause

Fix

Decision table, verified

Note

Summary by CodeRabbit

Uh oh!

cursor Bot commented Jul 29, 2026

Uh oh!

vercel Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

github-actions Bot commented Jul 29, 2026

Code-analysis diff

Uh oh!

github-actions Bot commented Jul 29, 2026

🧪 UI test report — ✅ all green

Suites

📊 Coverage (unit)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Hugo0 commented Jul 29, 2026 •

edited by coderabbitai Bot

Loading

vercel Bot commented Jul 29, 2026 •

edited

Loading

coderabbitai Bot commented Jul 29, 2026 •

edited

Loading