Skip to content

fix(platform-dev): read dev's alerts in the health sweep - #494

Merged
bdchatham merged 4 commits into
mainfrom
fix/platform-dev-dev-alerts
Oct 13, 2026
Merged

bdchatham merged 4 commits into
mainfrom
fix/platform-dev-dev-alerts

Conversation

@bdchatham

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #493, for the non-blocking finding that seidroid raised on its final commit. #493 merged before the finding could be addressed.

Dev is not in Thanos, so the health sweep's firing query never sees dev's alerts. Procedure A's dev step checked only dev's Flux objects and pod phases. A dev alert, for example on disk or capacity, could fire while every pod ran, and the sweep would still report no alerts.

  • Procedure A step 7 now reads dev's firing alerts from dev's own Prometheus, through a read-only kubectl port-forward. If it cannot, the report must say that dev's alert state was not checked.
  • The health-sweep eval expects the same.
  • The verify sweep row for the federated cells now names procedure A, which depends on that fact.

Test plan

  • The port-forward and query in step 7 ran against dev and returned no firing alerts except Watchdog.
  • scripts/skill-package-checks.sh --skill-dir .claude/skills/platform-dev: every block and warn rule passes.
  • make check: every gate passed.
  • ./writing/scripts/lint.sh .claude/skills/platform-dev: 0 errors, 0 warnings, 0 suggestions.

The skill's name, description and model do not change, so seidroid sessions discover the same skill.

🤖 Generated with Claude Code

Dev is not in Thanos, so the sweep's firing query never sees its
alerts. Procedure A's dev step now reads dev's firing alerts from its
own Prometheus through a read-only port-forward, or says in the report
that dev's alert state was not checked. The health-sweep eval expects
the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

PR Summary

Low Risk
Documentation, procedure, and eval changes only; no runtime platform or application code.

Overview
Closes a gap in the read-only health sweep where dev was checked for Flux and pod health but not for firing alerts, because dev is outside Thanos and the fleet firing query never sees it.

Procedure A step 7 now documents a read-only kubectl port-forward to dev’s in-cluster Prometheus and a PromQL query for firing alerts (Watchdog excluded), including failure handling when the read cannot complete. Guardrails, the access table, and observability notes state that dev is the exception to using /grafana for alerts and logs.

The happy-path-health-sweep eval and maintenance verify row are updated so agents must report dev alert state (or explicitly say it was not checked) and must not port-forward to federated cells when Grafana MCP tools are available.

Reviewed by Cursor Bugbot for commit 4905caf. Bugbot is set up for automated code reviews on this repo. Configure here.

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Health-sweep step 7 now reads dev's firing alerts from dev's own Prometheus through a read-only port-forward, and the eval and maintenance row are updated to match. The procedure is read-only and sound, so nothing blocks; the snippet can still fail silently or query the wrong Prometheus, which matters because the report depends on telling those cases apart from 'nothing firing'.

Non-blocking

2 findings on the changed lines, as inline comments.

seidroid review · decision approve · session d7481e78e539492184d9d9052e090485 · turn resp_claude_1a694f454444523be8f665e10e01621c · item 881b3284594457c3b2b2364cc7ad8d06

Findings: 0 blocking | 2 non-blocking | 2 posted inline

…ts tunnel

The snippet now waits for the port-forward with curl's connection
retries, fails on an error with -sS --fail and a timeout, uses local
port 19093 so it cannot reach another Prometheus port-forward, and stops
the tunnel by PID, because a job spec does not work in a non-interactive
shell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bdchatham

Copy link
Copy Markdown
Contributor Author

@seidroid review

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit 1397638 rewrites the step 7 snippet: it waits for the port-forward with --retry-connrefused, fails loudly with -sS --fail --max-time, uses local port 19093 and stops the tunnel by PID. That fixes both of my earlier findings, including codex's port-forward race from the first round, so nothing is left open; codex found nothing on this round, and I found nothing new.

seidroid review · decision approve · session d7481e78e539492184d9d9052e090485 · turn resp_claude_5dc4494b6482a0857ae2d950a1eb68a0 · item e789bf18634159179414e599d061a6e1

Findings: 0 blocking | 0 non-blocking | 0 posted inline

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Guardrail 8, the access table, the observability reference and the
health-sweep eval now all say that dev is the one place where an agent
reads Prometheus through a port-forward, because Grafana has no
datasource for it. The eval no longer forbids the dev port-forward that
procedure A asks for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bdchatham

Copy link
Copy Markdown
Contributor Author

@seidroid review

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since my last review, the PR makes dev the one documented exception to the Grafana-only rule in four places: SKILL.md guardrail 8, its access table, observability §1 and the eval's forbidden signal. The step 7 snippet is unchanged and still fixes both of my earlier findings. Codex found nothing on this round and I found nothing new, so nothing blocks.

seidroid review · decision approve · session d7481e78e539492184d9d9052e090485 · turn resp_claude_459272bd54a0ecd6ff81f9dbd12c382a · item c754733e51a25de1bd6409bf6cb434f7

Findings: 0 blocking | 0 non-blocking | 0 posted inline

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e8ff742. Configure here.

Comment thread .claude/skills/platform-dev/SKILL.md Resolved
…tely

Guardrail 8 and the access table now say that dev's alerts come from its
own Prometheus and its pod logs from kubectl, instead of implying that
every dev read goes through Prometheus.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bdchatham

Copy link
Copy Markdown
Contributor Author

@seidroid review

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since my last review, the only change is in SKILL.md: guardrail 8 and the access table now also send dev's pod logs to kubectl --context dev logs, because dev is not in loki-fleet. That is consistent with the rest of the skill, my earlier findings stay fixed, and codex found nothing, which matches my reading, so nothing blocks.

seidroid review · decision approve · session d7481e78e539492184d9d9052e090485 · turn resp_claude_b4dfe849c2480ef391c1583df23a00db · item cc4349bda83654c0a68abaad2dbd4e08

Findings: 0 blocking | 0 non-blocking | 0 posted inline

@bdchatham
bdchatham merged commit f7a7f58 into main Oct 13, 2026
16 checks passed
@bdchatham
bdchatham deleted the fix/platform-dev-dev-alerts branch October 13, 2026 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant