You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(observability): alert on stale backups, degraded databases and expiring certificates - #426
When the old Prometheus stack was retired, three warnings lost their home: a backup that failed or stopped running, a database left without a recent backup or a working replica, and a certificate about to expire. Today these are found only when someone looks, or when the backup is needed.
What
Production clusters now run a small read-only check every six hours that posts to the existing Slack alerts channel whenever a backup is failed or stale, a database is degraded or unbacked, or a certificate expires within 14 days. Autoscaler failures are deliberately left to the existing observability tool, and the alerting guide now explains all of this. Delivery to Slack on a real production cluster is still to be observed.
…xpiring certificates
Retiring kube-prometheus-stack left three signals without an alert. A
production-only CronJob in the observability namespace now lists Velero
schedules and backups, CloudNativePG clusters and backups, and cert-manager
certificates every six hours, and posts one Slack message through the
existing webhook when a backup failed or is stale, a database has no recent
backup, a missing replica or a failing WAL archive, or a certificate
expires within 14 days. Its identity can only list those kinds.
Cluster Autoscaler failures are left to Coroot: KSail owns the autoscaler,
and its effect, pods left Pending, already surfaces as a Coroot incident.
A new test runs the rendered script against recorded API responses, the
observability validator pins the rendered resources and their absence in
local, and the alerting guide lists the new signals.
Part of #423
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: ASSERTIVE
Plan: Advanced
Run ID: 7ade4e26-b393-4282-95af-9115ad57195f
Commits
Reviewing files that changed from the base of the PR and between 42c04cb and efd1010.
Files selected for processing: 2
docs/dr/alerting.md
scripts/validate-observability-option.sh
No actionable comments were generated in the recent review.
Walkthrough
Adds a production-only, six-hour CronJob to check Velero backups, CloudNativePG health, and certificate expiry. The job uses scoped Kubernetes read permissions and sends findings through a configured webhook. Missing APIs are skipped, while other read and delivery failures fail the job. Adds tests, scaffold validation, a workflow test step, and documentation updates.
Possibly related issues
Issue 423 identifies missing alerts for stale or failed backups and expiring certificates. This PR adds production checks and alerts for those conditions.
Issue 65 calls for observability alert updates as part of the Coroot migration. This PR adds backup, database, and certificate signals that do not depend on Prometheus.
This change adds a production-only six-hour health check that alerts on stale backups, degraded databases and expiring certificates. It has scoped read-only permissions and failure handling covered by tests. No actionable merge-blocking risk remains.
Pre-merge checks: 5
Check name
Status
Explanation
Title check
The title clearly summarizes the main change: alerts for stale backups, degraded databases, and expiring certificates.
Description check
The description explains the production health check, the alerts it sends, and the observability documentation changes.
Docstring Coverage
Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage
Explanation
Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 2 files. (10 skipped: 10 unsupported.)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.
Recent review details
Additional context used
Code guidelines: 1
AGENTS.md — auto-discovered
Path-based instructions: 2
Source excerpt: Manifest naming follows the reference platform's conventions (platform#2315), enforced by the shared `devantler-tech/.github/actions/validate-naming` action (a CI gate on manifest, naming-configuration, and naming-workflow c...
Learnt from: CR
Repo: devantler-tech/platform-template
Timestamp: 2026-11-10T23:55:50.444Z
Learning: Source excerpt:
# AGENTS.md
## Validation
### File and directory naming conventions
Manifest naming follows the reference platform's conventions (platform#2315),
enforced by the shared `devantler-tech/.github/actions/validate-naming` action (a CI gate
on manifest, naming-configuration, and naming-workflow changes):
- **One Kubernetes resource per file** (vendored upstream operator bundles exempt).
Learnt from: CR
Repo: devantler-tech/platform-template
Timestamp: 2026-11-10T23:55:50.444Z
Learning: Source excerpt:
# AGENTS.md
## Validation
### File and directory naming conventions
Manifest naming follows the reference platform's conventions (platform#2315),
enforced by the shared `devantler-tech/.github/actions/validate-naming` action (a CI gate
on manifest, naming-configuration, and naming-workflow changes):
- **Kebab-case** directories and file stems everywhere under `k8s/` and `talos*/`
(vendored upstream files inside CR folders — e.g. the testkube CRD dumps —
keep their upstream names).
Learnt from: CR
Repo: devantler-tech/platform-template
Timestamp: 2026-11-10T23:55:50.444Z
Learning: Source excerpt:
# AGENTS.md
## Validation
### File and directory naming conventions
Manifest naming follows the reference platform's conventions (platform#2315),
enforced by the shared `devantler-tech/.github/actions/validate-naming` action (a CI gate
on manifest, naming-configuration, and naming-workflow changes):
- **Filenames lead with the kebab-cased Kind**: `<kind>.yaml` or
`<kind>-<purpose>.yaml` (e.g. `cilium-network-policy.yaml`, `http-route.yaml`,
`config-map-corefile.yaml`).
🤖 Generated by the Agentic Engineer (interactive Claude Code session)
User trial.
Rendered the production controller layer and ran the health-check script exactly as rendered (with sh, and again with dash) against recorded Kubernetes API responses. A healthy cluster posts nothing. A failed and a stale Velero schedule, a database with no backup, a database short of replicas past the grace period, a failing WAL archive and a certificate 10 days from expiry each produce one line in a single Slack message; a database still inside its grace period and a certificate 16 days out produce none.
The local placeholder webhook logs the findings and does not deliver. Clusters without these APIs are skipped. An API error, a malformed list, a failed delivery, a plain-http webhook and a webhook with a newline each fail the job.
The observability validator passes, rejects a widened ClusterRole and a CronJob on the wrong service account, and confirms local renders none of it.
Not yet proven (why this says Part of #423): the first two criteria need a Slack notification from a real production cluster, which this host cannot run. The autoscaler decision and the guide update are done here.
The platform-health-alert pod has no policy of its own: the observability
namespace's allow-coroot policy selects every pod but the heartbeat and
allows the API server, DNS and, in production, hooks.slack.com:443. The
validator now requires the pod to stay outside the heartbeat carve-out
and allow-coroot to keep those three egress rules, so narrowing either
fails CI instead of silently stopping the alerts.
Part of #423
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.
🤖 Generated by the Agentic Engineer (interactive Claude Code session)
User trial at efd1010. Everything in the earlier trial still holds at this head. New here: in the rendered production controller layer, the namespace's existing allow-coroot network policy selects the check's pod (every pod but the heartbeat) and allows the Kubernetes API, DNS and hooks.slack.com:443, so the job can read and deliver without a policy of its own. The validator now requires that, and fails when the pod gains the heartbeat label, when the Slack destination is removed, or when the API rule is narrowed to a port.
Still Part of #423: a Slack notification from a real production cluster is the one thing this host cannot show.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
When the old Prometheus stack was retired, three warnings lost their home: a backup that failed or stopped running, a database left without a recent backup or a working replica, and a certificate about to expire. Today these are found only when someone looks, or when the backup is needed.
What
Production clusters now run a small read-only check every six hours that posts to the existing Slack alerts channel whenever a backup is failed or stale, a database is degraded or unbacked, or a certificate expires within 14 days. Autoscaler failures are deliberately left to the existing observability tool, and the alerting guide now explains all of this. Delivery to Slack on a real production cluster is still to be observed.
Part of #423
🤖 Generated with Claude Code