feat(observability): alert on stale backups, degraded databases and expiring certificates

.github/workflows/validate-scaffold.yaml

@@ -63,6 +63,12 @@ jobs:
         # aliases, and fails on anything that would bring the retired stacks back.
         run: ./scripts/validate-observability-option.sh
 
+      - name: 🩺 Validate the backup, database and certificate check
+        # Runs the production platform-health-alert script against recorded API
+        # responses: every problem shape is reported, healthy and uninstalled
+        # kinds stay quiet, and a broken read or delivery fails the check.
+        run: ./scripts/tests/test-platform-health-alert.sh
+
       - name: 🔔 Validate Flux alert coverage
         # The production Alert is the only notification for a failed HelmRelease,
         # and it has no namespace wildcard: every namespace holding a release or a

docs/TEMPLATING.md

@@ -144,8 +144,9 @@ deny policy from blocking the new workload during an upgrade. The invalid URL
 fallback keeps local and unconfigured instances inert.
 Flux notification-controller now owns reconciliation errors in
 production, including dependency waits.
-Some Prometheus alert rules have no replacement: certificate expiry, backup
-freshness and autoscaler failures are not alerted today.
+In production, a `platform-health-alert` CronJob reports failed or stale backups,
+degraded CloudNativePG clusters and certificates close to expiry through the same
+webhook. Cluster Autoscaler failures are left to Coroot.
 [`docs/dr/alerting.md`](dr/alerting.md) lists what is and is not alerted.
 
 Open the Coroot UI at `https://observability.<your-domain>`. Community Edition

docs/dr/alerting.md

@@ -4,14 +4,18 @@
 and incidents, collected by an eBPF node agent with no application changes.
 Self-hosted, no SaaS metrics tier. There is no kube-prometheus-stack: no
 Prometheus operator, Alertmanager, node-exporter or kube-state-metrics, and no
-`monitoring` namespace. Alerting has three parts:
+`monitoring` namespace. Alerting has four parts:
 
 1. **Coroot incidents go to Slack.** Coroot inspects every workload and node
    and posts incident and resolution notifications in production.
 2. **Flux reports its own failures.** Flux notification-controller posts
    reconciliation errors for every `Kustomization` and every `HelmRelease`.
 3. **One off-cluster dead-man check.** The `cluster-heartbeat` CronJob pings an
    external monitor, so a cluster that is down still gets noticed.
+4. **A health check for backups, databases and certificates.** The
+   `platform-health-alert` CronJob posts what Coroot and Flux do not see: a
+   failed or stale backup, a degraded CloudNativePG cluster and a certificate
+   close to expiry. See [Backup, database and certificate check](#backup-database-and-certificate-check).
 
 There is one UI, behind SSO: Coroot at `https://observability.${domain}`,
 reached only through oauth2-proxy.
@@ -23,18 +27,19 @@ says how an instance that ran them migrates.
 
 ## Components
 
-| Component           | Role                                               | Persistence (prod) |
-| ------------------- | -------------------------------------------------- | ------------------ |
-| Coroot              | UI, incidents, service map, alert inspections      | PVC, 2 Gi          |
-| Coroot Prometheus   | Coroot's own metric store                          | PVC, 10 Gi         |
-| Coroot ClickHouse   | Logs, traces and profiles, 3 d retention           | PVC, 10 Gi         |
-| Coroot node agent   | eBPF telemetry from every node (DaemonSet)         | n/a                |
-| Audit log forwarder | Ships Talos host audit logs to Coroot (prod only)  | n/a                |
-| Flux notifications  | `Provider` and two `Alert` objects (prod only)     | n/a                |
-| cluster-heartbeat   | Dead-man ping every five minutes (CronJob)         | n/a                |
-
-Local/CI runs the same stack without the audit log forwarder and without the
-Flux notification path. Coroot's bundled Prometheus is its own metric store; it
+| Component             | Role                                               | Persistence (prod) |
+| --------------------- | -------------------------------------------------- | ------------------ |
+| Coroot                | UI, incidents, service map, alert inspections      | PVC, 2 Gi          |
+| Coroot Prometheus     | Coroot's own metric store                          | PVC, 10 Gi         |
+| Coroot ClickHouse     | Logs, traces and profiles, 3 d retention           | PVC, 10 Gi         |
+| Coroot node agent     | eBPF telemetry from every node (DaemonSet)         | n/a                |
+| Audit log forwarder   | Ships Talos host audit logs to Coroot (prod only)  | n/a                |
+| Flux notifications    | `Provider` and two `Alert` objects (prod only)     | n/a                |
+| cluster-heartbeat     | Dead-man ping every five minutes (CronJob)         | n/a                |
+| platform-health-alert | Backup, database and certificate check (prod only) | n/a                |
+
+Local/CI runs the same stack without the audit log forwarder, the health check and
+the Flux notification path. Coroot's bundled Prometheus is its own metric store; it
 is not a Prometheus operator and installs no `monitoring.coreos.com` resources.
 
 ## Alert routing → Slack
@@ -98,26 +103,54 @@ and ~10 min grace and connect it to Slack. The CronJob deliberately accepts only
 
 ## What is alerted, and what is not
 
-| Signal                                      | Where it shows                              |
-| ------------------------------------------- | ------------------------------------------- |
-| A workload misses its objectives            | Coroot incident → Slack                     |
-| Node, pod, memory, disk and restart health  | Coroot inspections, in the Coroot UI        |
-| A `Kustomization` fails to build or apply   | Flux `Alert` → Slack                        |
-| A `HelmRelease` fails to install or upgrade | Flux `Alert` → Slack                        |
-| A prerequisite layer is not ready           | Flux `Alert` → Slack (dependency wait)      |
-| The cluster stops answering                 | External monitor → Slack                    |
-
-**Not alerted today.** The Prometheus rules that left with kube-prometheus-stack
-had no direct equivalent in Coroot's model, and these are not replaced:
-
-- certificate expiry (`CertificateExpiringSoon`);
-- a failed or stale Velero backup, and a CloudNativePG cluster without a recent
-  backup or a streaming replica;
-- cluster-autoscaler scale-up failures.
-
-Check backups with the restore drill in [restore-drill.md](./restore-drill.md)
-and certificates with `kubectl get certificates -A` until an instance adds its
-own check for them.
+| Signal                                       | Where it shows                                |
+| -------------------------------------------- | --------------------------------------------- |
+| A workload misses its objectives             | Coroot incident → Slack                       |
+| Node, pod, memory, disk and restart health   | Coroot inspections, in the Coroot UI          |
+| A `Kustomization` fails to build or apply    | Flux `Alert` → Slack                          |
+| A `HelmRelease` fails to install or upgrade  | Flux `Alert` → Slack                          |
+| A prerequisite layer is not ready            | Flux `Alert` → Slack (dependency wait)        |
+| The cluster stops answering                  | External monitor → Slack                      |
+| A Velero backup failed or is stale           | Health check → Slack                          |
+| A CloudNativePG cluster is degraded or stale | Health check → Slack                          |
+| A certificate expires within 14 days         | Health check → Slack                          |
+| Cluster Autoscaler cannot add a node         | Coroot incident for the workload left Pending |
+
+**Cluster Autoscaler failures are left to Coroot.** KSail deploys and owns the
+autoscaler, not this repository's manifests, so a check here would have to
+guess its namespace and its event format. What matters to an operator is the
+effect: pods that stay Pending, which Coroot reports as the workload missing
+its objectives. Read the autoscaler's own state from its
+`cluster-autoscaler-status` ConfigMap.
+
+## Backup, database and certificate check
+
+The `platform-health-alert` CronJob replaces the Prometheus rules that left
+with kube-prometheus-stack. It runs every six hours in production, in the
+`observability` namespace, and posts one Slack message listing every problem
+it finds:
+
+- a Velero schedule whose latest backup failed, or that has no completed
+  backup in the last 26 hours;
+- a CloudNativePG cluster with no completed `Backup` in the last 26 hours,
+  fewer ready instances than it asks for, or a configured WAL archive that is
+  not working, each for longer than 15 minutes;
+- a cert-manager `Certificate` that expires within 14 days, which means
+  cert-manager has not renewed it.
+
+It posts to the same webhook as Coroot and Flux, so there is nothing new to
+provision. Its network path is the `observability` namespace's `allow-coroot`
+policy, which already lets every pod but the heartbeat reach the API server, DNS
+and, in production, `hooks.slack.com:443`; the validator fails if that changes. A problem is reported again on every run until it is fixed. Its
+identity can only list Velero schedules and backups, CloudNativePG clusters
+and backups, and certificates. A kind whose API is not installed is skipped;
+any other read or delivery failure fails the Job, so check
+`kubectl -n observability get jobs` after a quiet week if you doubt it.
+`scripts/tests/test-platform-health-alert.sh` runs the check against recorded
+API responses in CI.
+
+Every CloudNativePG cluster is expected to have a recent backup. A database you
+add without a `ScheduledBackup` is reported until it has one.
 
 ## Off-cluster backup
 

k8s/providers/hetzner/infrastructure/controllers/kustomization.yaml

@@ -41,6 +41,8 @@ resources:
   - kubelet-serving-cert-approver/
   - longhorn/
   - origin-ca-issuer/
+  # Backup, database and certificate checks that post to Slack (#423).
+  - platform-health-alert/
 patches:
   - path: openbao/patches/store-data-on-hcloud.yaml
   - path: coroot/patches/allow-slack-webhook-egress.yaml

k8s/providers/hetzner/infrastructure/controllers/platform-health-alert/cluster-role-binding.yaml

@@ -0,0 +1,13 @@
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: platform-health-alert
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: platform-health-alert
+subjects:
+  - kind: ServiceAccount
+    name: platform-health-alert
+    namespace: observability

k8s/providers/hetzner/infrastructure/controllers/platform-health-alert/cluster-role.yaml

@@ -0,0 +1,16 @@
+---
+# Read-only: the check only lists the objects whose status it judges.
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: platform-health-alert
+rules:
+  - apiGroups: ["velero.io"]
+    resources: ["schedules", "backups"]
+    verbs: ["list"]
+  - apiGroups: ["postgresql.cnpg.io"]
+    resources: ["clusters", "backups"]
+    verbs: ["list"]
+  - apiGroups: ["cert-manager.io"]
+    resources: ["certificates"]
+    verbs: ["list"]