Repository navigation
Add monitoring: VictoriaMetrics stack and Grafana on grafana.int - #106
Merged
Merged
Conversation
The plan replaces Uptime Kuma with victoria-metrics-k8s-stack. The operator is its own release in infrastructure/controllers so the VM CRDs exist before the stack and the scrape objects in infrastructure/configs are applied. - VMSingle keeps 30 days (40 GB disks, 30 s scrape interval). - k3s runs the controller manager, scheduler and kube-proxy inside the k3s process and has no etcd, so those scrapes, rules and dashboards are off. - Grafana signs in through Ory (client grafana, infra-admins only, as server admin), is reachable from the mesh only and has no PVC. - Extra scrapes: Traefik, cloudflared, cert-manager, CNPG operator, Flux, the VM operator; CNPG and Garage next to the data app, Stalwart next to mail (the Prometheus endpoint is switched on in Stalwart's settings). - kube-state-metrics exports the bulbashenko.com/* labels; the dashboard "Cluster by layer" groups resources and health by them. - Alertmanager has no receiver yet (Telegram is #75). Refs #74 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #74.
What
infrastructure/controllers/victoria-metrics-operator.yaml: namespacemonitoringand VM operator 0.68.1 (v0.75.0). CRDs come from the plain crds/ dir, with CreateReplace. The webhook cert comes from cert-manager.infrastructure/configs/monitoring/: victoria-metrics-k8s-stack 0.96.0 (VM v1.154.0), releasevm:cluster=hel1.bulbashenko.com/layerandbulbashenko.com/applabels.grafana.int, and OIDC via Hydra clientgrafana(auth#18). Onlyinfra-adminsget in, and they get GrafanaAdmin.dashboards/cluster-by-layer.json): CPU, memory, restarts and readiness by layer and by app, plus a table of scrape targets.apps/production/data/scrapes.yaml: CNPGpg(9187) and Garage (/metricson the admin port, with the existing metrics token).apps/mail/scrape.yamlplusSTALWART_METRICS_USERNAME/STALWART_METRICS_PASSWORDin Secretstalwart.apps/production/auth/secret.sops.yaml:OIDC_GRAFANA_CLIENT_SECRET, the same value as inmonitoring/grafana.grafana.int → 100.64.0.5. Headscale restarts because the ConfigMap hash changes.Order and owner steps
infrastructure; it retries.deploy/authPR it opens. The hydra-clients Job then creates thegrafanaclient.kubectl -n mail rollout restart deploy/stalwart, so the pod gets the new env vars. Mail is down for a few seconds; senders retry.vmagent, authSecret = Environment variableSTALWART_METRICS_PASSWORD.Checked locally
kubectl kustomizepasses for every overlay CI builds.helm templateof both charts with these values renders; VMPodScrape fields were checked against the CRD schema.scripts/check-sops.shpasses; the client secret is identical in both Secrets (compared by hash).🤖 Generated with Claude Code