Repository navigation
docs(plan): plan deployment phase 2 — monitoring and readiness for real users - #89
Merged
Merged
Conversation
Phase 1 put GameMesh in production; phase 2 closes what stands between it and real users: the data's only copy is the server's disk, nobody is told when the service or its certificate fails (finding #85), and production has no monitoring (#68). The plan sets nine measurable completion criteria (daily encrypted offsite backups, a restore drill onto an empty cluster with RPO and RTO measured, alerts for stale backups, k3s state backed up apart from its key (#80), outside monitoring of the endpoint and the certificate, metrics reachable only over the tailnet, basic alert rules, a capacity decision from measurements, an updated runbook), records the measured starting point, and splits the work into backups (E), outside monitoring and alerting (F), production monitoring (G) and the dated maintenance left from phase 1 (H), with the decisions each step needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The consultant's criteria for phase 2 arrived: Prometheus in the cluster with every target up, the application metrics, four Grafana panels, a public read-only dashboard at status.gamemesh.app and a k6 run that moves them, with alerting, log aggregation, long retention and polish kept out of scope as a README roadmap. They become part G and criteria K10 to K15, with the consultant's three points as design principles: an anonymous Grafana is a Viewer and Prometheus is never exposed, labels take bounded values only, and p95 comes from histogram_quantile over buckets placed around the measured latency. The earlier items (backups, outside monitoring, the capacity decision, the runbook, dated maintenance) keep their numbers K1 to K9. Alert rules were in the earlier plan and are out of scope for the consultant: they stay, last and marked as awaiting a decision. Checking the code against the cardinality point found finding #90: the HTTP metrics' method label takes any value. Locally 25 made-up methods opened 25 new series, and in production Traefik passed a made-up method through to the gateway. It is fixed first (G3), before Prometheus scrapes production. Refs: Bulgu #90 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AlpNuhoglu
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New
docs/deployment-phase2-plan.md(Turkish, same structure as phase 1).Part G — the consultant's scope (K10–K15): Prometheus in the cluster with all 7 targets up; application metrics (request counter, latency histogram, error counter, active WebSocket gauge) with bounded labels and buckets around the measured latency; Grafana with 4 panels (req/s, p95/p99, error %, active connections) as code; a public read-only dashboard at status.gamemesh.app (anonymous Viewer, no admin, Prometheus never exposed); a k6 run that moves the panels, with a screenshot; out-of-scope items as a README roadmap. The consultant's three points (security, cardinality, histograms) are the design principles.
Kept from the earlier plan (K1–K9): daily encrypted offsite Postgres backups, a restore drill with RPO/RTO, alerts for stale backups, k3s state backed up apart from its key (#80), outside monitoring of the endpoint and certificate, the capacity decision, the runbook, dated maintenance. Alert rules (Alertmanager) were in the earlier plan but are out of the consultant's scope: kept last, marked as awaiting a decision.
Finding #90 (measured): the HTTP metrics'
methodlabel is unbounded — 25 made-up methods opened 25 series locally, and Traefik passes made-up methods to the production gateway. Fixed first (G3), before Prometheus scrapes production.Docs only.
🤖 Generated with Claude Code