Repository navigation
fix(metrics): bounded labels, WebSocket sessions, 5xx errors, fitted buckets - #91
Merged
Merged
Conversation
The method label of the HTTP request metrics was the client's method token as sent. Any token passes HTTP parsing and Traefik forwards it, so each made-up method opened new series: in production `curl -X ZZCHECK1` reached the gateway (404, "method":"ZZCHECK1" in its log), and every such request adds one counter series per metric plus 14 histogram series, forever. GET, POST, PUT, PATCH, DELETE, HEAD and OPTIONS stay as they are; any other token is recorded as "other". Paths were already bounded (route template or "unmatched"); status codes come from our own handlers. Measured with the new test: 100 made-up methods and 50 random paths gave 102 request, 101 error and 102 histogram series before, 3, 2 and 3 after. Plan: G3 Refs: Bulgu #90 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A WebSocket handshake's handler returns only when the session ends in the gateway (httputil.ReverseProxy blocks while it relays), so the request latency histogram recorded each session's lifetime. Locally, 27 sessions through the gateway all landed in +Inf (1151 s in total); with enough sessions, p99 over all routes would read 5 s, the largest finite bucket. Both services also counted the handshake as 200: the 101 is written on the hijacked connection, past gin's status. Metrics now notices a hijacked connection: such a request is counted with status 101 and is not observed by the histogram. A rejected handshake is an ordinary response and is recorded as before. Checked against the real stack (compose, gateway and websocket rebuilt): 11 k6 sessions through the gateway counted as 101 in both services, no /ws latency series; a handshake without a token counted as 401 with its latency. Plan: G3 Refs: Bulgu #92 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gamemesh_http_errors_total counted every status >= 400, so its rate was not an error rate: 404 is the normal answer to DELETE /queue once a match has dequeued the player, anyone on the internet can produce 401s and 404s (a scanner alone would move it), and 429 is the rate limit doing its job. The counter also only repeated what requests_total already holds by status. It now counts 5xx, including the gateway's 502/503/504 when an upstream is down, which users do see. 4xx stay visible in requests_total by status. The compose dashboard's error panel keeps its query and now shows the server error rate; its title says so. Plan: G3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The request latency histogram used Prometheus's default buckets (1, 5, 10, 25, 50, 100, 250 ms, ...). Measured server-side at the gateway (2026-10-22): idle in production p95 2-3 ms; locally under the README's matchmaking load (500 VUs) p50 3.2, p95 16.2, p99 30.2 ms over the non-auth routes, and login/register at 200-240 ms (bcrypt). p95 and p99 fell in the 10-25 and 25-50 ms buckets, where histogram_quantile interpolates linearly. New bounds: 1, 2.5, 5, 10, 15, 20, 30, 50, 75, 100, 150, 200, 250, 300, 500 ms, 1, 2.5, 5 s. Replaying the same 19,000 requests through histogram_quantile: p50 +21 % -> +7 %, p95 +29 % -> +3 %, p99 +18 % -> +1 % against the true values. Series: 18 bounds instead of 11, so 21 series per method and route instead of 14. Upper bound over all services with the method label bounded: 91 method/route pairs, 1911 series. Plan: G3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2, step G3: make the HTTP request metrics safe and meaningful before Prometheus scrapes production.
Changes
methodlabel (finding docs(plan): move alert rules to phase 3 #90). The method was the client's token as sent; Traefik forwards any token, and each one opened new series (curl -X ZZCHECK1reached the production gateway). GET, POST, PUT, PATCH, DELETE, HEAD and OPTIONS are kept; anything else isother. Test: 100 made-up methods plus 50 random paths opened 102 / 101 / 102 request / error / histogram series before, 3 / 2 / 3 after.+Inf, 1151 s in total. Both services also counted the handshake as 200. A hijacked request is now counted as101and not observed by the histogram. Checked against the real stack: 11 k6 sessions counted as101in gateway and websocket, no/wslatency series; a handshake without a token is still a 401 with its latency.gamemesh_http_errors_totalcounts 5xx only. 404 is the normal answer toDELETE /queueafter a match, anyone can produce 401/404, and 429 is the rate limit working. 4xx stay visible inrequests_totalby status. The compose dashboard's error panel keeps its query; its title now says "5xx".histogram_quantileagainst the true values: p95 error +29 % → +3 %, p99 +18 % → +1 %.Cardinality
21 series per method/route pair (was 14). Upper bound over all services with the method bounded: 91 pairs, 1911 histogram series; before this, unbounded.
Not in this PR
The plan's result note and the index rows for findings #91 and #92 follow in a
docs(plan)PR, after production is measured (a made-up method must not add series).Plan: G3 · Refs: Bulgu #90, #92
🤖 Generated with Claude Code