feat(observability): Datadog observability — metrics, readiness, log export, dashboards and alerts (CHOO-2807)

[christian-mcdermott]

Operational observability for switch-core: what it measures, how that leaves the deployment, and what to look at when something breaks.

Jira: CHOO-2807

What was already there, and what wasn't

Structured logging landed earlier — JSON keyed for Datadog, with tenant, request, agent and user stamped on every line. That part needed nothing.

Everything else was missing, and two things were worse than missing:

• /health always answered ok without checking anything, and Helm used it for both probes. A pod with an unreachable database reported healthy.
• Nothing collects the container's output in this cluster. There is no log agent, so the Datadog-shaped JSON reached the container and stopped.

Approach

Hand-built OTLP over httpx, following Switch Console's telemetry client rather than pulling in the OpenTelemetry SDK. Everything a deployment sends is visible in one file, which matters for a product self-hosted by people entitled to read exactly that. The cost is owning the wire format, so protobuf JSON's 64-bit-integers-as-strings rule is tested directly — a strict receiver rejects a payload that gets it wrong and the relay answers 200 either way.

Everything is off until OTLP_ENDPOINT names a collector. Readiness is the exception: it runs regardless, because how Kubernetes routes traffic cannot depend on whether anyone is collecting metrics.

The parts

A metric catalogue. Nothing is emitted that is not declared in observability/catalogue.py, attributes included, and the registry rejects the rest. On a multi-tenant server an unchecked label is two problems at once: a cardinality bill, and a disclosure — dashboards are read by people who are not entitled to a given tenant's rows. Every attribute value comes from a set the code controls; the HTTP route is a template, the platform comes off the bridge, and send_event's arbitrary event type is classified into four values rather than passed through.

Readiness, deliberately narrow. /health stays the cheap always-ok reply — besides liveness, the gateway Deployment and the setup Job wait on it at boot, so anything it checked would become a boot-ordering dependency for them. /health/ready is new, and only the database gates it. switch-core is pinned to one replica with Recreate, so failing readiness does not shift traffic to a healthy pod — it empties the Service. A crashed Slack adapter is reported and alerted on, and must not take Switch off the air.

Instrumentation at the choke points that already exist rather than per handler: the bridge's inbound tracer, the transport's send and deliver funnels, one ASGI middleware. Failures that were previously invisible are now counted — the delivery loop swallows a room's error by design so one bad room cannot stop the others, which is also what made a room that had stopped delivering leave nothing but a log line.

Delivery lag measured per message, as the age of a row when it reached a handler, rather than inferred from a queue depth.

The process reports on itself — memory, CPU, descriptors, GC — because there is no infrastructure agent here. Event-loop lag is the one no agent could produce: the server is single-threaded, so one blocking call stalls every room and bridge at once. The connection sweep already measured its own oversleep and threw it away.

Logs shipped as well as written. stderr is never replaced, so kubectl logs keeps working and a collector outage costs a copy rather than the record. Off by default — a second copy over the network is a volume decision that belongs to whoever pays for it.

Dashboard and nine monitors as JSON in deploy/observability/, so a change to a metric and a change to the panel reading it land in the same review. No account, key or handle in them. The alert messages say what the alert means for Switch rather than restating the threshold; three things are deliberately not alerted on and the README says why, latency first — Switch holds long polls open on purpose, so a duration threshold pages on healthy traffic.

What is not here

Tracing. The relay Switch reports to does not serve /v1/traces — I probed it, it 404s. The flag exists and defaults off, because enabling it today means every export failing. Two things have to happen: the collector accepts the signal, and the server produces spans. Log records already carry traceId/spanId where set, so correlation needs no further change to the log path. Flagged in the room; awaiting a decision on whether to build it dark.

Pod and node metrics. Container restarts, OOM kills, evictions, Datadog's Kubernetes views — the process cannot see those. That needs the Datadog agent or the k8s integration, and is infrastructure work nothing here depends on.

Verification

• Full suite green: 3371 passed. ruff clean; mypy unchanged from main (36 pre-existing).
• Helm rendered across every documented combination, including all three invalid ones, which abort the release rather than producing a pod that reports nothing.
• End to end against the real relay: a metrics payload carrying a sum, a gauge and a histogram is accepted, and so is a log batch carrying an exception with its stack.

Review found two bugs, both fixed

Both the same mistake — a prefix match without a boundary — and both silent when they fire.

route_label dropped the mount prefix for an inner route whose name merely starts with the mount's string; reproduced /gatewayish under /gateway being labelled /gatewayish, which is exactly the collision the label exists to prevent. Nothing routes that way today.

The log handler's self-exclusion did nothing about the HTTP client underneath the exporter. httpcore logs per connection at DEBUG, so with log shipping on at that level, each export manufactured the records the next export had to send. Now anything logged inside an export's own window is dropped, scoped by context variable so a concurrent request still ships.

Notes for the reviewer

• docs/old/observability.md is the orientation; observability/catalogue.py is the reference for what exists.
• One guardrail in this repo earned its keep: the tenant-exemption allowlist failed the build the moment the readiness check opened an unbound session. The SELECT 1 is deliberately unbound — the database's health is not a question about one customer's rows — so it is allowlisted with that reasoning.
• No version numbers touched.

🤖 Generated with Claude Code