Repository navigation
fix: the five P0 correctness findings from the demo evaluation (#232) - #233
Merged
Merged
Conversation
…ed with The service performance card showed Operations, P95 and Error rate taken from points.at(-1) — the newest five-minute bucket — while its own footer labelled the card with the whole window. On demo.fanout.run the frontend card read "1,281 operations, P95 48.2ms, healthy" for a window that actually held 15,283 spans at a P95 of 13.7s, which the card's own Compare tab printed correctly one tab away. The newest bucket is also still filling when it is read, so the headline understated both traffic and latency. Performance now carries Totals for the whole scope. The two comparison halves already tile the window, so folding them together reproduces what performanceAggregateQuery returns over the full scope — sum the spans, weight the averages by span count, take the larger P95 — without a third query. Reported in #232 (P0 item 1). Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
overallHealth returned "healthy" whenever no service was unhealthy or degraded — including when no service reported at all. Typing a namespace that matches nothing rendered "Health: Healthy · 0 services · Error rate 0.00%" over an empty distribution bar: the same answer a genuinely clean system gets. The summary the model reads said "0 services: 0 unhealthy, 0 degraded, 0 healthy", which reads as a pass rather than as silence. Health gains an "unknown" state for a window with nothing in it. The overview summary says so in words, the dashboard tile reads "No data" with a neutral colour, and the rate and distribution — neither of which has anything to describe — are withheld rather than shown as zero. Reported in #232 (P0 item 2). Stacked on fix/performance-window-totals. Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
normalizeSeverity upper-cased SeverityText and returned it, so the store held INFO, INFORMATION and "" for the same level: .NET writes "Information", Envoy's access logs carry no severity at all, and each spelling answered a different query. On the demo that left zero ERROR or WARN logs across a six-hour window in which the platform reported a 1.77% error rate — the 504s were in Envoy's unlabelled access logs, which no severity filter could reach. severity_number now decides when it is set, since the specification makes it authoritative and the text a display form. Text is mapped otherwise, covering OTel's numbered short names, the syslog names and the long forms from .NET and Python; an unrecognised word is kept, since it is the only grading the record carries. A record with neither is stored as UNSPECIFIED, which the log-bucket query already labels that way — a level visible in the chart should be one you can filter on. This corrects what is written from here on; rows already in Parquet keep the spelling they arrived with. Reported in #232 (P0 item 3). Stacked on fix/empty-window-not-healthy. Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
TraceWidget forwarded only trace_id, so a widget stored with
{"service": "checkout"} — which is what the dashboard generator writes, and
what /api/observability/trace accepts — asked for the worst trace anywhere in
the window instead. The generated "Slow PlaceOrder Trace" card on the demo
showed a two-span flagd evaluation between recommendation and flagd: the
longest trace in the window, and nothing to do with checkout.
The service reaches the query now, and the configure dialog offers it for trace
widgets so a stored scope is visible and editable rather than silently dropped.
Reported in #232 (P0 item 4). Stacked on fix/canonical-log-severity.
Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
service_rollup took p50 and p95 over every span a service emitted, so a subscription held open by design set the service's latency. On the demo, fraud-detection, product-reviews, ad and recommendation each carry a CLIENT span of exactly 600s — their subscription to flagd's EventStream — and each was marked unhealthy for it while serving requests in single-digit milliseconds. Nine of twenty services were red, which is also why the demo's headline read "Unhealthy" continuously. Latency percentiles now exclude CLIENT and PRODUCER spans, which measure a dependency rather than the service's own work. Error rate still counts every span: a failing outbound call is the caller's problem too. A service that only makes outbound calls — a load generator, a cron worker — falls back to all of its spans rather than reporting no latency at all. Genuinely slow work is untouched: frontend's 103s POST /api/checkout is a SERVER span and still counts. flagd holds 600s server-side event streams and will stay unhealthy until streaming spans are classified separately; that is a narrower problem than this one and is left for its own change. Buckets already rolled up keep their old values; new buckets use the new definition. Reported in #232 (P0 item 5). Stacked on fix/trace-widget-service-scope. Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
5 tasks done
Excluding CLIENT and PRODUCER spans inside each bucket was not enough on its own. A minute in which a held-open subscription is the only span still has to report something, so it fell back to that subscription — and since a service's window P95 is the maximum across its buckets, that one minute decided the whole window anyway. Running the change against a local build showed reviews still reporting 600000ms and still marked unhealthy, which is the symptom this was supposed to remove. service_rollup now records served_spans, the count of spans describing work the service performed. Readers prefer the buckets that have some, and fall back to the rest only when a service served nothing all window — a load generator keeps its 250ms rather than reporting a flat zero, which would read as healthy for the same reason an empty window used to. Verified against a local instance: reviews now reports 48ms and healthy while holding three ten-minute subscriptions, load-generator keeps 250ms, and checkout's genuinely slow 9s served request still marks it unhealthy. Reported in #232 (P0 item 5). Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
5 tasks done
totalsOf took the larger P95 of the two comparison halves and weighted P50 across both. A half spent holding a subscription open has no served latency, so its fallback figure would decide the card — putting the card back in disagreement with the overview beside it, which reads the whole window and prefers the buckets that served something. That disagreement is what these totals exist to end. Latency now comes from the halves with served spans, falling back to both when a service served nothing all window. Counts and error rate still cover every span: an operation the service waited on is still an operation it performed, and a failed outbound call is still its problem. Found while re-reading the diff before merge. Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
Adding served_spans made ensureCacheTable drop and recreate service_rollup on upgrade, but the rollup only rebuilds what its watermark says is missing. The stored watermark still pointed past every dropped bucket, so an existing installation would have come back with an empty service_rollup that never refilled, and the overview, performance, heatmap and alert queries would all have reported no data for that history — permanent silent loss on upgrade, not the "buckets keep their old percentiles" this branch claimed. Recreating a cache table now discards the progress keys named for it, and the service rollup's key moves to v3 because the meaning of its p50 and p95 changed along with its shape. The regression test walks the upgrade: roll up, drop the column an older build lacked, and require the bucket to come back. Also from review: fold the card's P50 with the same served-span weight the window query uses, weight windowP50SQL by served_spans rather than by every span in the bucket, document served_spans and the new latency semantics in the LLM-facing schema, and delete the doc comment above normalizeSeverity that described the behaviour this branch replaced. Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
Both changes touch internal/ingest/server_test.go at its end; the resolution keeps each test. Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
This was referenced Sep 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
The five P0 items from #232 — the findings from driving demo.fanout.run as a user, with every number verified against the API. Each is a separate commit and stands on its own.
1. Headline metrics cover the window they are labelled with (
90652a8e)The service card read Operations, P95 and Error rate off
points.at(-1)— the newest five-minute bucket, still filling — while its footer said "Last 1h". The frontend card showed "1,281 operations · P95 48.2ms · healthy" for a window holding 15,283 spans at a P95 of 13.7s, which the card's own Compare tab printed correctly one tab away.Performancenow carries windowTotals, folded from the two comparison halves the Compare tab already fetches, so no extra query runs.2. An empty window reports no data, not health (
b6523181)overallHealthreturned "healthy" whenever nothing was unhealthy or degraded, including when nothing reported at all. A namespace filter matching nothing rendered "Health: Healthy · 0 services · 0.00%".Healthgains anunknownstate; the tile reads "No data" in a neutral colour, the rate and distribution are withheld, and the summary the model reads says the window is empty.3. Log severity is canonical (
9526ba79)normalizeSeverityupper-casedSeverityTextand returned it, so the store heldINFO,INFORMATIONand""for one level — .NET writes "Information", Envoy's access logs carry no severity. Six hours of demo telemetry contained zero ERROR or WARN logs while the platform reported a 1.77% error rate; the 504s were in Envoy's unlabelled lines, which no filter could reach.severity_numbernow decides when set, text is mapped otherwise, and a record with neither is stored asUNSPECIFIED— the value the log-bucket query already displays.4. The trace widget honours its service (
3bd7c31a)TraceWidgetforwarded onlytrace_id, so a widget stored with{"service": "checkout"}— what the dashboard generator writes and what the endpoint accepts — asked for the worst trace in the window instead. The generated "Slow PlaceOrder Trace" card showed a two-span flagd event stream. The service reaches the query now, and the configure dialog exposes it.5. A service is graded on its own work (
9107929b)service_rolluptook percentiles over every span a service emitted. fraud-detection, product-reviews, ad and recommendation each hold a 600s CLIENT span — their subscription to flagd'sEventStream— and were marked unhealthy for it while serving requests in single-digit milliseconds. Nine of twenty services were red, which is why the demo's headline read "Unhealthy" continuously. Percentiles now exclude CLIENT and PRODUCER spans; error rate still counts every span, since a failing outbound call is the caller's problem too. A service that only calls out falls back to all of its spans rather than reporting no latency.Genuinely slow work is untouched: frontend's 103s
POST /api/checkoutis a SERVER span and still counts. flagd serves 600s event streams and stays unhealthy until streaming spans are classified separately — a narrower problem, left for its own change.Notes
Items 3 and 5 change what is written from here on. Rows already in Parquet keep the spelling they arrived with, and buckets already rolled up keep their old percentiles.
Verification
just checkjust test-racewhen auth, API, ingest, query, MCP, or agent paths changedNew tests cover each behaviour: window totals against the newest bucket, an empty overview reporting
unknown, severity spellings from four emitters, the trace widget's request carrying its service, and rollup percentiles ignoring a held-open subscription while still measuring a call-only service.Contract changes to note:
Performancegainstotals, andHealthgainsunknown— both mirrored inui/contracts.ts.Closes the P0 section of #232.