Skip to content

fix: the five P0 correctness findings from the demo evaluation (#232) - #233

Merged
vishr merged 9 commits into
mainfrom
fix/p0-demo-findings
Sep 7, 2026
Merged

vishr merged 9 commits into
mainfrom
fix/p0-demo-findings

Conversation

@vishr

@vishr vishr commented Sep 7, 2026 •

Copy link
Copy Markdown
Member

What changed

The five P0 items from #232 — the findings from driving demo.fanout.run as a user, with every number verified against the API. Each is a separate commit and stands on its own.

1. Headline metrics cover the window they are labelled with (90652a8e)

The service card read Operations, P95 and Error rate off points.at(-1) — the newest five-minute bucket, still filling — while its footer said "Last 1h". The frontend card showed "1,281 operations · P95 48.2ms · healthy" for a window holding 15,283 spans at a P95 of 13.7s, which the card's own Compare tab printed correctly one tab away. Performance now carries window Totals, folded from the two comparison halves the Compare tab already fetches, so no extra query runs.

2. An empty window reports no data, not health (b6523181)

overallHealth returned "healthy" whenever nothing was unhealthy or degraded, including when nothing reported at all. A namespace filter matching nothing rendered "Health: Healthy · 0 services · 0.00%". Health gains an unknown state; the tile reads "No data" in a neutral colour, the rate and distribution are withheld, and the summary the model reads says the window is empty.

3. Log severity is canonical (9526ba79)

normalizeSeverity upper-cased SeverityText and returned it, so the store held INFO, INFORMATION and "" for one level — .NET writes "Information", Envoy's access logs carry no severity. Six hours of demo telemetry contained zero ERROR or WARN logs while the platform reported a 1.77% error rate; the 504s were in Envoy's unlabelled lines, which no filter could reach. severity_number now decides when set, text is mapped otherwise, and a record with neither is stored as UNSPECIFIED — the value the log-bucket query already displays.

4. The trace widget honours its service (3bd7c31a)

TraceWidget forwarded only trace_id, so a widget stored with {"service": "checkout"} — what the dashboard generator writes and what the endpoint accepts — asked for the worst trace in the window instead. The generated "Slow PlaceOrder Trace" card showed a two-span flagd event stream. The service reaches the query now, and the configure dialog exposes it.

5. A service is graded on its own work (9107929b)

service_rollup took percentiles over every span a service emitted. fraud-detection, product-reviews, ad and recommendation each hold a 600s CLIENT span — their subscription to flagd's EventStream — and were marked unhealthy for it while serving requests in single-digit milliseconds. Nine of twenty services were red, which is why the demo's headline read "Unhealthy" continuously. Percentiles now exclude CLIENT and PRODUCER spans; error rate still counts every span, since a failing outbound call is the caller's problem too. A service that only calls out falls back to all of its spans rather than reporting no latency.

Genuinely slow work is untouched: frontend's 103s POST /api/checkout is a SERVER span and still counts. flagd serves 600s event streams and stays unhealthy until streaming spans are classified separately — a narrower problem, left for its own change.

Notes

Items 3 and 5 change what is written from here on. Rows already in Parquet keep the spelling they arrived with, and buckets already rolled up keep their old percentiles.

Verification

  • just check
  • just test-race when auth, API, ingest, query, MCP, or agent paths changed
  • User-facing behavior and configuration docs are current
  • No credentials, private telemetry, host details, or enterprise-only source are included
  • API, migration, ingest, MCP/AG-UI, or release-contract changes are called out

New tests cover each behaviour: window totals against the newest bucket, an empty overview reporting unknown, severity spellings from four emitters, the trace widget's request carrying its service, and rollup percentiles ignoring a held-open subscription while still measuring a call-only service.

Contract changes to note: Performance gains totals, and Health gains unknown — both mirrored in ui/contracts.ts.

Closes the P0 section of #232.

…ed with

The service performance card showed Operations, P95 and Error rate taken from
points.at(-1) — the newest five-minute bucket — while its own footer labelled
the card with the whole window. On demo.fanout.run the frontend card read
"1,281 operations, P95 48.2ms, healthy" for a window that actually held 15,283
spans at a P95 of 13.7s, which the card's own Compare tab printed correctly one
tab away. The newest bucket is also still filling when it is read, so the
headline understated both traffic and latency.

Performance now carries Totals for the whole scope. The two comparison halves
already tile the window, so folding them together reproduces what
performanceAggregateQuery returns over the full scope — sum the spans, weight
the averages by span count, take the larger P95 — without a third query.

Reported in #232 (P0 item 1).

Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
overallHealth returned "healthy" whenever no service was unhealthy or
degraded — including when no service reported at all. Typing a namespace that
matches nothing rendered "Health: Healthy · 0 services · Error rate 0.00%" over
an empty distribution bar: the same answer a genuinely clean system gets. The
summary the model reads said "0 services: 0 unhealthy, 0 degraded, 0 healthy",
which reads as a pass rather than as silence.

Health gains an "unknown" state for a window with nothing in it. The overview
summary says so in words, the dashboard tile reads "No data" with a neutral
colour, and the rate and distribution — neither of which has anything to
describe — are withheld rather than shown as zero.

Reported in #232 (P0 item 2). Stacked on fix/performance-window-totals.

Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
normalizeSeverity upper-cased SeverityText and returned it, so the store held
INFO, INFORMATION and "" for the same level: .NET writes "Information", Envoy's
access logs carry no severity at all, and each spelling answered a different
query. On the demo that left zero ERROR or WARN logs across a six-hour window
in which the platform reported a 1.77% error rate — the 504s were in Envoy's
unlabelled access logs, which no severity filter could reach.

severity_number now decides when it is set, since the specification makes it
authoritative and the text a display form. Text is mapped otherwise, covering
OTel's numbered short names, the syslog names and the long forms from .NET and
Python; an unrecognised word is kept, since it is the only grading the record
carries. A record with neither is stored as UNSPECIFIED, which the log-bucket
query already labels that way — a level visible in the chart should be one you
can filter on.

This corrects what is written from here on; rows already in Parquet keep the
spelling they arrived with.

Reported in #232 (P0 item 3). Stacked on fix/empty-window-not-healthy.

Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
TraceWidget forwarded only trace_id, so a widget stored with
{"service": "checkout"} — which is what the dashboard generator writes, and
what /api/observability/trace accepts — asked for the worst trace anywhere in
the window instead. The generated "Slow PlaceOrder Trace" card on the demo
showed a two-span flagd evaluation between recommendation and flagd: the
longest trace in the window, and nothing to do with checkout.

The service reaches the query now, and the configure dialog offers it for trace
widgets so a stored scope is visible and editable rather than silently dropped.

Reported in #232 (P0 item 4). Stacked on fix/canonical-log-severity.

Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
service_rollup took p50 and p95 over every span a service emitted, so a
subscription held open by design set the service's latency. On the demo,
fraud-detection, product-reviews, ad and recommendation each carry a CLIENT
span of exactly 600s — their subscription to flagd's EventStream — and each was
marked unhealthy for it while serving requests in single-digit milliseconds.
Nine of twenty services were red, which is also why the demo's headline read
"Unhealthy" continuously.

Latency percentiles now exclude CLIENT and PRODUCER spans, which measure a
dependency rather than the service's own work. Error rate still counts every
span: a failing outbound call is the caller's problem too. A service that only
makes outbound calls — a load generator, a cron worker — falls back to all of
its spans rather than reporting no latency at all.

Genuinely slow work is untouched: frontend's 103s POST /api/checkout is a
SERVER span and still counts. flagd holds 600s server-side event streams and
will stay unhealthy until streaming spans are classified separately; that is a
narrower problem than this one and is left for its own change.

Buckets already rolled up keep their old values; new buckets use the new
definition.

Reported in #232 (P0 item 5). Stacked on fix/trace-widget-service-scope.

Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
Excluding CLIENT and PRODUCER spans inside each bucket was not enough on its
own. A minute in which a held-open subscription is the only span still has to
report something, so it fell back to that subscription — and since a service's
window P95 is the maximum across its buckets, that one minute decided the whole
window anyway. Running the change against a local build showed reviews still
reporting 600000ms and still marked unhealthy, which is the symptom this was
supposed to remove.

service_rollup now records served_spans, the count of spans describing work the
service performed. Readers prefer the buckets that have some, and fall back to
the rest only when a service served nothing all window — a load generator keeps
its 250ms rather than reporting a flat zero, which would read as healthy for
the same reason an empty window used to.

Verified against a local instance: reviews now reports 48ms and healthy while
holding three ten-minute subscriptions, load-generator keeps 250ms, and
checkout's genuinely slow 9s served request still marks it unhealthy.

Reported in #232 (P0 item 5).

Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
totalsOf took the larger P95 of the two comparison halves and weighted P50
across both. A half spent holding a subscription open has no served latency, so
its fallback figure would decide the card — putting the card back in
disagreement with the overview beside it, which reads the whole window and
prefers the buckets that served something. That disagreement is what these
totals exist to end.

Latency now comes from the halves with served spans, falling back to both when
a service served nothing all window. Counts and error rate still cover every
span: an operation the service waited on is still an operation it performed,
and a failed outbound call is still its problem.

Found while re-reading the diff before merge.

Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
Adding served_spans made ensureCacheTable drop and recreate service_rollup on
upgrade, but the rollup only rebuilds what its watermark says is missing. The
stored watermark still pointed past every dropped bucket, so an existing
installation would have come back with an empty service_rollup that never
refilled, and the overview, performance, heatmap and alert queries would all
have reported no data for that history — permanent silent loss on upgrade, not
the "buckets keep their old percentiles" this branch claimed.

Recreating a cache table now discards the progress keys named for it, and the
service rollup's key moves to v3 because the meaning of its p50 and p95 changed
along with its shape. The regression test walks the upgrade: roll up, drop the
column an older build lacked, and require the bucket to come back.

Also from review: fold the card's P50 with the same served-span weight the
window query uses, weight windowP50SQL by served_spans rather than by every
span in the bucket, document served_spans and the new latency semantics in the
LLM-facing schema, and delete the doc comment above normalizeSeverity that
described the behaviour this branch replaced.

Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
Both changes touch internal/ingest/server_test.go at its end; the resolution
keeps each test.

Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
@vishr
vishr merged commit fc75da7 into main Sep 7, 2026
8 checks passed
@vishr
vishr deleted the fix/p0-demo-findings branch September 7, 2026 03:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant