Skip to content

fix(observability): report the trace, not the page, in trace_detail - #253

Merged
vishr merged 1 commit into
mainfrom
fix/trace-detail-truncated-trace
Sep 20, 2026
Merged

vishr merged 1 commit into
mainfrom
fix/trace-detail-truncated-trace

Conversation

@vishr

@vishr vishr commented Sep 20, 2026

Copy link
Copy Markdown
Member

Problem

trace_detail derived services, duration_ms and has_error from the span slice that limit admitted, then presented them as facts about the trace. Nothing in the response indicated that a page had been returned rather than the whole trace.

Reproduced against a live instance. The same trace ID, two calls:

limit: 1

"services": ["cart"],
"summary": "Trace b9ec1ae558572a610bb870fd3b2b6a92 contains 1 spans across 1 services"

limit: 500

"services": ["cart","flagd"],
"summary": "Trace b9ec1ae558572a610bb870fd3b2b6a92 contains 5 spans across 2 services"

The truncated answer is not merely incomplete, it misattributes the fault. At limit: 1 the returned span is cart's client span — STATUS_CODE_ERROR, no status_message — with cart as the only service named, so a reader concludes cart is broken. The real fault is three levels down:

{"service":"flagd","operation":"resolveBoolean","status":"STATUS_CODE_ERROR",
 "status_message":"error evaluating flag with key failedReadinessProbe"}

duration_ms survived that example only because the root span happened to be the longest. A trace whose longest span falls outside the page reports a wrong duration too.

This matters most for model callers: the summary is assertive, carries no truncation signal, and there was no field in the payload that could contradict it.

Change

services and spans continue to describe the page. Three fields now describe the trace:

  • span_count — spans in the trace
  • service_count — services the trace crosses
  • truncated — whether the page is narrower than the trace

duration_ms and has_error now come from a single-row aggregate over the whole trace instead of the page. The summary gains ; showing N when truncated:

Trace b9ec1ae5… contains 5 spans across 2 services; showing 1

The aggregate counts inside DuckDB and returns one row, so it does not materialise the trace in the process.

Note on the aggregate's columns

It reads the BIGINT start_unix_nano / end_unix_nano columns, not start_time / end_time. The latter are TIMESTAMP (internal/query/views.go:19-20); subtracting them yields an INTERVAL, and DuckDB rejects dividing that by a float:

Binder Error: Cannot mix values of type INTERVAL and INTEGER_LITERAL in COALESCE operator

The first version of this change had exactly that defect and would have failed on every trace_detail call. The existing sqlmock tests could not catch it because they never execute the SQL, so this adds TestTraceSummaryQueryComputesDurationInMilliseconds, which runs the query against a real in-memory DuckDB.

Tests

  • TestTracePagedSpansStillDescribeTheWholeTrace — three spans across two services, error only in the span the page omits, limit: 1. Asserts the page stays bounded while the counts, has_error and the summary describe the trace. This is the bug above, as a test.
  • TestTraceWholeTraceIsNotReportedAsTruncated — an untruncated trace must not carry the qualifier.
  • TestTraceSummaryQueryComputesDurationInMilliseconds — DuckDB-backed, covers the aggregate's own arithmetic.

Six existing trace tests needed the new query expectation; each one's values were set to match its own fixture and assertions rather than relaxing them.

just check passes.

Contract

ui/contracts.ts mirrors the Go struct, as contracts.go requires. No consumer changes were needed — both trace widgets read spans.

trace_detail computed services, duration_ms and has_error from the span
slice that `limit` admitted, then stated them as facts about the trace. A
call with limit=1 against a five-span trace returned "contains 1 spans
across 1 services" and named only the service of the one span it happened
to return, while the actual error sat three levels down in a service the
page omitted. The failure is silent and the output reads as authoritative,
so a caller has no reason to doubt it.

Those fields now come from a single-row aggregate over the whole trace.
spans and services still describe the page; span_count and service_count
describe the trace; truncated says whether the two differ, and the summary
gains "; showing N" when they do. The aggregate counts in DuckDB and
returns one row, so it does not materialise the trace in this process.

The aggregate reads the BIGINT start_unix_nano/end_unix_nano columns
rather than start_time/end_time: those are TIMESTAMP, and subtracting
them yields an INTERVAL that DuckDB will not divide. A DuckDB-backed test
covers the query directly, because the sqlmock tests never execute the SQL
and cannot catch that class of error.
@vishr
vishr merged commit e10f44d into main Sep 20, 2026
8 checks passed
@vishr
vishr deleted the fix/trace-detail-truncated-trace branch September 20, 2026 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant