fix(observability): report the trace, not the page, in trace_detail - #253
Merged
Merged
Conversation
trace_detail computed services, duration_ms and has_error from the span slice that `limit` admitted, then stated them as facts about the trace. A call with limit=1 against a five-span trace returned "contains 1 spans across 1 services" and named only the service of the one span it happened to return, while the actual error sat three levels down in a service the page omitted. The failure is silent and the output reads as authoritative, so a caller has no reason to doubt it. Those fields now come from a single-row aggregate over the whole trace. spans and services still describe the page; span_count and service_count describe the trace; truncated says whether the two differ, and the summary gains "; showing N" when they do. The aggregate counts in DuckDB and returns one row, so it does not materialise the trace in this process. The aggregate reads the BIGINT start_unix_nano/end_unix_nano columns rather than start_time/end_time: those are TIMESTAMP, and subtracting them yields an INTERVAL that DuckDB will not divide. A DuckDB-backed test covers the query directly, because the sqlmock tests never execute the SQL and cannot catch that class of error.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
trace_detailderivedservices,duration_msandhas_errorfrom the span slice thatlimitadmitted, then presented them as facts about the trace. Nothing in the response indicated that a page had been returned rather than the whole trace.Reproduced against a live instance. The same trace ID, two calls:
limit: 1limit: 500The truncated answer is not merely incomplete, it misattributes the fault. At
limit: 1the returned span is cart's client span —STATUS_CODE_ERROR, nostatus_message— with cart as the only service named, so a reader concludes cart is broken. The real fault is three levels down:{"service":"flagd","operation":"resolveBoolean","status":"STATUS_CODE_ERROR", "status_message":"error evaluating flag with key failedReadinessProbe"}duration_mssurvived that example only because the root span happened to be the longest. A trace whose longest span falls outside the page reports a wrong duration too.This matters most for model callers: the summary is assertive, carries no truncation signal, and there was no field in the payload that could contradict it.
Change
servicesandspanscontinue to describe the page. Three fields now describe the trace:span_count— spans in the traceservice_count— services the trace crossestruncated— whether the page is narrower than the traceduration_msandhas_errornow come from a single-row aggregate over the whole trace instead of the page. The summary gains; showing Nwhen truncated:The aggregate counts inside DuckDB and returns one row, so it does not materialise the trace in the process.
Note on the aggregate's columns
It reads the BIGINT
start_unix_nano/end_unix_nanocolumns, notstart_time/end_time. The latter areTIMESTAMP(internal/query/views.go:19-20); subtracting them yields anINTERVAL, and DuckDB rejects dividing that by a float:The first version of this change had exactly that defect and would have failed on every
trace_detailcall. The existing sqlmock tests could not catch it because they never execute the SQL, so this addsTestTraceSummaryQueryComputesDurationInMilliseconds, which runs the query against a real in-memory DuckDB.Tests
TestTracePagedSpansStillDescribeTheWholeTrace— three spans across two services, error only in the span the page omits,limit: 1. Asserts the page stays bounded while the counts,has_errorand the summary describe the trace. This is the bug above, as a test.TestTraceWholeTraceIsNotReportedAsTruncated— an untruncated trace must not carry the qualifier.TestTraceSummaryQueryComputesDurationInMilliseconds— DuckDB-backed, covers the aggregate's own arithmetic.Six existing trace tests needed the new query expectation; each one's values were set to match its own fixture and assertions rather than relaxing them.
just checkpasses.Contract
ui/contracts.tsmirrors the Go struct, ascontracts.gorequires. No consumer changes were needed — both trace widgets readspans.