Skip to content

perf(observability): scan client spans once for external service-map peers - #329

Merged
vishr merged 1 commit into
mainfrom
perf/service-map
Oct 17, 2026
Merged

vishr merged 1 commit into
mainfrom
perf/service-map

Conversation

@vishr

@vishr vishr commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

Summary

The service map's external-peer query reused a clients CTE in two anti-joins. DuckDB inlined it, so each run scanned spans eight times. On an 8-vCPU host running the OpenTelemetry Demo (v2026.10.10), the panel took 6.5 s p50 and 9.9 s p95 across 20 dashboard refreshes, at the edge of the 10-second deadline.

WITH clients AS MATERIALIZED computes the client projection once. External peers still come from raw client spans: edge_rollup has no entries for these external pairs and stores no peer latency quantiles, so it cannot replace them.

Evidence

One-hour Demo snapshot, frozen window:

Before After
Parquet scans 8 1
Bytes read (profile) 366 MB 80 MB
Cold panel 544 ms 321 ms
Warm (3 runs) 423 / 498 / 435 ms 155 / 129 / 142 ms

All 45 frame rows match in order; average latency differs only by floating-point roundoff (2e-14 ms). A new regression test covers peers, rates, latencies, scopes and ordering.

Verification

  • go test ./internal/observability/... ./internal/panel/...: passed.
  • just check (pre-push): passed.

…peers

The service map's external-peer query reused a clients CTE in two
anti-joins. DuckDB inlined it, so each run scanned spans eight times
(366 MB on a one-hour OpenTelemetry Demo snapshot). On an 8-vCPU Demo host
the panel took 6.5 s at the median and 9.9 s at p95 during dashboard
refreshes, at the edge of the 10-second deadline.

Materialize the client projection once. On the snapshot the plan has one
Parquet scan reading 80 MB, and the warm panel time falls from about 450
to 140 ms. All 45 frame rows match in order.

Claude-Session: https://claude.ai/code/session_01XA52nWyQ3R5f5kK8kjpAQW
@vishr
vishr merged commit dd72cf5 into main Oct 17, 2026
14 checks passed
@vishr
vishr deleted the perf/service-map branch October 17, 2026 04:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant