Skip to content

Commit 403ff69

Browse files
committed
perf: bound the two statements whose cost grew with the data
Measurement said the stall was pool contention, so the first instinct was to make the pool bigger. That was wrong: the requests waiting for a connection were waiting behind two statements doing far more work than they needed. Log redaction ran on every row in the window. Four chained regexp_replace sat in the same SELECT as ORDER BY time DESC LIMIT, and a projection there is evaluated below the top-N — so returning a hundred entries redacted every row the window held. Measured at 140ms for 100 rows out of 372,000 against 8ms once the top-N runs first, and independently at 449ms against 11ms over 400,000 rows with more realistic bodies — the plan confirming the projection moved above the top-N and EXCEPT ALL confirming the rows are identical either way. It was the one statement whose cost grew with retention rather than with the answer — with no search term; a search still has to redact every row it tests, because it must match the text the reader is shown rather than the secret underneath it. The endpoint boundary scan read the whole window. Its bounds came from a CTE joined against the scan, and a bound derived from a joined row is not a constant DuckDB can push into a Parquet scan — so a query that wants the two partial minutes at the window's edges read everything between them: 799,333 rows of 800,000 to return 7,333, and 240,000 to return 667 in a smaller independent reproduction. The bounds are computed in Go and bound as parameters now; the same queries scan 11,333 and 35,200. Dashboard queries get a twenty-second deadline. The SQL API has always had one; these endpoints, which every widget calls, had none, so a pathological query held its connection until the client gave up. A query that exceeds it answers 503: running out of time under load is not an internal error, and reporting it as one logs at ERROR and counts towards the 5xx rate that pages someone. And the half of the picture the last commit lacked: how long a statement takes to execute. The connection is acquired before the timer starts, because DB.QueryContext would fold the wait for a free connection into the statement's duration — and telling queueing from execution is the entire reason to measure it. A test holds the pool's only connection to prove the two do not overlap. Measured across the same seed and load — 96 batches of 4,000 spans over ten hours, continuous ingest, eight concurrent clients, 150s, four-connection pool in both arms: req p50 p95 p99 cheap p99 before 11,970 45ms 275ms 498ms 178ms after 15,642 31ms 191ms 358ms 122ms Throughput up 31%, p95 down 31%. The max column is deliberately absent: with a closed-loop generator one server-side stall appears as eight slow requests, so a single run cannot separate a shifted distribution from one bad event. No pool-sizing change. An earlier version of this commit rewrote resolveDuckDBMaxConns, and the arm that appeared to justify it compared a hand-pinned pool of four against an auto-sized seven — which is not what the rule would have done on any host: unchanged on the four-core reference host, and a reduction on larger ones that nothing here measured. The evidence says a cheap query should not queue behind an expensive one; it does not say what the pool should be. The statement histogram and the pool gauges are what should answer that, on a real host. Claude-Session: https://claude.ai/code/session_017ovmTezqzvxVP8P9MGDQNk
1 parent 8b65703 commit 403ff69

9 files changed

Lines changed: 429 additions & 53 deletions

File tree

‎internal/api/observability.go‎

Lines changed: 41 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -29,15 +29,42 @@ func NewObservabilityHandler(queries ObservabilityQueries) *ObservabilityHandler
2929
return &ObservabilityHandler{queries: queries, now: time.Now}
3030
}
3131

32-
// Register mounts the deterministic product API. These endpoints are for
33-
// dashboards and drill-downs; the browser does not need an agent for ordinary
34-
// telemetry navigation.
32+
// observabilityDeadline bounds one dashboard query.
33+
//
34+
// The SQL API has always been bounded; these endpoints, which every widget
35+
// calls, had no deadline of their own — a query that went pathological held a
36+
// connection until the client gave up, and the client's own timeout is not
37+
// something the server can rely on.
38+
//
39+
// Twenty seconds is well clear of the slowest query measured under load here,
40+
// about two seconds, and stricter than the SQL API's own ceiling. It is
41+
// deliberately shorter than the worst legitimate wait behind a stuck
42+
// publication — the drain remainder plus the swap budget can exceed a minute —
43+
// so a publication that goes wrong answers the dashboard instead of holding
44+
// every widget's connection until it finishes.
45+
const observabilityDeadline = 20 * time.Second
46+
47+
// Register mounts the deterministic product API, each endpoint under a
48+
// deadline. These endpoints are for dashboards and drill-downs; the browser
49+
// does not need an agent for ordinary telemetry navigation.
3550
func (h *ObservabilityHandler) Register(group *echo.Group) {
36-
group.GET("/overview", h.overview)
37-
group.GET("/topology", h.topology)
38-
group.GET("/performance", h.performance)
39-
group.GET("/trace", h.trace)
40-
group.GET("/logs", h.logs)
51+
group.GET("/overview", h.bounded(h.overview))
52+
group.GET("/topology", h.bounded(h.topology))
53+
group.GET("/performance", h.bounded(h.performance))
54+
group.GET("/trace", h.bounded(h.trace))
55+
group.GET("/logs", h.bounded(h.logs))
56+
}
57+
58+
// bounded gives a handler a deadline. The context reaches DuckDB through
59+
// QueryContext, so an expired one stops the statement rather than only the
60+
// reply — and releases the pool connection it was holding.
61+
func (h *ObservabilityHandler) bounded(handler func(*echo.Context) error) func(*echo.Context) error {
62+
return func(c *echo.Context) error {
63+
ctx, cancel := context.WithTimeout(c.Request().Context(), observabilityDeadline)
64+
defer cancel()
65+
c.SetRequest(c.Request().WithContext(ctx))
66+
return handler(c)
67+
}
4168
}
4269

4370
func (h *ObservabilityHandler) performance(c *echo.Context) error {
@@ -129,5 +156,11 @@ func mapQueryError(err error) error {
129156
if errors.Is(err, observability.ErrInvalidScope) || errors.Is(err, observability.ErrInvalidLimit) {
130157
return echo.NewHTTPError(http.StatusBadRequest, err.Error())
131158
}
159+
// A query that ran out of time is a load condition, not a fault: reporting
160+
// it as a server error logs it at ERROR and counts it towards the 5xx rate
161+
// that pages someone.
162+
if errors.Is(err, context.DeadlineExceeded) {
163+
return echo.NewHTTPError(http.StatusServiceUnavailable, "telemetry query took too long").Wrap(err)
164+
}
132165
return echo.NewHTTPError(http.StatusInternalServerError, "telemetry query failed").Wrap(err)
133166
}

‎internal/api/observability_test.go‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@ package api
33
import (
44
"context"
55
"encoding/json"
6+
"errors"
7+
"fmt"
68
"net/http"
79
"net/http/httptest"
810
"testing"
@@ -91,3 +93,98 @@ func TestRouteRejectsInvalidWindow(t *testing.T) {
9193
t.Fatalf("status = %d, want 400", rec.Code)
9294
}
9395
}
96+
97+
// A dashboard query that runs out of time is a load condition. Reporting it as
98+
// a server error logs it at ERROR and counts towards the 5xx rate that pages
99+
// someone; the deadline itself exists so a pathological query stops holding a
100+
// connection once the answer can no longer be useful.
101+
func TestObservabilityQueryDeadlineIsNotAServerError(t *testing.T) {
102+
err := mapQueryError(fmt.Errorf("query endpoints: %w", context.DeadlineExceeded))
103+
var httpErr *echo.HTTPError
104+
if !errors.As(err, &httpErr) {
105+
t.Fatalf("mapQueryError returned %T, want an *echo.HTTPError", err)
106+
}
107+
if httpErr.Code != http.StatusServiceUnavailable {
108+
t.Fatalf("a timed-out query maps to %d, want %d", httpErr.Code, http.StatusServiceUnavailable)
109+
}
110+
// A genuine failure still reads as one.
111+
var failure *echo.HTTPError
112+
if !errors.As(mapQueryError(errors.New("boom")), &failure) || failure.Code != http.StatusInternalServerError {
113+
t.Fatal("a real query failure no longer maps to 500")
114+
}
115+
}
116+
117+
// Every registered route carries the deadline, not just the wrapper in
118+
// isolation: dropping bounded() from Register is the mistake a refactor makes,
119+
// and it left the whole suite green.
120+
func TestRegisteredObservabilityRoutesCarryADeadline(t *testing.T) {
121+
deadlines := map[string]bool{}
122+
handler := &ObservabilityHandler{queries: deadlineProbe{seen: deadlines}, now: time.Now}
123+
e := echo.New()
124+
handler.Register(e.Group("/api/observability"))
125+
for _, path := range []string{"/api/observability/overview", "/api/observability/topology", "/api/observability/performance", "/api/observability/logs", "/api/observability/trace"} {
126+
recorder := httptest.NewRecorder()
127+
e.ServeHTTP(recorder, httptest.NewRequest(http.MethodGet, path+"?window=1h", nil))
128+
if !deadlines[path] {
129+
t.Errorf("%s reached the query layer with no deadline", path)
130+
}
131+
}
132+
}
133+
134+
// deadlineProbe records, per route, whether the context that reached it had a
135+
// deadline. It answers every query with an empty result.
136+
type deadlineProbe struct{ seen map[string]bool }
137+
138+
func (p deadlineProbe) note(ctx context.Context, path string) {
139+
if _, ok := ctx.Deadline(); ok {
140+
p.seen[path] = true
141+
}
142+
}
143+
144+
func (p deadlineProbe) Overview(ctx context.Context, scope observability.Scope, limit int) (observability.Result[observability.Overview], error) {
145+
p.note(ctx, "/api/observability/overview")
146+
return observability.Result[observability.Overview]{}, nil
147+
}
148+
149+
func (p deadlineProbe) Topology(ctx context.Context, scope observability.Scope, limit int) (observability.Result[observability.Topology], error) {
150+
p.note(ctx, "/api/observability/topology")
151+
return observability.Result[observability.Topology]{}, nil
152+
}
153+
154+
func (p deadlineProbe) Performance(ctx context.Context, scope observability.Scope, service string, limit int) (observability.Result[observability.Performance], error) {
155+
p.note(ctx, "/api/observability/performance")
156+
return observability.Result[observability.Performance]{}, nil
157+
}
158+
159+
func (p deadlineProbe) Logs(ctx context.Context, scope observability.Scope, service, severity, search string, limit int) (observability.Result[observability.Logs], error) {
160+
p.note(ctx, "/api/observability/logs")
161+
return observability.Result[observability.Logs]{}, nil
162+
}
163+
164+
func (p deadlineProbe) Trace(ctx context.Context, scope observability.Scope, traceID, service string, limit int) (observability.Result[observability.TraceDetail], error) {
165+
p.note(ctx, "/api/observability/trace")
166+
return observability.Result[observability.TraceDetail]{}, nil
167+
}
168+
169+
// The handlers carry a deadline of their own, because the client's timeout is
170+
// not something the server can rely on.
171+
func TestObservabilityHandlersCarryADeadline(t *testing.T) {
172+
var seen time.Duration
173+
handler := (&ObservabilityHandler{}).bounded(func(c *echo.Context) error {
174+
deadline, ok := c.Request().Context().Deadline()
175+
if !ok {
176+
t.Fatal("handler ran with no deadline")
177+
}
178+
seen = time.Until(deadline)
179+
return nil
180+
})
181+
e := echo.New()
182+
request := httptest.NewRequest(http.MethodGet, "/api/observability/overview", nil)
183+
c := e.NewContext(request, httptest.NewRecorder())
184+
if err := handler(c); err != nil {
185+
t.Fatalf("bounded handler: %v", err)
186+
}
187+
if seen <= 0 || seen > observabilityDeadline {
188+
t.Fatalf("deadline left %s, want a positive value no greater than %s", seen, observabilityDeadline)
189+
}
190+
}

‎internal/metrics/metrics.go‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import (
77

88
"github.com/prometheus/client_golang/prometheus"
99
"github.com/prometheus/client_golang/prometheus/promauto"
10+
dto "github.com/prometheus/client_model/go"
1011
)
1112

1213
type RollupComponent string
@@ -232,8 +233,19 @@ var (
232233
Help: "Readers that gave up waiting to enter the Parquet snapshot",
233234
}, []string{"reader"})
234235

236+
// The wait to enter the snapshot and the wait for a connection are both
237+
// measured; how long the statement itself takes was not, so a slow request
238+
// could be attributed to queueing or to execution only by elimination.
239+
// The connection is acquired before this timer starts, so the two do not
240+
// overlap.
241+
DuckDBStatement = promauto.NewHistogram(prometheus.HistogramOpts{
242+
Name: "fanout_duckdb_statement_seconds",
243+
Help: "Time a read statement spent executing in DuckDB, excluding any wait to start it",
244+
Buckets: []float64{.0005, .001, .005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10, 30},
245+
})
246+
235247
// An explicit CHECKPOINT blocks new transactions for its duration, and the
236-
// non-FORCE form fails outright while any transaction is open — so both
248+
// non-FORCE form refuses while another write transaction is open — so both
237249
// how long it takes and how often it does not happen are worth knowing.
238250
DuckDBCheckpoint = promauto.NewHistogram(prometheus.HistogramOpts{
239251
Name: "fanout_duckdb_checkpoint_seconds",
@@ -365,6 +377,16 @@ func ParquetReadRefusalsForTest(reader string) prometheus.Counter {
365377
return parquetReadRefusals.WithLabelValues(reader)
366378
}
367379

380+
// DuckDBStatementSecondsForTest reports the statement histogram's running
381+
// total, so a test can prove what the timer does and does not cover.
382+
func DuckDBStatementSecondsForTest() float64 {
383+
metric := &dto.Metric{}
384+
if err := DuckDBStatement.Write(metric); err != nil {
385+
return 0
386+
}
387+
return metric.GetHistogram().GetSampleSum()
388+
}
389+
368390
// DuckDBPoolStatsForTest exposes what the pool gauges are reading, so a test
369391
// can prove they point at the read pool rather than the write handle.
370392
func DuckDBPoolStatsForTest() sql.DBStats { return poolStats() }

‎internal/observability/logs.go‎

Lines changed: 21 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -17,12 +17,28 @@ WHERE time >= ? AND time < ?
1717
// DuckDB answers the two questions the API asks: the newest `limit` entries
1818
// and per-bucket counts. LIMIT bounds the entry stream and GROUP BY bounds the
1919
// histogram stream regardless of the retained Parquet row count.
20+
//
21+
// Redaction runs outside the LIMIT, and that placement is the whole cost of
22+
// this query. A projection in the same SELECT as ORDER BY ... LIMIT is
23+
// evaluated below the top-N operator, so four chained regexp_replace ran over
24+
// every row in the window to return a hundred: measured at 140ms for a page of
25+
// 100 out of 372,000 rows, and 347ms with four DuckDB threads, against 8ms
26+
// once the top-N runs first. It was the one statement here whose cost grew
27+
// with retained rows rather than with the answer.
28+
//
29+
// The search predicate stays inside, on purpose: a search has to match what
30+
// the reader will be shown, so it matches the redacted text. It is skipped
31+
// entirely when no search term is given.
2032
var logEntriesQuery = `
21-
SELECT time, severity, coalesce(service, ''), ` + redactLogBodySQL("body") + `,
22-
coalesce(trace_id, ''), coalesce(span_id, '')
23-
FROM logs` + logFilters + `
24-
ORDER BY time DESC
25-
LIMIT ?`
33+
SELECT time, severity, service, ` + redactLogBodySQL("body") + `, trace_id, span_id
34+
FROM (
35+
SELECT time, severity, coalesce(service, '') AS service, body,
36+
coalesce(trace_id, '') AS trace_id, coalesce(span_id, '') AS span_id
37+
FROM logs` + logFilters + `
38+
ORDER BY time DESC
39+
LIMIT ?
40+
)
41+
ORDER BY time DESC`
2642

2743
var logBucketsQueryTemplate = `
2844
SELECT time_bucket(INTERVAL '%s', time) AS point_time,

‎internal/observability/performance.go‎

Lines changed: 52 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -51,35 +51,50 @@ const endpointRollupMatureQuery = `
5151
SELECT COUNT(*) >= 5
5252
FROM (SELECT DISTINCT bucket FROM endpoint_rollup LIMIT 5)`
5353

54+
// endpointInteriorBounds returns the half-open minute range the rollup cache
55+
// answers for, given the requested window and the cache's watermark.
56+
//
57+
// The interior starts at the first whole minute inside the window — a window
58+
// that begins mid-minute has a partial minute the cache cannot answer — and
59+
// ends at the earlier of the window's last whole minute and the watermark's,
60+
// because the cache knows nothing past what it has rolled up. Raw spans cover
61+
// everything outside that range.
62+
func endpointInteriorBounds(start, end, watermark time.Time) (interiorStart, interiorEnd time.Time) {
63+
interiorStart = start.Truncate(time.Minute)
64+
if !interiorStart.Equal(start) {
65+
interiorStart = interiorStart.Add(time.Minute)
66+
}
67+
interiorEnd = end.Truncate(time.Minute)
68+
if rolled := watermark.Truncate(time.Minute); rolled.Before(interiorEnd) {
69+
interiorEnd = rolled
70+
}
71+
return interiorStart, interiorEnd
72+
}
73+
5474
// endpointRollupQuery uses the minute cache only through its current watermark.
5575
// Raw spans cover both partial boundary minutes and any newer complete minutes,
56-
// so the cache lag cannot appear as zero traffic and raw work stays bounded.
76+
// so the cache lag cannot appear as zero traffic.
77+
//
78+
// The interior bounds are computed in Go and bound as parameters rather than
79+
// derived in a CTE. They read the same either way, but a bound derived from a
80+
// joined row is not a constant DuckDB can push into the Parquet scan: the
81+
// boundary exclusion below was applied after reading the window, so a query
82+
// that wants the two partial minutes at its edges read every row between them —
83+
// measured at 799,333 rows scanned out of 800,000 to return 7,333. Bound
84+
// directly, the same filter prunes to 11,333 rows read.
85+
//
86+
// The raw half is bounded by the cache being current, not by this query. When
87+
// the watermark sits behind the window the interior is empty, the boundary
88+
// predicate admits everything, and the scan reads the whole window — which is
89+
// correct, and is the case worth watching after an outage or a first run.
5790
var endpointRollupQuery = `
58-
WITH params AS (
59-
SELECT
60-
?::TIMESTAMP AS start_time,
61-
?::TIMESTAMP AS end_time,
62-
?::TIMESTAMP AS rollup_end,
63-
?::VARCHAR AS namespace,
64-
?::VARCHAR AS service
65-
),
66-
bounds AS (
67-
SELECT
68-
*,
69-
CASE
70-
WHEN start_time = date_trunc('minute', start_time) THEN start_time
71-
ELSE date_trunc('minute', start_time) + INTERVAL 1 MINUTE
72-
END AS interior_start,
73-
LEAST(date_trunc('minute', end_time), date_trunc('minute', rollup_end)) AS interior_end
74-
FROM params
75-
),
76-
rollup_source AS (
91+
WITH rollup_source AS (
7792
SELECT e.method, e.path, e.calls, e.error_count, e.duration_count, e.duration_buckets
78-
FROM endpoint_rollup e, bounds b
79-
WHERE e.bucket >= b.interior_start
80-
AND e.bucket < b.interior_end
81-
AND (b.namespace = '' OR e.namespace = b.namespace)
82-
AND (b.service = '' OR e.service = b.service)
93+
FROM endpoint_rollup e
94+
WHERE e.bucket >= ?
95+
AND e.bucket < ?
96+
AND (? = '' OR e.namespace = ?)
97+
AND (? = '' OR e.service = ?)
8398
),
8499
boundary_source AS (
85100
SELECT
@@ -107,12 +122,12 @@ boundary_source AS (
107122
le_30000 := COUNT(*) FILTER (WHERE s.duration_ms <= 30000),
108123
le_300000 := COUNT(*) FILTER (WHERE s.duration_ms <= 300000)
109124
) AS duration_buckets
110-
FROM spans s, bounds b
111-
WHERE s.start_time >= b.start_time
112-
AND s.start_time < b.end_time
113-
AND (s.start_time < b.interior_start OR s.start_time >= b.interior_end)
114-
AND (b.namespace = '' OR s.namespace = b.namespace)
115-
AND (b.service = '' OR COALESCE(s.service, '') = b.service)
125+
FROM spans s
126+
WHERE s.start_time >= ?
127+
AND s.start_time < ?
128+
AND (s.start_time < ? OR s.start_time >= ?)
129+
AND (? = '' OR s.namespace = ?)
130+
AND (? = '' OR COALESCE(s.service, '') = ?)
116131
GROUP BY method, path
117132
),
118133
sources AS (
@@ -286,7 +301,12 @@ func (s *Service) queryEndpoints(ctx context.Context, scope Scope, service strin
286301
source := "spans"
287302
if ready {
288303
query = endpointRollupQuery
289-
args = []any{scope.Start, scope.End, watermark, scope.Namespace, service, limit}
304+
interiorStart, interiorEnd := endpointInteriorBounds(scope.Start, scope.End, watermark)
305+
args = []any{
306+
interiorStart, interiorEnd, scope.Namespace, scope.Namespace, service, service,
307+
scope.Start, scope.End, interiorStart, interiorEnd, scope.Namespace, scope.Namespace, service, service,
308+
limit,
309+
}
290310
source = "endpoint_rollup + raw spans (interpolated histogram percentiles)"
291311
}
292312
rows, err := s.db.QueryContext(ctx, query, args...)

0 commit comments

Comments
 (0)