Skip to content

perf: reduce read latency and checkpoint contention - #473

Merged
vishr merged 3 commits into
mainfrom
perf/read-throughput
Nov 15, 2026
Merged

vishr merged 3 commits into
mainfrom
perf/read-throughput

Conversation

@vishr

@vishr vishr commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Summary

Dashboard reads can stall behind checkpoint vacuum locks and repeat avoidable native setup work. Reduce that contention while keeping resource budgets and admission limits unchanged.

 telemetry reads
- broad globs, watermark SQL per panel, indexed scan-only summaries
+ pinned file lists, committed watermark preflight, scan-only summary heaps
  • Preserve NOT NULL keys and transactional grouped replacement for span, log and endpoint summaries; advance disposable cache versions to 19/5.
  • Coalesce fresh fixed-window replacements, use projection UNNEST for histograms, and footer-prune reaggregation from the same captured catalog.
  • Start maintenance before initial rollup catch-up and retry only the pinned engine's exact checkpoint conflict within a bounded wait budget.
  • Add lifecycle, rollback, upgrade, cancellation and native lock regressions; repair verified format-5 replay, compile opt-in benchmarks in CI, and document the experiments and rejected alternatives.
  • Refresh generated dashboard SQL fixtures and use the current Pitot spill-footer namespace after updating the branch to main.

Companion runner changes: labstack/pitot-bench#13

Evidence

Local synthetic 90-second comparison on Apple M3 Max, four DuckDB threads, 4 GB native budget, eight read connections, fixed history and no stored-result reuse. Both runs offered 40 Executor requests/s alongside 20,480 accepted telemetry records/s, with accelerated five-second maintenance.

Measure Indexed summaries Scan-only summaries
Single-panel p95 / p99 32.23 / 91.61 ms 12.02 / 16.45 ms
Twelve-panel p95 / p99 275.21 / 543.80 ms 74.14 / 106.39 ms
Backend refusals 9 0
Generator misses 8 2
Dispatched / successful 3,592 / 3,583 3,598 / 3,598

The after run still fails zero-loss offered-load acceptance because of two generator misses. Percentiles describe successful requests; these local runs establish neither stable production capacity nor a maximum read rate. The separate 30-second candidate run completed all 1,200 offered requests.

  • Before: a compound-key scan waits on the database vacuum lock during checkpointing.
    After: the native control proves an identical unindexed table returns all 96 expected rows while checkpoint and indexed control remain pending.
  • Regression coverage checks exact aggregates and logical-key uniqueness across late repair, chunking, retries and rollback; restart from cache 18/4 preserves canonical telemetry/anomalies and rebuilds 19/5 caches.
  • Passed just check (repository-wide Go tests, 1,790 UI unit tests, script tests, lint, generated fixtures/docs, embedded assets, dependency audits and site checks).
  • Passed repository-wide just test-race, format-5 replay regressions, cache-upgrade and cold-start guards, opt-in harness compilation, and staged secret scanning.

See the investigation for methodology, further measurements and limitations.

Merge Danger

Door: two-way

The first start after upgrade rebuilds derived DuckDB caches. Rolling back also requires a disposable-cache rebuild; canonical Parquet telemetry, SQLite control state and anomaly history are preserved.

Blast Radius: reads

Changes read execution and analytical-cache maintenance. The three scan-only summaries now rely on serialized delete-and-grouped-insert uniqueness rather than SQL primary-key enforcement, with explicit lifecycle tests. Snapshot-bound coverage remains transactional; the engine pin, telemetry format, public contracts and admission limits are unchanged.

service_rollup and edge_rollup still carried compound ART primary keys
while the span, log and endpoint summaries dropped theirs. Those two are
the larger indexes, and the pinned engine kept probing them under the
checkpoint vacuum lock this change set out to avoid, so the contention fix
was only half applied.

Drop both keys under the same invariant that already protects the scan
summaries: refreshServiceRollup replaces each affected scope in one
serialized transaction. service_rollup inserts a single grouped aggregate;
edge_rollup unions three branches whose edge_type literals ('call',
'messaging', 'peer') are disjoint and each groups by the full key, so the
complete key stays unique without the index. Keep every key column NOT NULL.

Extend the rollup-key tests to assert both tables drop their index and keep
NOT NULL keys, to check their key uniqueness after every refresh, and to
prove the cache rebuild drops a previously-present primary key on reopen.
@vishr
vishr added this pull request to the merge queue Nov 15, 2026
Merged via the queue into main with commit 801856d Nov 15, 2026
31 checks passed
@vishr
vishr deleted the perf/read-throughput branch November 15, 2026 03:13
vishr added a commit to labstack/pitot-bench that referenced this pull request Nov 15, 2026
## Summary

Companion product changes: labstack/pitot#473

Read reports can hide scheduler misses and queue delay, and count drain
completions toward the offered load. Account for every scheduled arrival
and qualify reads using completion time from the scheduled arrival.

```text
offered arrivals
  → scheduler misses / shed / dispatched
  → dispatched queue + HTTP operation
  → success during load / success during drain / failure / cancellation
```

Report intended and performed operation kinds, including fallback
attribution. Add the proposed `interactive-http-draft/v1` per-kind
p95/p99 profile to workload metadata and qualification. Preserve
historical service-time fields; legacy and undersampled reports cannot
qualify against the new profile.

## Evidence

- **Before:** ticker losses and scheduler queue time were absent from
read evidence; drain completions could increase reported goodput.
**After:** tests reconcile every offered/dispatched outcome, delayed
scheduling, fallback mix, queue and completion latency, phase
cutoff/drain, profile matching, and the
1,000-successful-samples-per-kind gate.
- Passed `go test ./...`, `go test -race ./...`, `go vet ./...`, and
staged secret scanning.
- No new production capacity result is claimed. HTTP completion remains
separate from browser paint.

## Merge Danger

**Door:** two-way

New read qualification is intentionally stricter. Standard short sweeps
may use the documented best-completion topology fallback because they
cannot provide enough p99 samples. Published measurements are not
rewritten.

**Blast Radius:** reporting

Changes read-load scheduling, mixed-topology eligibility, report
metadata and qualification; ingest measurement is unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant