From eb124051d4bafe1afa11b329a648e668cc25ad7a Mon Sep 17 00:00:00 2001 From: Vishal Rana Date: Sat, 14 Nov 2026 19:03:35 -0800 Subject: [PATCH 1/3] docs: draft engine redesign proposals from the storage audit Record three structural bets surfaced by the read-only engine audit as Draft proposals, so the reasoning and trade-offs are reviewable before any build: - 0007: additive per-batch rollup aggregates, generalizing the existing read_trace_parts model to the summary rollups so maintenance cost tracks new rows instead of the touched event-time span. - 0008: derive service percentiles from the mergeable span_rollup histogram so alerts, overview and dashboards stop disagreeing, and drop the stored t-digest result. - 0009: append-only manifest catalog, WAL with a head block, and parallel time-ordered compaction, with the freshness-guarantee decision called out as the gating question. Each proposal names the related audit issues and marks itself measurement-gated and incremental; the cheap in-place fixes ship first regardless. --- .../0007-additive-rollup-aggregates.md | 110 ++++++++++++++++++ .../0008-mergeable-latency-sketch.md | 84 +++++++++++++ .../0009-manifest-catalog-wal-compaction.md | 96 +++++++++++++++ 3 files changed, 290 insertions(+) create mode 100644 docs/proposals/0007-additive-rollup-aggregates.md create mode 100644 docs/proposals/0008-mergeable-latency-sketch.md create mode 100644 docs/proposals/0009-manifest-catalog-wal-compaction.md diff --git a/docs/proposals/0007-additive-rollup-aggregates.md b/docs/proposals/0007-additive-rollup-aggregates.md new file mode 100644 index 00000000..7c854a23 --- /dev/null +++ b/docs/proposals/0007-additive-rollup-aggregates.md @@ -0,0 +1,110 @@ +# Additive per-batch rollup aggregates + +Status: Draft +Date: 2026-11-14 +Pull request: TBD + +## Summary + +Replace the raw re-aggregation rollup model for the scan summaries +(`span_rollup`, `log_rollup`, `endpoint_rollup`, and the resource/receiver +summaries) with additive per-batch partial aggregates that are merged at read +and folded during compaction. Cost becomes proportional to newly ingested rows +rather than to the event-time span touched by a pass. Keep watermark +re-aggregation only for the rollups whose output is not a simple additive +merge of per-batch partials. + +## Motivation + +`refreshServiceRollup` discovers the affected keys for a pass, derives one +event-time hull `[min(bucket), max(bucket)+1m)`, and re-scans raw Parquet over +that hull once per key-chunk to rebuild every summary. This has three +structural costs, all surfaced by the engine audit: + +- **A poison pill.** One span with an old `start_time` widens the hull to days, + and the unfiltered `log_agg`/`metric_agg` statements then OOM or hold the + write gate, freezing the watermark for every service-derived rollup. + (See issue: "one late or clock-skewed span stalls every service-derived + rollup".) +- **Cost = chunks × hull.** Unordered affected keys reuse the global `[lo,hi)`, + and the `struct_pack(...) IN (...)` predicate does not push into the Parquet + scan, so each 1000-key chunk decodes the whole hull. A 60k-key backfill is + ~60 chunks × four full span scans. +- **Delete churn and a convention-only uniqueness invariant.** Each pass deletes + its window and reinserts grouped rows, and correctness depends on every + insert path keeping the delete-first/group-by discipline. + +The repository already runs the additive model for `read_trace_parts` and +`read_resource_seconds` via `read_batches` markers and +`transferBatchCacheLocked` (`internal/engine/batch_cache.go:555-589`). This +proposal generalizes that proven mechanism to the summary rollups. + +## Goals and non-goals + +Goals: + +- Make summary-rollup maintenance cost proportional to new rows. +- Remove the event-time hull, the IN-list chunking, the poison pill, and the + per-pass delete churn for the summaries moved onto the model. +- Preserve exact aggregates, histogram bins, NOT NULL keys, and the existing + read contracts. + +Non-goals: + +- No change to the canonical Parquet telemetry format in the first phase (the + partials live in the disposable DuckDB read cache, which is already + version-gated and rebuildable). +- No change to `edge_rollup`, `receiver_rollup`, or the top-128 route fold, + which are join-shaped or order-dependent and stay on watermark + re-aggregation. + +## Design + +Each summary rollup becomes a set of per-batch partial aggregates keyed by the +summary's logical key plus the contributing batch ID, written once when a batch +first enters the read cache, exactly as `read_trace_parts` is today. A read or +a compaction merges the partials additively: + +- Counts and sums add directly. +- `min`/`max` take the extreme. +- Histograms merge by bin (the scale-3 layout already in `span_rollup.hist` + is mergeable). + +Because partials are additive, a duplicate per logical key is expected and +merges correctly, so the uniqueness invariant disappears for these tables and +the ART primary keys become unnecessary (already dropped; see the read-perf +work). A batch's partials are retired when the batch is compacted away, with the +compaction output contributing its own merged partials — the same lifecycle the +trace caches already follow. + +A later phase can compute the partials in Go at `Repository.Commit` and persist +them as an immutable per-batch sidecar, removing DuckDB writes, gates, WAL and +checkpoints for these tables entirely, at the cost of a batch-format bump. + +## Alternatives + +- **Keep raw re-aggregation, fix the cliffs only.** Bucket-order the affected + keys with per-chunk bounds and key-filter the log/metric aggregates. Removes + the poison pill and most of the chunks × hull cost without a model change. + This is the lower-risk fallback if the additive model does not pay off, and + the two are not mutually exclusive — the cliff fixes can ship first. +- **Materialized views.** DuckDB has no incremental materialized views that fit + this ingest pattern; emulating them would reproduce the current rescan. + +## Risks + +- Correctness of additive merges for every statistic, especially histogram bin + alignment and the endpoint top-128 fold (which is why the fold stays on + re-aggregation). +- Read-time merge cost for very long windows; mitigated by folding partials + during compaction so a read never merges more partials than live batches. +- Cache-version and rebuild semantics must cover the new partial tables. + +## Open questions + +- Which summaries move in phase one (proposal: span and log first, endpoint and + resource second)? +- Does the read-time merge stay within the panel latency budget at 30-day + windows, or does it force the compaction-fold step into phase one? +- Measurement: A/B the additive model against the current model on non-pico + data before committing. diff --git a/docs/proposals/0008-mergeable-latency-sketch.md b/docs/proposals/0008-mergeable-latency-sketch.md new file mode 100644 index 00000000..edea9350 --- /dev/null +++ b/docs/proposals/0008-mergeable-latency-sketch.md @@ -0,0 +1,84 @@ +# Mergeable latency sketch and derived service rollup + +Status: Draft +Date: 2026-11-14 +Pull request: TBD + +## Summary + +Stop storing final, unmergeable p50/p95 values in `service_rollup`. Derive +service latency percentiles, error rate, and counts from the mergeable +`span_rollup` histogram and counts at read time, so every surface computes the +same number and any window is re-aggregable. `service_rollup` becomes a thin +derivation (or is removed) rather than an independent summary. + +## Motivation + +`service_rollup` stores a per-minute t-digest result as final `p50_ms`/`p95_ms` +columns. A t-digest result cannot be merged across minutes, so each consumer +invents its own cross-minute combination: + +- Alerts take an unweighted `avg(p95_ms)` and `avg(error_rate)` + (`internal/alert/engine.go:520-530`). +- The overview takes `MAX(p95_ms)` (`internal/observability/latency_sql.go:15`). +- Dashboards merge the `span_rollup` histogram correctly + (`internal/query/rollup_compile.go:75-81`). + +The same service and window therefore reports three different p95 values. (See +issue: "three percentile merges disagree for the same service and window".) +The `span_rollup` histogram is already a mergeable, OTel-style scale-3 +exponential histogram, so the correct value is already computable — it is simply +not used consistently. + +## Goals and non-goals + +Goals: + +- One definition of service p50/p95/error rate, computed from mergeable inputs, + used by alerts, the overview, and dashboards. +- Re-aggregable over any window without a stored final percentile. +- Remove the t-digest scan and the `COUNT(DISTINCT m.name)` aggregate from the + rollup pass. + +Non-goals: + +- No change to the histogram layout or its bin semantics. +- No new precision guarantee beyond what the existing scale-3 histogram gives; + this proposal makes the existing precision consistent, not finer. + +## Design + +Percentiles for a service over a window are read from the summed `span_rollup` +histogram for that service and window, using the same bin-interpolation the +dashboards already use. Error rate is `sum(error spans) / sum(spans)` from the +span counts. Log and metric counts come from `log_rollup` and +`read_metric_names`. Alerts, the overview, and dashboards call one shared helper +so the definition cannot drift again. + +`service_rollup` keeps only the genuinely additive per-minute columns it still +needs as a convenience (counts), or is dropped entirely if nothing needs a +pre-joined per-minute row once percentiles are derived. The rollup tables are a +rebuildable disposable cache, so this needs no batch-format change. + +## Alternatives + +- **Store a mergeable sketch state** (DDSketch or ClickHouse-style + `quantileState`) in `service_rollup` instead of deriving from the histogram. + More storage and a second sketch to maintain, when the span histogram already + carries the information. +- **Pick one of the three existing merges and apply it everywhere.** `avg` and + `MAX` are both wrong; standardizing on a wrong merge is not acceptable. + +## Risks + +- The histogram's bin resolution sets percentile accuracy; confirm it meets the + alerting and overview needs before removing the t-digest. +- Read-time percentile computation cost over long windows; bounded by the + rollup grain and mitigated by the additive-partials proposal (0007). + +## Open questions + +- Keep a slim `service_rollup` for counts, or remove it and read counts from + `span_rollup`/`log_rollup` directly? +- Does deriving percentiles change any alert threshold's observed behavior + enough to need a migration note for existing alert rules? diff --git a/docs/proposals/0009-manifest-catalog-wal-compaction.md b/docs/proposals/0009-manifest-catalog-wal-compaction.md new file mode 100644 index 00000000..90a9a85b --- /dev/null +++ b/docs/proposals/0009-manifest-catalog-wal-compaction.md @@ -0,0 +1,96 @@ +# Manifest catalog, write-ahead log, and parallel compaction + +Status: Draft +Date: 2026-11-14 +Pull request: TBD + +## Summary + +Replace the filesystem-as-catalog model with an append-only manifest, add a +write-ahead log with a queryable head block so an acknowledgement is not a full +Parquet publish, and run compaction in parallel lanes with time-leading span +order. These are coupled bets about the ingest and storage layer; this proposal +records them together and the trade-offs that gate them. + +## Motivation + +The engine audit found the ingest/storage layer correct and crash-safe but +structurally expensive: + +- **Filesystem as catalog.** Boot is proportional to stored data (per batch: + `metadata.json` read twice, three footers, the whole `trace.fidx`), and + `BatchMetadata()` copies and sorts the entire catalog under a lock on every + statement. (Issues: "BatchMetadata copies and sorts the whole catalog"; + startup cost.) +- **No WAL.** Every group commit is a sort, up to three wide Parquet encodes, + up to seven fsyncs, a rename, and a footer re-parse, and it creates a + generation-0 directory re-encoded at least twice. The debt limit, admission + bands, and pressure windows all exist to manage this flood. +- **Single compaction lane with discard-on-preempt**, and span files ordered + trace-hash-first so they have no time locality (day merges capped at 250k + rows, narrow-time queries decode whole files). + +## Goals and non-goals + +Goals: + +- Boot proportional to the manifest, not to stored data. +- One refcounted catalog snapshot per epoch, shared by all readers, replacing + the pin/epoch scheme and the per-statement catalog copy. +- Acknowledge ingest after a bounded append, not a full publish. +- Parallel compaction with no discarded work and time-prunable large files. + +Non-goals: + +- Not a move to an external catalog service or a second storage engine; the + manifest stays a plain local file so ingest fsyncs stay independent of control + state. +- Not a change to UTC `TIMESTAMPTZ_NS` instant semantics. + +## Design + +**Manifest.** An append-only log of edit records (add / replace / prune), each +fsynced, carrying per-batch row counts and event-time bounds, plus namespace and +service sets for pruning. It replaces `metadata.json` re-reads, `replaces.json`, +`pruned`, `COMPACTION.json`, the `.retired` renames, and the large-directory +fsync. Boot is a log replay; a snapshot is a refcounted manifest version, which +subsumes `pinMu`/epoch and lets `BatchMetadata()` become one immutable load. + +**WAL + head block.** Ingest appends group-committed rows to a WAL and +acknowledges after one `fdatasync`; a sorted Parquet batch is cut at a size or +time threshold (~200k–1M rows or a few seconds). This removes one compaction +generation and most of the debt and admission machinery. The head block must +stay queryable, or the product accepts a few seconds of freshness lag — this +conflicts with today's "immediately visible" behavior and must be decided first. + +**Parallel tiered compaction.** Two or more lanes over disjoint input groups, +with no discard-on-preempt. In a new batch-format version, order spans by time +bucket then trace hash, with `trace.fidx` mapping a hash to several ranges, so +large multi-day files stay prunable by row-group statistics and the 250k-row cap +can go. + +## Alternatives + +- **Incremental fixes without the manifest.** The atomic-snapshot fix for + `BatchMetadata()` (issue) and header-only boot loading recover much of the + per-statement and boot cost without the manifest. These should ship first + regardless; the manifest is justified only if those prove insufficient. +- **SQLite as the catalog.** Couples ingest durability to control-plane state + and adds a second fsync domain; rejected in favor of a plain append-only file. + +## Risks + +- The freshness-guarantee decision for the WAL head block is load-bearing and + product-visible. +- A format bump for span ordering must be validated on non-pico data before + adoption, per the no-curve-fitting rule. +- Manifest replay and compaction crash-recovery must match today's + fsync-before-rename durability exactly. + +## Open questions + +- Keep "immediately visible" by making the head block queryable, or accept a + bounded freshness lag? +- Can the manifest ship incrementally (catalog snapshot first, WAL second, + compaction/order third), and which steps need the format bump? +- What is the measured boot and ingest win versus the incremental fixes alone? From 796e4e0805a626ad1ce446a830e9208b97303644 Mon Sep 17 00:00:00 2001 From: Vishal Rana Date: Sun, 15 Nov 2026 08:25:30 -0800 Subject: [PATCH 2/3] docs(proposals): fold three-agent research validation into 0007/0008/0009 Add a dated research-validation section to each engine-redesign proposal, capturing a prior-art/best-in-class pass, an in-codebase correctness and crash-safety pass, and a performance/sequencing pass: - 0007: confirmed as ClickHouse AggregatingMergeTree (-State/-Merge); make the compaction fold mandatory in phase one, state the exactly-once and sum+count-not-avg invariants, and resolve the cross-gate snapshot design. - 0008: confirmed as the Prometheus/OTel exponential-histogram approach; the shared helper must apply the served-kind filter, replicate the served-vs-all choice, pin interpolation, and handle the zero bucket; state the 4.33% bound and add an alert-migration note. Lowest risk; ship first. - 0009: manifest/WAL/compaction each match best-in-class, but parallel compaction is not safe as drafted (single marker/attempt/recovery, overlapping input selection, and a one-merge memory budget all cause data loss or OOM); add manifest checkpointing and a WAL replay bound; incremental fixes ship first (BatchMetadata snapshot already landed in #488). Proposals stay Draft; no production code changed. --- .../0007-additive-rollup-aggregates.md | 54 ++++++++++++++ .../0008-mergeable-latency-sketch.md | 50 +++++++++++++ .../0009-manifest-catalog-wal-compaction.md | 72 +++++++++++++++++++ 3 files changed, 176 insertions(+) diff --git a/docs/proposals/0007-additive-rollup-aggregates.md b/docs/proposals/0007-additive-rollup-aggregates.md index 7c854a23..4a77e2e6 100644 --- a/docs/proposals/0007-additive-rollup-aggregates.md +++ b/docs/proposals/0007-additive-rollup-aggregates.md @@ -108,3 +108,57 @@ checkpoints for these tables entirely, at the cost of a batch-format bump. windows, or does it force the compaction-fold step into phase one? - Measurement: A/B the additive model against the current model on non-pico data before committing. + +## Research validation (2026-11-15) + +A three-angle research pass (prior art, in-codebase fit, performance) was run +before build. Findings: + +**Prior art.** This design is ClickHouse's `AggregatingMergeTree` with the +`-State` / `-Merge` combinators: write a partial aggregate state per batch, +combine states at read, fold states during a background merge. Apache Druid's +ingestion-time rollup is the same idea for the phase-2 immutable sidecar. The +repository already runs this pattern for `read_trace_parts` via +`transferBatchCacheLocked`, so there is an in-repo precedent, not only external +ones. Keeping only associative/commutative stats on the additive model and +leaving non-additive ones (endpoint top-128 fold, joins) on re-aggregation +matches ClickHouse and Pinot guidance exactly. + +**Required revisions before build:** + +1. **Make the compaction fold mandatory in phase one, not an open question.** + ClickHouse's best-known production failure is "too many unmerged parts": a + read-time `-Merge` over thousands of per-batch states is slow, and the whole + design depends on background merges keeping the state count low. Phase one + without the fold reproduces that slowness at long windows. The in-repo + `transferBatchCacheLocked` already folds inputs into the compaction output, + so commit to it from the start. +2. **State the correctness invariants explicitly:** exactly-once contribution (a + batch's partial is counted once across retries via its `read_batches` marker, + and a compaction output *replaces* its inputs, never adds to them); store + `sum` + `count`, never `avg`, and divide at read for error rate and means; + every batch histogram is at the same scale-3 layout so bin-merge is exact + (reject or downscale-align anything else). +3. **Resolve the cross-gate snapshot design.** Moving span/log/endpoint rollups + onto per-batch partials moves them from the writeGate/watermark world to the + cacheGate/`read_batches` world, while `service_rollup` (0008) and + `edge_rollup` stay analytical. A consumer that reads a span partial *and* an + analytical rollup then spans two gates/snapshots and can see a torn view. + AGENTS.md requires maintenance to hold both gates for checkpointing; the + proposal must specify how a reader gets a consistent cross-gate snapshot. +4. **Name the residual.** Phase one keeps partials in the version-gated + disposable cache, so a cache-version bump re-derives all partials from raw + Parquet — the exact full-hull scan this removes, merely deferred to rebuild. + Only the phase-2 immutable Parquet sidecar removes that scan permanently. + +**Status confirmed.** The ART primary keys on `span`/`log`/`endpoint`/`service`/ +`edge_rollup` are already dropped on `main` (`rollup_schema_version` 5), so the +"already dropped" premise holds; the additive model's duplicate-per-batch rows +will not violate a PK. Versioning: a `readCacheVersion` + `rollup_schema_version` +bump (rebuildable, no format bump, no Goose) for phase one; a batch-format bump +only for the phase-2 sidecar. + +**Build readiness:** buildable after the cross-gate design (item 3) is specified +and the compaction fold is committed (item 1). Lower-risk fallback remains the +0007 "cliff fixes" (bucket-order affected keys, per-chunk bounds, key-filter the +log/metric aggregates) if the 30-day read-merge A/B blows the panel budget. diff --git a/docs/proposals/0008-mergeable-latency-sketch.md b/docs/proposals/0008-mergeable-latency-sketch.md index edea9350..61a97be3 100644 --- a/docs/proposals/0008-mergeable-latency-sketch.md +++ b/docs/proposals/0008-mergeable-latency-sketch.md @@ -82,3 +82,53 @@ rebuildable disposable cache, so this needs no batch-format change. `span_rollup`/`log_rollup` directly? - Does deriving percentiles change any alert threshold's observed behavior enough to need a migration note for existing alert rules? + +## Research validation (2026-11-15) + +A three-angle research pass (prior art, in-codebase fit, performance) was run +before build. Findings: + +**Prior art.** Deriving percentiles at read from a summed, mergeable +exponential histogram is exactly how Prometheus native histograms +(`histogram_quantile`), Mimir, and the existing dashboards compute them. The +`span_rollup` histogram is already a scale-3 OTel exponential histogram, whose +relative error is a fixed `(base-1)/(base+1) ≈ 4.33%` per bucket that holds +*through merges* — i.e. it already has DDSketch-class guarantees, so adding a +separate DDSketch/`quantileState` structure buys no new information. Averaging +quantiles (`avg(p95)`, alerts) and `MAX(p95)` (overview) are textbook-wrong; the +fix direction is unambiguously correct. + +**Framing to tighten:** the bug is the stored *scalar* `p50_ms`/`p95_ms` (a +t-digest *result*), which is categorically non-mergeable. A t-digest *state* +would be mergeable but has no formal post-merge error bound (reported >100% tail +error after many merges), whereas the exponential histogram's bound holds through +merges — so "derive from the histogram" is the clearly superior option, not just +a convenient one. State the ±4.33% scale-3 bound in the proposal and use it to +answer the alert-migration question: a threshold near a bucket edge can flip, so +a migration note is required. + +**Required for the shared helper (all surfaces must use it):** + +1. **Apply the served-span kind filter** (`kind NOT IN CLIENT/PRODUCER`, with the + COALESCE-to-all fallback) before merging, or every latency number silently + changes and the "stream held open marks service unhealthy" fix regresses. +2. **Replicate the served-vs-all choice.** Today the overview prefers served + buckets *per bucket*; a window-summed histogram can only choose served-only or + all-span over the whole window. The helper needs both merges and a rule ("use + served if served count > 0 over the window, else all") — close but not + identical to today's per-bucket COALESCE. This is a deliberate, documented + behavior change; decide it before building. +3. **Pin the bin-interpolation method** (match the OTel/Prometheus convention). + Identical bins still disagree under different interpolation — this is the + likeliest way the three-way disagreement returns. +4. Handle the **zero/empty bucket** and gaps, and ensure the histogram's total + count equals the error-rate denominator (**count-base consistency**). + +Also move both alert windows (previous and current) to the helper, and grep for +any remaining reader of `service_rollup.metric_count` before dropping it. + +**Build readiness:** lowest risk of the three; read-side only, no format change, +a `rollup_schema_version` bump (rebuildable). Ship first — it also proves the +`span_rollup` histogram is the canonical input, de-risking 0007. Gate before +build: the item-2 behavior-change decision plus the accuracy A/B +(histogram-derived vs t-digest p95 on the same spans). diff --git a/docs/proposals/0009-manifest-catalog-wal-compaction.md b/docs/proposals/0009-manifest-catalog-wal-compaction.md index 90a9a85b..08301192 100644 --- a/docs/proposals/0009-manifest-catalog-wal-compaction.md +++ b/docs/proposals/0009-manifest-catalog-wal-compaction.md @@ -94,3 +94,75 @@ can go. - Can the manifest ship incrementally (catalog snapshot first, WAL second, compaction/order third), and which steps need the format bump? - What is the measured boot and ingest win versus the incremental fixes alone? + +## Research validation (2026-11-15) + +A three-angle research pass (prior art, in-codebase fit, performance) was run +before build. The direction of all three bets matches best-in-class systems, but +the parallel-compaction bet is **not safe to build as drafted**. Findings by bet: + +**Manifest catalog — strongest-grounded.** Append-only manifest + boot-as-replay ++ one refcounted immutable snapshot shared by all readers is Iceberg +(`metadata.json` → manifest list), Delta Lake (`_delta_log` + checkpoints), and +Lucene (`segments_N` commit points) — the last maps directly onto replacing the +`pinMu`/epoch scheme. **Missing piece (highest value): manifest +checkpointing.** An add/replace/prune log grows unbounded, so boot-replay cost +creeps back to O(history) — the problem relocated into the log. Add a periodic +manifest checkpoint/snapshot, old-record pruning, orphan-file cleanup, and +explicit tombstone / expire-before-delete crash semantics (Delta checkpoints +every ~10 commits; Iceberg has `rewrite_manifests` + snapshot expiry). The +manifest must preserve the pin/epoch guarantee exactly: no reader excluded or +waited for, directories survive until no pin ≤ epoch. + +**WAL + head block.** Ack-after-`fdatasync` + cut a batch at a size/time +threshold is RocksDB/Elasticsearch-standard; a queryable in-memory head is +Prometheus TSDB / Druid real-time / Lucene NRT. **Recommend the queryable-head +path** to preserve today's "immediately visible" contract (AGENTS.md); the +freshness-lag option is a genuine product regression, a fallback only. **Missing +piece: a WAL replay bound** — truncate/rotate/checkpoint the WAL once the head is +cut to Parquet, paired with the manifest checkpoint, or replay is unbounded. + +**Parallel tiered compaction — do NOT build as drafted.** Parallel lanes over +disjoint groups with no discard-on-preempt is RocksDB subcompactions / Lucene +CMS; time-leading span order with a trace-id→row-group index is Grafana Tempo's +vParquet. But against today's single-marker machinery it loses data: + +1. **Per-lane durable state.** There is one `COMPACTION.json`, one + `COMPACTION-ATTEMPT.json`, one staging root, and one `recoverCompaction` that + runs before any pass. Two lanes overwrite each other's marker → on crash the + other lane's already-retired inputs are orphaned with no rollback record + (data loss; the inputs hold the only copy). Needs per-lane markers + attempt + records + per-lane preempt size, and recovery that iterates *all* markers and + protects *both* outputs' `.retired-` sets in cleanup. +2. **Disjoint input partitioning is mandatory.** `selectCompactionBatchesWith- + HigherCap` is not lane-aware; two concurrent calls on the same snapshot can + select overlapping batches (a day group and an hour group both containing + batch X), double-retiring X. Lanes must claim non-overlapping batch sets + atomically under `compactionMu` before releasing to merge, each excluding the + other lane's claimed IDs. +3. **Split the memory budget.** `maxCompactionBytes` (768 MiB) is sized for ONE + merge against the 4 GB DuckDB budget; two concurrent merges double peak RSS + and can OOM. Removing `errCompactionPreempted` is safe **only after** disjoint + partitioning *and* a split/halved per-lane byte budget — otherwise a + preempt-and-retry is traded for an OOM. Reserve ≥1 lane for small/recent + parts so a multi-day merge cannot starve freshness-critical compactions + (ClickHouse/LSM lesson). +4. **Span reorder needs a format bump** (format 5 → 6; the offline sort-tuple + check updates with it), validated on non-pico data per the no-curve-fitting + rule. Accept and document the trace-by-id point-lookup regression: one hash + maps to several row-group ranges, so `trace.fidx` must store *all* ranges and + get-trace unions them (Tempo accepts this with bloom filters + index). + +**Incremental fixes ship first, regardless of the manifest.** The +`BatchMetadata()` atomic pre-sorted snapshot is **already done** (PR #488), so +the per-statement catalog-copy win is banked and the manifest is now justified +only by residual boot cost. Header-only boot loading is still to do and is the +next cheap, invariant-safe win. + +**Build sequence:** (1) decide the WAL freshness guarantee (product call, no +code); (2) header-only boot A/B to settle whether the manifest is even needed; +(3) manifest + checkpointing (no format bump, backfillable); (4) WAL + queryable +head (freshness-gated); (5) parallel compaction + span reorder (format bump, +per-lane durability + disjoint partition + split budget, validated on chip). +Steps 3–5 are each independently revertible pre-release except the format bump, +whose rollback is "rebuild," not "downgrade." From 81b30ffbfaa88d13e3af6fccba1740acc06b8d71 Mon Sep 17 00:00:00 2001 From: Vishal Rana Date: Sun, 15 Nov 2026 17:49:53 -0800 Subject: [PATCH 3/3] docs(proposals): draft 0010 result-cache extent key and 0011 engine admission; index 0007-0011 0010 addresses the result-cache key fragmentation and sub-refresh TTL (issue #477) with a normalized key and a watermark-split extent cache. 0011 addresses partial, cost-unaware read admission (issue #480) with one admission layer and interactive/panel/background/agent classes. Both are drafts and still need the three-angle research pass before build. Also adds the missing index rows for 0007-0011. --- .../proposals/0010-result-cache-extent-key.md | 130 ++++++++++++++++++ .../0011-engine-admission-classes.md | 125 +++++++++++++++++ docs/proposals/README.md | 5 + 3 files changed, 260 insertions(+) create mode 100644 docs/proposals/0010-result-cache-extent-key.md create mode 100644 docs/proposals/0011-engine-admission-classes.md diff --git a/docs/proposals/0010-result-cache-extent-key.md b/docs/proposals/0010-result-cache-extent-key.md new file mode 100644 index 00000000..fe6899b4 --- /dev/null +++ b/docs/proposals/0010-result-cache-extent-key.md @@ -0,0 +1,130 @@ +# Result-cache key normalization and an extent cache + +Status: Draft +Date: 2026-11-15 +Pull request: TBD + +## Summary + +Rework the dashboard result cache so a panel's cache key depends only on what +actually changes its result, and so the stable, already-computed part of a +relative window is reused instead of recomputed every refresh. Key on a +normalized panel spec, the resolved bucket interval, and only the variables the +panel references; align relative windows to the resolved step; and cache the +closed buckets older than the rollup watermark with a long TTL, re-querying only +the moving tail. This follows the query-frontend extent cache in Mimir, Cortex, +and Loki, and Druid's per-segment result cache. + +## Motivation + +The result cache (`internal/query/exec.go`, `internal/query/result_cache.go`) +rarely hits, so a single viewer re-runs DuckDB for work it just did. The engine +audit found several independent reasons, all in how the key is built and how +windows are chosen: + +- **Raw pixel width in the key.** The key uses the raw viewport `Width` + (`exec.go:417,478`, `result_cache.go:445-452`) even though `AutoInterval` + (`bucket.go:40-52`) quantizes it to a handful of intervals. Every viewport + width therefore starts its own DuckDB run, although most resolve to the same + bucket grain. +- **Whole-dashboard hashing.** The key hashes the whole dashboard, so any edit + invalidates every panel, and two identical panels on different dashboards + never share a result. +- **All variables, not referenced ones.** The key includes every dashboard + variable rather than only those the panel's query references, so changing an + unrelated variable misses the cache. +- **TTL shorter than the refresh period.** The default TTL is 10s + (`config.go:101`) against a 30s refresh (`exec.go:167-168`), so an entry + expires before the next refresh reads it: a single viewer sees a 0% hit rate. +- **Unaligned relative windows.** Relative windows are not step-aligned, so no + two refreshes cover the same buckets and the cache cannot reuse a prior run. + +(See issue: "result-cache key fragments and TTL is shorter than the refresh +period".) + +## Goals and non-goals + +Goals: + +- A cache key that changes only when the panel's result would change: the + normalized panel spec, the resolved interval, and the referenced variables. +- Hits across refreshes of the same panel, across identical panels on different + dashboards, and across viewport widths that resolve to the same interval. +- Reuse of the stable, closed-bucket portion of a relative window, recomputing + only the tail that is still changing. + +Non-goals: + +- No change to what a panel computes or to the arbitrary-SQL safety boundary. + Normalization is for keying only; it never alters the statement sent to the + engine. +- No change to the rollup watermark or freshness semantics; the extent cache is + bounded by the existing watermark, not a new freshness guarantee. + +## Design + +**Normalized key.** Build the key from a canonical form of the single panel's +query (spec fields in a fixed order, with display-only fields such as raw pixel +width removed), the interval that `AutoInterval` resolves the width to, and only +the variables the panel's query references. Panels that produce the same +statement over the same interval and variables share one entry regardless of the +dashboard they live on or the exact viewport width. + +**Step-aligned relative windows.** Snap a relative window's start and end to the +resolved step before keying and querying, so successive refreshes of a "last 1h" +panel cover the same bucket boundaries and a refresh can reuse the previous +run's buckets. + +**Extent cache.** Split a window at the rollup watermark. Buckets older than the +watermark are closed: their result cannot change, so cache them with a long TTL +keyed by the aligned extent. Re-query only the open tail between the watermark +and now on each refresh, and stitch the cached closed extent to the fresh tail. +This is the query-frontend extent cache used by Mimir, Cortex, and Loki, and +Druid's per-segment result cache. The TTL question above disappears for the +closed portion, because a closed extent is immutable. + +**TTL.** Set the live (tail) TTL at least as long as the refresh period so a +single viewer's own refresh hits, while the closed extents carry their own long +TTL. + +## Alternatives + +- **Just raise the TTL.** Lengthening the TTL past the refresh period fixes the + single-viewer 0% hit rate but leaves the key fragmentation (pixel width, whole + dashboard, all variables, unaligned windows), so cross-panel and cross-refresh + reuse still miss. The TTL change is the cheapest first step and can ship alone, + but it is not sufficient. +- **Cache only whole-window results.** Simpler than an extent cache, but a + relative window never repeats exactly, so a whole-window entry for "last 1h" + is stale on the next refresh. Extents are what make a moving window reusable. + +## Risks + +- Normalization must be exact: two panels share an entry only if their + statements and results are truly identical, so the canonical form must capture + every field that affects the result and nothing that does not. A missed field + would serve a wrong result. +- Stitching a cached closed extent to a fresh tail must preserve the exact + bucket boundaries and any cross-bucket aggregation, or a boundary bucket could + be double-counted or dropped. +- The watermark split must use the same watermark the rollups use, or a closed + extent could cache a bucket that is still changing. + +## Open questions + +- Does the extent cache live at the result-cache layer (keyed extents of the + frame) or deeper, and how does it interact with the result cache's existing + single-flight coalescing? +- Which variables count as "referenced" — only those in the compiled statement, + or also those in filters that were dropped as All? +- What is the measured hit-rate and latency improvement versus the TTL-only + change alone, on non-pico data? + +## Research validation + +This proposal has not yet had the three-angle research pass (prior art, +in-codebase fit, performance) that precedes a build. The prior art named above +(Mimir/Cortex/Loki query-frontend extent cache, Druid per-segment cache) comes +from the audit; it must be validated against the current result-cache and +watermark code before implementation, and the TTL-only first step measured +separately. diff --git a/docs/proposals/0011-engine-admission-classes.md b/docs/proposals/0011-engine-admission-classes.md new file mode 100644 index 00000000..26d1869a --- /dev/null +++ b/docs/proposals/0011-engine-admission-classes.md @@ -0,0 +1,125 @@ +# One cost-aware admission layer at the engine boundary + +Status: Draft +Date: 2026-11-15 +Pull request: TBD + +## Summary + +Put every read through one admission layer at the engine boundary, with a small +set of classes (interactive, panel, background, agent) that shed by deadline +rather than by connection count, and give each panel run a single pinned +snapshot and connection shared by its sub-reads. Today only some readers are +admitted, a slot is just a connection with no notion of thread or memory cost, +and the single background slot lets one rollup pass block the read-cache +refresh. + +## Motivation + +Admission is partial and not cost-aware (`internal/query/exec.go:120-126` and the +engine's slot machinery). The engine audit found three structural problems: + +- **Admission is not global.** Only panels, exemplars, and trace detail pass + through admission. The observability readers + (`internal/observability/{overview,topology,dependencies,trace,health_trend}.go`), + annotations, and variable-option queries (`internal/query/variables.go:70-73,107`) + call the engine directly and compete for the two spare connections and all + DuckDB threads without being admitted or shed. (See issue: "read admission is + neither global nor cost-aware".) +- **A slot has no cost.** A slot is one connection, with no notion of the thread + or memory a statement will use, so a cheap point read and a wide multi-day scan + count the same against the pool. +- **The background slot is a single choke point.** `backgroundSlots` has + capacity 1 (`internal/engine/duck.go:540`), so a rollup pass blocks + `refreshReadCaches` for its full duration. This contradicts the stated intent + that completed-batch writes have their own gate and slot. + +## Goals and non-goals + +Goals: + +- Every read, whatever its entry point, passes through one admission layer, so + load is actually bounded and shedding is uniform. +- Classes with distinct policies: interactive and panel reads are latency-first + and shed by deadline; background and agent work yields to them. +- Admission that accounts for cost (threads, and a memory estimate) rather than + only a connection count. +- One pinned snapshot and one connection per panel run, shared by its sub-reads, + so a panel's several statements see one consistent catalog and do not each take + a separate slot. +- Background work (rollups, read-cache refresh) that no longer serializes behind + a single slot where the design calls for separate gates. + +Non-goals: + +- Not a general query scheduler or priority preemption of running statements in + the first phase; shedding is at admission (by deadline), not mid-flight + cancellation beyond what the existing deadlines already do. +- No change to the pinned-snapshot correctness contract or the write gates' + semantics; this reorganizes who waits, not what a reader sees. + +## Design + +**One admission layer.** Route every engine read — panels, exemplars, trace +detail, the observability readers, annotations, and variable options — through a +single admit call at the engine boundary. Nothing reaches a connection without +being admitted. + +**Classes.** Define a small set: interactive (health checks, variable options, +single-trace detail), panel (dashboard panel runs), background (rollups, +read-cache refresh, compaction-adjacent reads), and agent (LLM-driven queries). +Interactive and panel are latency-first and carry a deadline; admission sheds +them when the wait would exceed the deadline rather than queueing unboundedly. +Background and agent yield to interactive and panel. + +**Cost awareness.** Admit against thread and memory budgets, not only a +connection count, so one wide scan does not crowd out several cheap reads and +the pool reflects what statements actually consume. + +**One snapshot and connection per panel run.** A panel run pins one catalog +snapshot and takes one connection that all of its sub-reads share, so the run is +internally consistent and counts once against admission instead of once per +sub-read. + +**Separate background lanes.** Give the rollup pass and the read-cache refresh +their own admission so a long rollup does not block the read-cache refresh, +honoring the "completed-batch writes have their own gate and slot" intent. + +## Alternatives + +- **Widen the existing slot pool.** More connections without classes or cost + awareness defers the problem: a burst of wide scans still starves interactive + reads, and background work still competes directly. +- **Admit only at the HTTP layer.** Admitting requests before they reach the + engine misses the engine's own background work and the sub-reads a single + request fans out into, which is where the contention actually is. + +## Risks + +- A single admission layer is a central chokepoint; its own overhead and + fairness must be measured so it does not become the bottleneck it manages. +- Deadline-based shedding changes observed behavior: a read that used to queue + and eventually run may now be shed. The classes and deadlines must be chosen + so this only sheds work that would have missed its deadline anyway. +- A memory estimate per statement is approximate; a bad estimate could admit too + much (OOM) or too little (idle pool). Start from a coarse, conservative + estimate and measure. + +## Open questions + +- How is per-statement cost estimated before execution — from the compiled plan, + the file count and window, or a learned estimate? +- Do agent queries get their own class with a separate budget, or share the + panel class with a lower priority? +- How does this layer interact with the result cache's single-flight coalescing + and with the write gates, so a reader is not admitted twice or counted against + two budgets? + +## Research validation + +This proposal has not yet had the three-angle research pass (prior art, +in-codebase fit, performance) that precedes a build. Admission-control and +cost-based class scheduling have strong prior art (database workload management +and admission controllers) that must be surveyed and mapped onto the current +slot and gate machinery before implementation. This is a design-heavy change and +should not be built as a single pass without that validation. diff --git a/docs/proposals/README.md b/docs/proposals/README.md index 385fac3f..6579b786 100644 --- a/docs/proposals/README.md +++ b/docs/proposals/README.md @@ -38,5 +38,10 @@ Earlier proposals predate the template and keep their original structure. | 0004 | [Full UI without an LLM key, and PitotQL](0004-pitotql.md) | Accepted | | 0005 | [Dedicated attribute columns and batch format 5](0005-dedicated-attribute-columns.md) | Accepted | | 0006 | [Read performance under sustained ingest](0006-read-performance.md) | Draft | +| 0007 | [Additive per-batch rollup aggregates](0007-additive-rollup-aggregates.md) | Draft | +| 0008 | [Mergeable latency sketch and derived service rollup](0008-mergeable-latency-sketch.md) | Draft | +| 0009 | [Manifest catalog, write-ahead log, and parallel compaction](0009-manifest-catalog-wal-compaction.md) | Draft | +| 0010 | [Result-cache key normalization and an extent cache](0010-result-cache-extent-key.md) | Draft | +| 0011 | [One cost-aware admission layer at the engine boundary](0011-engine-admission-classes.md) | Draft | Settled decisions that a proposal leads to are recorded in [../adr](../adr/README.md).