Skip to content

docs(proposals): engine redesign proposals 0007-0011 from the storage audit - #498

Draft
vishr wants to merge 3 commits into
mainfrom
docs/engine-redesign-proposals
Draft

vishr wants to merge 3 commits into
mainfrom
docs/engine-redesign-proposals

Conversation

@vishr

@vishr vishr commented Nov 16, 2026

Copy link
Copy Markdown
Member

Draft. Design proposals that came out of the read-only engine/storage audit (the same audit that produced issues #474–#487). These are for discussion, not implementation — each is Status: Draft and the three design-heavy ones carry a research-validation section; the two new ones (0010, 0011) still need their research pass.

Proposals

Also adds the missing index rows for 0007–0011 in docs/proposals/README.md.

No code changes.

Record three structural bets surfaced by the read-only engine audit as Draft
proposals, so the reasoning and trade-offs are reviewable before any build:

- 0007: additive per-batch rollup aggregates, generalizing the existing
  read_trace_parts model to the summary rollups so maintenance cost tracks new
  rows instead of the touched event-time span.
- 0008: derive service percentiles from the mergeable span_rollup histogram so
  alerts, overview and dashboards stop disagreeing, and drop the stored
  t-digest result.
- 0009: append-only manifest catalog, WAL with a head block, and parallel
  time-ordered compaction, with the freshness-guarantee decision called out as
  the gating question.

Each proposal names the related audit issues and marks itself measurement-gated
and incremental; the cheap in-place fixes ship first regardless.
…0009

Add a dated research-validation section to each engine-redesign proposal,
capturing a prior-art/best-in-class pass, an in-codebase correctness and
crash-safety pass, and a performance/sequencing pass:

- 0007: confirmed as ClickHouse AggregatingMergeTree (-State/-Merge); make the
  compaction fold mandatory in phase one, state the exactly-once and
  sum+count-not-avg invariants, and resolve the cross-gate snapshot design.
- 0008: confirmed as the Prometheus/OTel exponential-histogram approach; the
  shared helper must apply the served-kind filter, replicate the served-vs-all
  choice, pin interpolation, and handle the zero bucket; state the 4.33% bound
  and add an alert-migration note. Lowest risk; ship first.
- 0009: manifest/WAL/compaction each match best-in-class, but parallel
  compaction is not safe as drafted (single marker/attempt/recovery, overlapping
  input selection, and a one-merge memory budget all cause data loss or OOM);
  add manifest checkpointing and a WAL replay bound; incremental fixes ship
  first (BatchMetadata snapshot already landed in #488).

Proposals stay Draft; no production code changed.
…dmission; index 0007-0011

0010 addresses the result-cache key fragmentation and sub-refresh TTL (issue #477) with a normalized key and a watermark-split extent cache. 0011 addresses partial, cost-unaware read admission (issue #480) with one admission layer and interactive/panel/background/agent classes. Both are drafts and still need the three-angle research pass before build. Also adds the missing index rows for 0007-0011.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant