Skip to content

feat: optimize manifest writing on commit - #137

Merged
hasyimibhar merged 1 commit into
mainfrom
perf/staged-commit
Oct 19, 2026
Merged

hasyimibhar merged 1 commit into
mainfrom
perf/staged-commit

Conversation

@hasyimibhar

@hasyimibhar hasyimibhar commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

A commit of several steps chained fast appends one step after another: each wrote its manifests, then its manifest list, then read that list back for the next step. Four round trips to object storage per step, and committing was 43% of the materializer's time.

Now each step's snapshot is planned in memory (polynya-dev/iceberg-rust#5, FastAppendAction::stage; polynya-patches-v0.10.1 at 0ded76c), then every step's manifests are written at once, then every step's manifest list, each naming its parent's manifests and its own: one read of the table's manifest list and two rounds of writes, however many steps.

TPROC-C with cloud latency (benchmark/tpcc, LATENCY=1), flush_rows: 100000, pinned to 2 cores, similar source load (~90k changes/s):

main this PR
materializer commit (commit_snapshots) 679 ms 354 ms
commits' share of materializer cycle time 43% 28%
rows materialized / s, under load 52.3k 61.0k
rows materialized / s, catching up 88.7k 101.3k
caught up after the load 130 s 75 s

A commit of several steps — a large transaction, or a materializer
catching up — chained fast appends, one step after another: each wrote
its manifests, then its manifest list, then read that list back for the
next step. Four round trips to object storage a step, while committing
was most of what the materializer did.

Now each step's snapshot is planned in memory (the fork's
`FastAppendAction::stage`), then every step's manifests are written at
once, then every step's manifest list, each naming its parent's
manifests and its own: a read of the table's manifest list and two
rounds of writes, however many steps. V3 tables, whose snapshots take
their row range from the written list, still commit step by step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hasyimibhar hasyimibhar changed the title perf: a commit writes its steps' manifests together feat: optimize manifest writing on commit Oct 19, 2026
@hasyimibhar
hasyimibhar merged commit f99dc05 into main Oct 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant