Skip to content

feat: write a v3 table's multi-step commits together - #151

Merged
hasyimibhar merged 2 commits into
mainfrom
perf/v3-chained-append
Nov 17, 2026
Merged

hasyimibhar merged 2 commits into
mainfrom
perf/v3-chained-append

Conversation

@hasyimibhar

@hasyimibhar hasyimibhar commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

A commit of several steps writes its snapshots together on v3 tables now, as it already did on v2. That covers a materializer catching up, or a transaction too big for one step.

Until now ChainedAppendAction fell back to committing a v3 table's steps one at a time: about four sequential object-store round trips per snapshot, versus two parallel rounds for the whole chain. The fork's staged snapshots couldn't record v3 row ranges. polynya-dev/iceberg-rust#8 adds that: each staged snapshot's rows take the row ids after the last's, and its data manifest keeps its ids when a later list carries it.

One case still commits one by one: a v3 table's first multi-step commit after an upgrade with data in it. That commit's first list gives the pre-upgrade files row ids too, a range a staged step can't record. ChainedAppendAction checks the parent's manifests for a data manifest without ids first.

hasyimibhar and others added 2 commits November 17, 2026 03:45
`ChainedAppendAction` wrote a v3 table's steps one by one — four
sequential round trips to object storage each — since the fork's staged
snapshots couldn't record v3 row ranges. polynya-dev/iceberg-rust#8 lets
them: each step's rows take the row ids after the last's.

A v3 table's first such commit since an upgrade with data still goes one
by one: its first list gives the files from before the upgrade row ids
too, a range a staged step can't record.

Pins it: the catalog test of a chain's round trips on v3 (and its row
ids), and DST cases of a several-step commit on v3 and right after an
upgrade, on the sim and iceberg-rust.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
polynya-dev/iceberg-rust#8 merged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hasyimibhar hasyimibhar changed the title perf: write a v3 table's multi-step commits together feat: write a v3 table's multi-step commits together Nov 17, 2026
@hasyimibhar
hasyimibhar merged commit d7495bc into main Nov 17, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant