Repository navigation
feat: optimize compaction catalog reads - #135
Merged
Merged
Conversation
Every commit — the materializer's, compaction's, expiry, a schema change — loaded the table to build the transaction, then iceberg-rust loaded it again before applying it: two catalog round trips. The transaction now declares the table it was built on fresh (`with_fresh_base`), so the first attempt commits on it; a conflict still reloads and retries. Also takes the fork's single concurrent manifest scan for rewrites: a compaction's commit read the table's every manifest twice, one at a time — seconds once a backlog's commits have piled manifests up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hasyimibhar
force-pushed
the
perf/commit-requests
branch
from
October 19, 2026 04:38
2ef9544 to
606dd48
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
iceberg-rust's rewrite read the table's manifest list and every manifest twice, one at a time, and a table gains two manifests with every materializer commit until a pass drops them. Every commit also loaded the table twice: once to build the transaction, once more in iceberg-rust before applying it.
This PR optimizes this:
polynya-patches-v0.10.1at 55d6a3c): a rewrite reads its manifests once, 16 at a time.with_fresh_base(), so the first attempt doesn't reload it; a conflict still reloads and retries.TPROC-C with cloud latency (
benchmark/tpcc,LATENCY=1),flush_rows: 100000, pinned to 2 cores:catalog.commit_compaction)catalog.commit_snapshots)