Skip to content

feat: optimize compaction catalog reads - #135

Merged
hasyimibhar merged 1 commit into
mainfrom
perf/commit-requests
Oct 19, 2026
Merged

hasyimibhar merged 1 commit into
mainfrom
perf/commit-requests

Conversation

@hasyimibhar

@hasyimibhar hasyimibhar commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

iceberg-rust's rewrite read the table's manifest list and every manifest twice, one at a time, and a table gains two manifests with every materializer commit until a pass drops them. Every commit also loaded the table twice: once to build the transaction, once more in iceberg-rust before applying it.

This PR optimizes this:

TPROC-C with cloud latency (benchmark/tpcc, LATENCY=1), flush_rows: 100000, pinned to 2 cores:

main this PR
compaction commit (catalog.commit_compaction) 7.6 s 0.48 s
materializer commit (catalog.commit_snapshots) 378 ms 355 ms
time the materializer made no progress, catching up 19% 0%
rows materialized / s, catching up 47.8k 71.0k

Every commit — the materializer's, compaction's, expiry, a schema change
— loaded the table to build the transaction, then iceberg-rust loaded it
again before applying it: two catalog round trips. The transaction now
declares the table it was built on fresh (`with_fresh_base`), so the
first attempt commits on it; a conflict still reloads and retries.

Also takes the fork's single concurrent manifest scan for rewrites: a
compaction's commit read the table's every manifest twice, one at a
time — seconds once a backlog's commits have piled manifests up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hasyimibhar hasyimibhar changed the title perf: a commit loads its table once; compaction commits scan manifests once feat: optimize compaction catalog reads Oct 19, 2026
@hasyimibhar
hasyimibhar merged commit b8d4f7e into main Oct 19, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant