Repository navigation
perf: one concurrent manifest scan per rewrite; Transaction::with_fresh_base - #4
Merged
Merged
Conversation
RewriteFilesAction read the current snapshot's manifest list and every manifest twice per commit attempt — once to find the files it removes, once to decide which manifests to keep — one manifest at a time. Each is a round trip to the object store, and a table gains manifests with every commit until a rewrite drops them: with fifty of them, a rewrite's commit spent seconds reading. Now one read serves both, sixteen manifests at a time on the runtime's IO pool, kept in the manifest list's order. It still runs on every attempt, against the table the attempt would land on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every commit attempt reloads the table first, so a transaction built on a stale table applies its actions to the current one. A caller that loaded the table just before pays that round trip twice. `with_fresh_base()` declares the base current: the first attempt commits on it; if the table changed meanwhile, it fails with a conflict and the retry reloads it, as every attempt does otherwise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two round-trip savings for pg2iceberg's commits, found tracing its TPROC-C benchmark against S3 and a REST catalog a request's latency away (25 ms / 50 ms).
RewriteFilesActionreads the table's manifests once, concurrently. It read the manifest list and every manifest twice per attempt (once for the files it removes, once for the manifests it keeps), one at a time. A table gains manifests with every commit until a rewrite drops them, so a compaction's commit spent seconds reading. Now one read serves both, 16 at a time on the runtime's IO pool, in the manifest list's order, still on every attempt against the table it would land on.Transaction::with_fresh_base(), opt-in: a caller that just loaded the table skips the reload before the first attempt. If the table changed meanwhile, that attempt fails with a conflict, and the retry reloads and re-applies as every attempt does by default. Tests pin the loads: none with a fresh base, one to retry a conflict, one per attempt without.In pg2iceberg's benchmark, a compaction's commit went from 7.6 s to 0.48 s on average (one table load, then its manifests read at once), and each commit saves a catalog round trip.
🤖 Generated with Claude Code