Skip to content

perf: one concurrent manifest scan per rewrite; Transaction::with_fresh_base - #4

Merged
hasyimibhar merged 2 commits into
polynya-patches-v0.10.1from
rewrite-single-scan
Oct 19, 2026
Merged

hasyimibhar merged 2 commits into
polynya-patches-v0.10.1from
rewrite-single-scan

Conversation

@hasyimibhar

Copy link
Copy Markdown

Two round-trip savings for pg2iceberg's commits, found tracing its TPROC-C benchmark against S3 and a REST catalog a request's latency away (25 ms / 50 ms).

  • RewriteFilesAction reads the table's manifests once, concurrently. It read the manifest list and every manifest twice per attempt (once for the files it removes, once for the manifests it keeps), one at a time. A table gains manifests with every commit until a rewrite drops them, so a compaction's commit spent seconds reading. Now one read serves both, 16 at a time on the runtime's IO pool, in the manifest list's order, still on every attempt against the table it would land on.
  • Transaction::with_fresh_base(), opt-in: a caller that just loaded the table skips the reload before the first attempt. If the table changed meanwhile, that attempt fails with a conflict, and the retry reloads and re-applies as every attempt does by default. Tests pin the loads: none with a fresh base, one to retry a conflict, one per attempt without.

In pg2iceberg's benchmark, a compaction's commit went from 7.6 s to 0.48 s on average (one table load, then its manifests read at once), and each commit saves a catalog round trip.

🤖 Generated with Claude Code

hasyimibhar and others added 2 commits October 19, 2026 10:39
RewriteFilesAction read the current snapshot's manifest list and every
manifest twice per commit attempt — once to find the files it removes,
once to decide which manifests to keep — one manifest at a time. Each is
a round trip to the object store, and a table gains manifests with
every commit until a rewrite drops them: with fifty of them, a rewrite's
commit spent seconds reading.

Now one read serves both, sixteen manifests at a time on the runtime's
IO pool, kept in the manifest list's order. It still runs on every
attempt, against the table the attempt would land on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every commit attempt reloads the table first, so a transaction built on
a stale table applies its actions to the current one. A caller that
loaded the table just before pays that round trip twice.
`with_fresh_base()` declares the base current: the first attempt
commits on it; if the table changed meanwhile, it fails with a conflict
and the retry reloads it, as every attempt does otherwise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hasyimibhar
hasyimibhar merged commit 55d6a3c into polynya-patches-v0.10.1 Oct 19, 2026
14 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant