Skip to content

fix: snapshot summaries count only each snapshot's own files - #97

Merged
hasyimibhar merged 1 commit into
mainfrom
fix/iceberg-0.10
Oct 6, 2026
Merged

hasyimibhar merged 1 commit into
mainfrom
fix/iceberg-0.10

Conversation

@hasyimibhar

@hasyimibhar hasyimibhar commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Every snapshot's summary reported total-records, total-data-files, total-delete-files and total-equality-deletes as if it were the table's first. iceberg-rust 0.9 looked up the previous summary through the new, uncommitted snapshot's id. Separately, a compaction recorded no removed files, so once totals accumulated it would have kept counting everything it replaced.

The fix is in iceberg-rust, see polynya-dev/iceberg-rust#3. This PR points iceberg-rust to this and updates DST to ensure the issue will be caught in the future if it regressed.

Every snapshot's summary reported `total-records`, `total-data-files`
and the delete totals as if it were the table's first: iceberg-rust 0.9
looked up the previous summary through the new, uncommitted snapshot's
id. Engines take these as the table's size — ClickHouse 26.2 answered
`count()` with them (6 for a 3098-row table). And a compaction recorded
no removed files, so once totals accumulated it would have kept
counting everything it replaced.

The fork is rebased onto upstream v0.10.1, which has the summary fix
(apache/iceberg-rust#2391), plus a patch making rewrites subtract the
files they remove (polynya-dev/iceberg-rust#3). pg2iceberg pins
`polynya-patches-v0.10.1` and moves to its API: `replace_schema()`
for the fork's whole-schema action, `Table::manifest_list_reader`,
`OpenDalStorageFactory::S3` without `configured_scheme`, and a runtime
for the tables `ChainedAppendAction` builds (0.10's `TableBuilder`
requires one).

DST invariant 15: on the prod backend, the current snapshot's summary
totals equal what its manifests hold. The harness runs iceberg-rust on
a tokio runtime of its own, as 0.10 needs one and the harness drives
futures with pollster. Also fixes the harness's drain to quiescence,
which took a second worker's failed cycle (its commit response lost)
for an idle one and stopped before the retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hasyimibhar
hasyimibhar merged commit 89a9b54 into main Oct 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant