Repository navigation
fix: snapshot summaries count only each snapshot's own files - #97
Merged
Merged
Conversation
Every snapshot's summary reported `total-records`, `total-data-files` and the delete totals as if it were the table's first: iceberg-rust 0.9 looked up the previous summary through the new, uncommitted snapshot's id. Engines take these as the table's size — ClickHouse 26.2 answered `count()` with them (6 for a 3098-row table). And a compaction recorded no removed files, so once totals accumulated it would have kept counting everything it replaced. The fork is rebased onto upstream v0.10.1, which has the summary fix (apache/iceberg-rust#2391), plus a patch making rewrites subtract the files they remove (polynya-dev/iceberg-rust#3). pg2iceberg pins `polynya-patches-v0.10.1` and moves to its API: `replace_schema()` for the fork's whole-schema action, `Table::manifest_list_reader`, `OpenDalStorageFactory::S3` without `configured_scheme`, and a runtime for the tables `ChainedAppendAction` builds (0.10's `TableBuilder` requires one). DST invariant 15: on the prod backend, the current snapshot's summary totals equal what its manifests hold. The harness runs iceberg-rust on a tokio runtime of its own, as 0.10 needs one and the harness drives futures with pollster. Also fixes the harness's drain to quiescence, which took a second worker's failed cycle (its commit response lost) for an idle one and stopped before the retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every snapshot's summary reported
total-records,total-data-files,total-delete-filesandtotal-equality-deletesas if it were the table's first. iceberg-rust 0.9 looked up the previous summary through the new, uncommitted snapshot's id. Separately, a compaction recorded no removed files, so once totals accumulated it would have kept counting everything it replaced.The fix is in iceberg-rust, see polynya-dev/iceberg-rust#3. This PR points iceberg-rust to this and updates DST to ensure the issue will be caught in the future if it regressed.