Repository navigation
Conversation
Add support for atomically replacing data files (compaction). - MergingSnapshotProducer: shared engine for operations that both add and delete files, with manifest filtering and summary computation - ManifestFilterManager: rewrites existing manifests to remove deleted file entries, with encryption support - RewriteFilesAction: thin action layer using Operation::Replace - Refactor SnapshotProducer to extract write_snapshot() shared method - Allow Operation::Replace in snapshot summary computation Closes apache#1607 Related: apache#2242, apache#624
…tering Previously, ManifestFilterManager rejected manifests with non-default partition spec IDs with a FeatureUnsupported error. This broke tables that had undergone partition evolution, as old manifests carry their original spec ID. Now the filter manager resolves each manifest's own partition spec from table metadata and uses it for rewriting, allowing correct handling of partition-evolved tables. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace pub(crate) field access with accessor methods (has_added_data_files, has_deleted_data_files) to improve encapsulation. Downstream actions now use these methods instead of reaching into internal Vec fields directly. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add ManifestContentType param to new_manifest_writer for future delete manifest support - Remove unused snapshot_properties field from MergingSnapshotProducer - Move test helpers (make_data_file, append_files) to mod.rs as pub(crate) for reuse across test modules - Add starting_snapshot_id to RewriteFilesAction for future conflict detection support - Fix action-specific error message in generic engine Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…omission - test_rewrite_files_empty_table: rewrite on table with no snapshot - test_rewrite_files_removes_empty_manifest: verify empty manifests are omitted, not kept as empty files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Return explicit errors instead of silently defaulting to 0 for sequence_number and file_sequence_number in manifest entries - Use standard manifest naming scheme (<uuid>-m<counter>.avro) instead of custom names (-m-filter-, -m-added) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
`ManifestFilterManager::filter_manifests` built each surviving entry with status Existing and the original snapshot id and sequence numbers, then handed it to `ManifestWriter::add_entry`, which rewrites the status to Added, replaces the snapshot id with the new snapshot's, and clears the file sequence number. Survivors of a rewrite were therefore recorded as newly added files of the rewrite snapshot. Use `add_existing_entry`, which only sets the status and leaves the ids the filter already resolved, and drop its `#[allow(dead_code)]`.
Java's `RewriteFiles.dataSequenceNumber` lets a compaction record the sequence number of the files it replaces, so the rewritten file does not shadow deletes written concurrently at a higher sequence number. Without it the added files inherit the rewrite snapshot's own sequence number. `RewriteFilesAction::data_sequence_number` sets it, and `write_added_manifest` then writes the added files through `ManifestWriter::add_file`, which records the given number on the entry. A number above the one the new snapshot will carry is rejected at commit; on a V1 table that is 0, matching the V1 entry layout, which records no sequence number at all. The parameter is `i64`, not `u64`, to match `ManifestWriter::add_file` and every other sequence number in the crate.
`commit_snapshot` handed `SnapshotProducer` an empty added-file list and went straight to `commit_with_manifests`, so neither `validate_added_data_files` nor `validate_duplicate_files` ever saw the files a rewrite adds. A file carrying a partition value the default spec cannot hold, or a path that is already live, was written into a manifest and committed. Pass the added files to the producer and run both validations first, and drop `write_added_manifest`'s own content-type check, which `validate_added_data_files` now makes unreachable.
`commit_snapshot` created a fresh `(0..)` counter on every call while the
commit uuid stayed fixed for the life of the action, so a retried commit
rewrote `{uuid}-m0.avro` over the manifest the previous attempt had
written under the same name.
Java keeps `manifestCount` on the snapshot producer and never resets it.
The counter cannot live on Rust's `SnapshotProducer`: that is built fresh
inside each `commit_snapshot` call and so resets with it. The object that
survives a retry is the action, which the transaction re-invokes through
an `Arc`, so the counter goes on `MergingSnapshotProducer` as an
`AtomicU64` — `commit` only has `&self`. `fast_append` is unaffected: it
draws a new commit uuid per attempt unless the caller pins one.
The filter carried a manifest forward on `has_deleted_files()`, so a manifest whose entries are all Deleted stayed in every following snapshot for the rest of the table's life. Java keeps an existing manifest on `hasAddedFiles() || hasExistingFiles()`; its third arm matches the snapshot being produced, which no manifest read from the current snapshot can carry. The manifests this commit rewrites are produced by the filter pass and kept on their own.
`starting_snapshot_id` was recorded and never read. A rewrite planned against one snapshot and committed against a later one silently drops any delete written in between: the replacement file keeps the deleted rows. Java resolves this per file in `validateNoNewDeletesForDataFiles`, which needs a delete-file index this crate does not have yet. Until then the commit fails closed when the current snapshot carries a delete manifest with a sequence number above the starting snapshot's.
Every rewrite test used the V3 fixture, so nothing covered the V2 manifest and entry layout the action also writes. Both cases now run on V2 as well, through a V2 catalog fixture alongside the V3 one.
`cargo fmt` over the two new modules, and `public-api.txt` regenerated for `Transaction::rewrite_files`. The clippy warning on the unnecessary parentheses around the manifest counter went with the counter itself.
Resolves two conflicts from the transaction module re-export refactor on main: action types are now re-exported at `iceberg::transaction::*`, so `RewriteFilesAction` is exported the same way and `public-api.txt` records it under the flat path. Co-authored-by: Jegor van Opdorp <jegorvanopdorp@gmail.com>
V1 manifest entries carry no sequence numbers, and `ManifestEntryV1` deserialization fills both in as 0. Nothing in the rewrite path treats that as missing, so a V1 table compacts like V2 and V3 and the surviving entries keep Existing status, their snapshot id and their zero sequence numbers. Cover it so it stays that way.
`validate_added_data_files` is shared by fast append and rewrite, but its content-type error said "for fast append", so a rewrite that was handed a delete file reported the wrong operation and not which file was wrong. Drop the operation from the message and name the file and its content type instead.
`validate_no_new_deletes` fell back to sequence number 0 when the starting snapshot was no longer in the table, which let a rewrite commit when the table held no delete manifest at all and otherwise blamed an unrelated delete manifest for the rejection. An expired starting snapshot means no delete can be ruled out, so say that instead.
`validate_no_new_deletes` skipped the check whenever either the starting snapshot id or the current snapshot was absent. A missing starting snapshot id means the table held nothing when the rewrite was planned, so everything it holds at commit time, delete files included, arrived after that -- the case the check is least able to skip. Reject it, and keep returning early only when the table still has no snapshot, where the filter rejects the rewrite for its missing sources anyway.
The conflict check is table-wide and always on, so a compaction fails on a delete committed anywhere in the table while it ran, not only on the files it replaces. Say so on the action, along with the two other cases that cannot rule out a delete and what the check does not cover, so a caller can tell an expected rejection from a bug. Also stop the public docs linking to the private MergingSnapshotProducer, which rustdoc drops on the generated page.
The macro from apache#2928 landed while this branch was open and snapshot.rs already uses it. Use it for the merging producer and the rewrite action too, so the whole transaction module reads the same way.
The filter resolves each manifest's own partition spec, but every test ran on the default spec. Add an identity(ts) -> day(ts) fixture on V2 and V3: a survivor stays Existing in a spec-0 manifest with its timestamptz value and partition summary intact, one rewrite filters a manifest of each spec, and retiring the last old-spec file drops the spec-0 manifest with the chained totals still exact.
build_summary merged the filter's removal collector into a fresh one; merge drops partition metrics unless both sides trust them and neither did, so a rewrite never reported changed-partition-count or the partitions.* summaries. Add the new files to the filter's collector instead, and honour write.summary.partition-limit as fast append does.
The filter dropped the entries of the files it removed, so nothing in the metadata said which snapshot removed them. Java's expiry walks exactly those entries -- IncrementalFileCleanup finds a data file to delete only through a DELETED entry whose snapshot has expired -- so a table compacted by this action kept every replaced file in storage for good, and a changelog reader saw no removals. Read back with Trino, the manifests reported deleted_data_files_count = 0 while the snapshot summary claimed deleted-data-files = 2. Write each removed entry with add_delete_entry, which stamps the removing snapshot and keeps the file's own sequence numbers. A manifest the commit empties is now written rather than omitted, holding only those entries; the next merging commit drops it, as it already did for manifests left all-deleted by an earlier snapshot. Reported by @rexminnis against Apache Polaris with Trino and PyIceberg. Two tests change with the contract. The emptied manifest is no longer absent, so they assert it has no live entry and that a later rewrite drops it -- an append carries it along, here as in Java, because only a merging commit rebuilds the manifest list. A filtered manifest's partition summary now spans its deleted entries too, matching Java, where every entry updates it.
apache#3267 made UnboundPartitionField's fields private and gave it source-ids, so UnboundPartitionSpecBuilder::add_partition_field now takes the field rather than its parts. The spec-evolution fixture built both of its specs the old way. Git merges the two changes without a word and only the compiler notices, so the port is part of this merge rather than a commit after it. Co-authored-by: Rexwell Minnis <rexminnis@gmail.com>
…tion-successor # Conflicts: # crates/iceberg/public-api.txt # crates/iceberg/src/transaction/mod.rs # crates/iceberg/src/transaction/snapshot.rs
This was referenced Oct 30, 2026
Merges CTTY's branch and drops the producer this branch carried from apache#3046.
Contributor
Author
6 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #1607.
What changes are included in this PR?
Stacks on #3564. Adds
RewriteFilesActionon top of itsMergingSnapshotProducer, continuing #3046 by @hotcache with the remove-only rewrite from hotcache#3 that @JanKaul asked for. The producer this branch carried before is gone, so the diff against #3564 is the action, its tests and the wiring. When squashing please keep:Are these changes tested?
The
transaction::rewritetests from #3046 and hotcache#3, adapted to the producer from #3564, read the committed manifests back.