Skip to content

feat(transaction): add RewriteFilesAction on MergingSnapshotProducer (stacks on #3564) - #3585

Open
jopdorp wants to merge 33 commits into
apache:mainfrom
jopdorp:feat/rewrite-files-action-successor
Open

jopdorp wants to merge 33 commits into
apache:mainfrom
jopdorp:feat/rewrite-files-action-successor

Conversation

@jopdorp

@jopdorp jopdorp commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #1607.

What changes are included in this PR?

Stacks on #3564. Adds RewriteFilesAction on top of its MergingSnapshotProducer, continuing #3046 by @hotcache with the remove-only rewrite from hotcache#3 that @JanKaul asked for. The producer this branch carried before is gone, so the diff against #3564 is the action, its tests and the wiring. When squashing please keep:

Co-authored-by: hotcache <coldcache.byeonghunhyeon@gmail.com>
Co-authored-by: Rexwell Minnis <rexminnis@gmail.com>

Are these changes tested?

The transaction::rewrite tests from #3046 and hotcache#3, adapted to the producer from #3564, read the committed manifests back.

hotcache and others added 30 commits August 22, 2026 08:42
Add support for atomically replacing data files (compaction).

- MergingSnapshotProducer: shared engine for operations that both add
  and delete files, with manifest filtering and summary computation
- ManifestFilterManager: rewrites existing manifests to remove deleted
  file entries, with encryption support
- RewriteFilesAction: thin action layer using Operation::Replace
- Refactor SnapshotProducer to extract write_snapshot() shared method
- Allow Operation::Replace in snapshot summary computation

Closes apache#1607
Related: apache#2242, apache#624
…tering

Previously, ManifestFilterManager rejected manifests with non-default
partition spec IDs with a FeatureUnsupported error. This broke tables
that had undergone partition evolution, as old manifests carry their
original spec ID.

Now the filter manager resolves each manifest's own partition spec
from table metadata and uses it for rewriting, allowing correct
handling of partition-evolved tables.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace pub(crate) field access with accessor methods
(has_added_data_files, has_deleted_data_files) to improve
encapsulation. Downstream actions now use these methods instead
of reaching into internal Vec fields directly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add ManifestContentType param to new_manifest_writer for future
  delete manifest support
- Remove unused snapshot_properties field from MergingSnapshotProducer
- Move test helpers (make_data_file, append_files) to mod.rs as
  pub(crate) for reuse across test modules
- Add starting_snapshot_id to RewriteFilesAction for future conflict
  detection support
- Fix action-specific error message in generic engine

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…omission

- test_rewrite_files_empty_table: rewrite on table with no snapshot
- test_rewrite_files_removes_empty_manifest: verify empty manifests
  are omitted, not kept as empty files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Return explicit errors instead of silently defaulting to 0 for
  sequence_number and file_sequence_number in manifest entries
- Use standard manifest naming scheme (<uuid>-m<counter>.avro)
  instead of custom names (-m-filter-, -m-added)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
`ManifestFilterManager::filter_manifests` built each surviving entry with
status Existing and the original snapshot id and sequence numbers, then
handed it to `ManifestWriter::add_entry`, which rewrites the status to
Added, replaces the snapshot id with the new snapshot's, and clears the
file sequence number. Survivors of a rewrite were therefore recorded as
newly added files of the rewrite snapshot.

Use `add_existing_entry`, which only sets the status and leaves the ids
the filter already resolved, and drop its `#[allow(dead_code)]`.
Java's `RewriteFiles.dataSequenceNumber` lets a compaction record the
sequence number of the files it replaces, so the rewritten file does not
shadow deletes written concurrently at a higher sequence number. Without
it the added files inherit the rewrite snapshot's own sequence number.

`RewriteFilesAction::data_sequence_number` sets it, and
`write_added_manifest` then writes the added files through
`ManifestWriter::add_file`, which records the given number on the entry.
A number above the one the new snapshot will carry is rejected at commit;
on a V1 table that is 0, matching the V1 entry layout, which records no
sequence number at all.

The parameter is `i64`, not `u64`, to match `ManifestWriter::add_file`
and every other sequence number in the crate.
`commit_snapshot` handed `SnapshotProducer` an empty added-file list and
went straight to `commit_with_manifests`, so neither
`validate_added_data_files` nor `validate_duplicate_files` ever saw the
files a rewrite adds. A file carrying a partition value the default spec
cannot hold, or a path that is already live, was written into a manifest
and committed.

Pass the added files to the producer and run both validations first, and
drop `write_added_manifest`'s own content-type check, which
`validate_added_data_files` now makes unreachable.
`commit_snapshot` created a fresh `(0..)` counter on every call while the
commit uuid stayed fixed for the life of the action, so a retried commit
rewrote `{uuid}-m0.avro` over the manifest the previous attempt had
written under the same name.

Java keeps `manifestCount` on the snapshot producer and never resets it.
The counter cannot live on Rust's `SnapshotProducer`: that is built fresh
inside each `commit_snapshot` call and so resets with it. The object that
survives a retry is the action, which the transaction re-invokes through
an `Arc`, so the counter goes on `MergingSnapshotProducer` as an
`AtomicU64` — `commit` only has `&self`. `fast_append` is unaffected: it
draws a new commit uuid per attempt unless the caller pins one.
The filter carried a manifest forward on `has_deleted_files()`, so a
manifest whose entries are all Deleted stayed in every following snapshot
for the rest of the table's life. Java keeps an existing manifest on
`hasAddedFiles() || hasExistingFiles()`; its third arm matches the
snapshot being produced, which no manifest read from the current snapshot
can carry. The manifests this commit rewrites are produced by the filter
pass and kept on their own.
`starting_snapshot_id` was recorded and never read. A rewrite planned
against one snapshot and committed against a later one silently drops any
delete written in between: the replacement file keeps the deleted rows.

Java resolves this per file in `validateNoNewDeletesForDataFiles`, which
needs a delete-file index this crate does not have yet. Until then the
commit fails closed when the current snapshot carries a delete manifest
with a sequence number above the starting snapshot's.
Every rewrite test used the V3 fixture, so nothing covered the V2
manifest and entry layout the action also writes. Both cases now run on
V2 as well, through a V2 catalog fixture alongside the V3 one.
`cargo fmt` over the two new modules, and `public-api.txt` regenerated
for `Transaction::rewrite_files`. The clippy warning on the unnecessary
parentheses around the manifest counter went with the counter itself.
Resolves two conflicts from the transaction module re-export refactor on
main: action types are now re-exported at `iceberg::transaction::*`, so
`RewriteFilesAction` is exported the same way and `public-api.txt` records
it under the flat path.

Co-authored-by: Jegor van Opdorp <jegorvanopdorp@gmail.com>
V1 manifest entries carry no sequence numbers, and `ManifestEntryV1`
deserialization fills both in as 0. Nothing in the rewrite path treats
that as missing, so a V1 table compacts like V2 and V3 and the surviving
entries keep Existing status, their snapshot id and their zero sequence
numbers. Cover it so it stays that way.
`validate_added_data_files` is shared by fast append and rewrite, but its
content-type error said "for fast append", so a rewrite that was handed a
delete file reported the wrong operation and not which file was wrong.
Drop the operation from the message and name the file and its content
type instead.
`validate_no_new_deletes` fell back to sequence number 0 when the starting
snapshot was no longer in the table, which let a rewrite commit when the
table held no delete manifest at all and otherwise blamed an unrelated
delete manifest for the rejection. An expired starting snapshot means no
delete can be ruled out, so say that instead.
`validate_no_new_deletes` skipped the check whenever either the starting
snapshot id or the current snapshot was absent. A missing starting
snapshot id means the table held nothing when the rewrite was planned, so
everything it holds at commit time, delete files included, arrived after
that -- the case the check is least able to skip. Reject it, and keep
returning early only when the table still has no snapshot, where the
filter rejects the rewrite for its missing sources anyway.
The conflict check is table-wide and always on, so a compaction fails on a
delete committed anywhere in the table while it ran, not only on the files
it replaces. Say so on the action, along with the two other cases that
cannot rule out a delete and what the check does not cover, so a caller
can tell an expected rejection from a bug.

Also stop the public docs linking to the private MergingSnapshotProducer,
which rustdoc drops on the generated page.
The macro from apache#2928 landed while this branch was open and snapshot.rs
already uses it. Use it for the merging producer and the rewrite action
too, so the whole transaction module reads the same way.
The filter resolves each manifest's own partition spec, but every test
ran on the default spec. Add an identity(ts) -> day(ts) fixture on V2
and V3: a survivor stays Existing in a spec-0 manifest with its
timestamptz value and partition summary intact, one rewrite filters a
manifest of each spec, and retiring the last old-spec file drops the
spec-0 manifest with the chained totals still exact.
build_summary merged the filter's removal collector into a fresh one;
merge drops partition metrics unless both sides trust them and neither
did, so a rewrite never reported changed-partition-count or the
partitions.* summaries. Add the new files to the filter's collector
instead, and honour write.summary.partition-limit as fast append does.
The filter dropped the entries of the files it removed, so nothing in the
metadata said which snapshot removed them. Java's expiry walks exactly
those entries -- IncrementalFileCleanup finds a data file to delete only
through a DELETED entry whose snapshot has expired -- so a table
compacted by this action kept every replaced file in storage for good,
and a changelog reader saw no removals. Read back with Trino, the
manifests reported deleted_data_files_count = 0 while the snapshot
summary claimed deleted-data-files = 2.

Write each removed entry with add_delete_entry, which stamps the removing
snapshot and keeps the file's own sequence numbers. A manifest the commit
empties is now written rather than omitted, holding only those entries;
the next merging commit drops it, as it already did for manifests left
all-deleted by an earlier snapshot.

Reported by @rexminnis against Apache Polaris with Trino and PyIceberg.

Two tests change with the contract. The emptied manifest is no longer
absent, so they assert it has no live entry and that a later rewrite
drops it -- an append carries it along, here as in Java, because only a
merging commit rebuilds the manifest list. A filtered manifest's
partition summary now spans its deleted entries too, matching Java, where
every entry updates it.
apache#3267 made UnboundPartitionField's fields private and gave it source-ids,
so UnboundPartitionSpecBuilder::add_partition_field now takes the field
rather than its parts. The spec-evolution fixture built both of its specs
the old way. Git merges the two changes without a word and only the
compiler notices, so the port is part of this merge rather than a commit
after it.

Co-authored-by: Rexwell Minnis <rexminnis@gmail.com>
…tion-successor

# Conflicts:
#	crates/iceberg/public-api.txt
#	crates/iceberg/src/transaction/mod.rs
#	crates/iceberg/src/transaction/snapshot.rs


Merges CTTY's branch and drops the producer this branch carried from apache#3046.
@jopdorp jopdorp changed the title feat(transaction): add MergingSnapshotProducer and RewriteFilesAction (continues #3046) feat(transaction): add RewriteFilesAction on MergingSnapshotProducer (stacks on #3564) Nov 3, 2026
@jopdorp

jopdorp commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Restacked on #3564 with 443a1f7. The producer this branch carried is gone, so the diff against that branch is the action, its tests and the wiring. Two producer gaps the rewrite tests ran into are on #3564.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants