Repository navigation
Conversation
Add a core copy-on-write rewrite primitive that plans candidate data files, applies a caller-provided RecordBatch rewriter, writes replacement data files, and returns removed/added/unchanged file lists for future overwrite-style commit actions. - Plan candidates through the existing scan path so manifest pruning and delete-file application stay consistent with normal reads; the row predicate is cleared only for the full-file rewrite input. - Stream rewritten batches to the replacement writer: buffer only the prefix before the first changed batch, then open the writer lazily; unchanged files open no writer, fully-deleted files produce no replacement. - Bind the replacement writer and partition key to the planned snapshot's schema so rewrites survive schema evolution. - Build the replacement parquet writer via ParquetWriterBuilder::from_table_properties so replacement files honor the table's write.parquet.* properties, carry the table's encryption manager so encrypted tables are not downgraded to plaintext, and refuse to run when write.format.default is not parquet. - Keep the planner internal; CowRewriteBuilder is the only public entry and CowRewriteFile fields sit behind accessors. Cover delete-style, update-style, no-op, full-file delete, no-match delete, delete-file planning, writer edge cases, schema evolution, multi-source replacement path uniqueness, and partition-value preservation (including the null partition) end-to-end. Part of apache#2269. Co-Authored-By: Claude <noreply@anthropic.com>
- derive the effective change flag as `changed || output.is_none()` so a
rewriter returning {changed: false, output: None} cannot silently drop
rows while leaving the file marked unchanged; add regression tests
- treat a candidate whose visible rows are all removed by delete files as
removed with no replacement, so it can be compacted away with its
delete files instead of landing in unchanged_data_files forever
- honor write.object-storage.enabled when building replacement writers
(add the missing table property) so object-storage-layout tables keep
hash-entropy paths
- count output_rows only for rows actually written to replacement files
- derive Debug for CowBatchRewrite
- document the sync rewriter contract, the read-only CowRewriteFile
surface, and the prefix buffer's whole-file worst case
- drop avoidable FileScanTask/DataFile clones and replace the writer
expect() with a let-else invariant
A dropped batch could mark a file changed while skipping the Some(output) branch that initialized the writer and flushed kept rows. Open the writer whenever a changed file has buffered or current output, then flush the prefix independently of the current batch output. Cover the keep-first/drop-second data-loss case and full coverage by a real position-delete file; clarify delete-file and schema contracts.
… trade-offs Address the three non-blocking follow-ups from the approval review of apache#2752: - Surface the prefix-buffer memory ceiling in the public rewrite() rustdoc: a file that never changes (or whose first change is at EOF) buffers its entire decoded contents, one file at a time, which can be several GB for a compaction-sized file. - Add a debug-only partition guard: replacement files record the source partition verbatim, so a rewriter that mutates a partition-source column would silently write rows under a partition they do not belong to. In debug builds every emitted batch is checked against the source file's partition; release builds are unchanged. - Document in the module docs that keeping the source partition spec and staying unsorted is, like the snapshot-schema choice, a deliberate divergence from Java's RewriteDataFiles / Spark DML.
u70b3
marked this pull request as draft
September 29, 2026 06:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Follow-up to #2752, addressing the three non-blocking items @laskoviymishka raised in the approval review. The approved PR is left untouched; this stacks on top of it (the diff collapses to the single new commit once #2752 merges).
Quoted from the approval review:
The worst-case bound was only stated in an internal comment.
CowRewriteBuilder::rewrite()now carries a# Memory usagerustdoc section: a file that never changes — or whose first change sits at its very end — buffers its entire decoded contents, one file at a time, which can be several GB for a compaction-sized file.Added a
DebugPartitionGuard: in debug builds every emitted batch (including ones only buffered into the prefix) is checked against the source file's partition using the existingPartitionValueCalculator/arrow_struct_to_literalinfrastructure, and trips adebug_assert_eq!with a message naming the contract. Compiled out entirely in release builds, where per-row partition calculation would be too expensive for the rewrite hot path. A#[should_panic]regression test drives a rewriter that bumps an identity-partitioned column and asserts the guard fires; the existing partition-preservation end-to-end test exercises the guard's happy path.The module docs now state explicitly that replacement files mirror their source file's layout — planned snapshot's schema (no promotion), source partition spec (no repartition), unsorted (no sort order applied) — as a deliberate divergence from Java's RewriteDataFiles / Spark DML, with an opt-in promote-to-current-schema rewrite noted as follow-up work.
No public API changes (
public-api.txtunchanged); release behavior is byte-for-byte identical.Does this PR introduce any user-facing change?
No. Rustdoc additions on the COW rewrite primitive plus a debug-build-only assertion.
How was this patch tested?
cargo fmt --all -- --checkcargo clippy -p iceberg --all-targets -- -D warningscargo test -p iceberg— 1797 passed, 0 failed (plus doctests)cow_rewrite_partition_column_mutation_trips_debug_guard(should-panic regression test for item 2)