Skip to content

fix(cow_rewrite): guard partition preservation and document rewrite() trade-offs - #3296

Closed
u70b3 wants to merge 4 commits into
apache:mainfrom
u70b3:fix/cow-rewrite-post-approval-followups
Closed

u70b3 wants to merge 4 commits into
apache:mainfrom
u70b3:fix/cow-rewrite-post-approval-followups

Conversation

@u70b3

@u70b3 u70b3 commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Why are these changes needed?

Follow-up to #2752, addressing the three non-blocking items @laskoviymishka raised in the approval review. The approved PR is left untouched; this stacks on top of it (the diff collapses to the single new commit once #2752 merges).

Quoted from the approval review:

  1. The prefix buffer can hold a whole decoded file in memory when a file never changes or its first change is near EOF. You've already noted the size-capped fallback as future work in the code — I'd just surface that memory ceiling in the public rewrite() docs so consumers compacting multi-GB files know the bound up front.

    The worst-case bound was only stated in an internal comment. CowRewriteBuilder::rewrite() now carries a # Memory usage rustdoc section: a file that never changes — or whose first change sits at its very end — buffers its entire decoded contents, one file at a time, which can be several GB for a compaction-sized file.

  2. Replacement partition metadata is trusted verbatim from the source file. A rewriter that mutates a partition-source column would silently record a partition that disagrees with its rows; a debug-assert (or making the commit adapter responsible for validation) would keep that a footgun rather than a corruption path.

    Added a DebugPartitionGuard: in debug builds every emitted batch (including ones only buffered into the prefix) is checked against the source file's partition using the existing PartitionValueCalculator / arrow_struct_to_literal infrastructure, and trips a debug_assert_eq! with a message naming the contract. Compiled out entirely in release builds, where per-row partition calculation would be too expensive for the rewrite hot path. A #[should_panic] regression test drives a rewriter that bumps an identity-partitioned column and asserts the guard fires; the existing partition-preservation end-to-end test exercises the guard's happy path.

  3. The sort-order and partition-spec choices are the same kind of divergence (replacements stay unsorted and keep the source spec, no repartition) and deserve a matching doc line.

    The module docs now state explicitly that replacement files mirror their source file's layout — planned snapshot's schema (no promotion), source partition spec (no repartition), unsorted (no sort order applied) — as a deliberate divergence from Java's RewriteDataFiles / Spark DML, with an opt-in promote-to-current-schema rewrite noted as follow-up work.

No public API changes (public-api.txt unchanged); release behavior is byte-for-byte identical.

Does this PR introduce any user-facing change?

No. Rustdoc additions on the COW rewrite primitive plus a debug-build-only assertion.

How was this patch tested?

  • cargo fmt --all -- --check
  • cargo clippy -p iceberg --all-targets -- -D warnings
  • cargo test -p iceberg — 1797 passed, 0 failed (plus doctests)
  • New: cow_rewrite_partition_column_mutation_trips_debug_guard (should-panic regression test for item 2)

u70b3 and others added 4 commits September 24, 2026 09:13
Add a core copy-on-write rewrite primitive that plans candidate data
files, applies a caller-provided RecordBatch rewriter, writes
replacement data files, and returns removed/added/unchanged file lists
for future overwrite-style commit actions.

- Plan candidates through the existing scan path so manifest pruning
  and delete-file application stay consistent with normal reads; the
  row predicate is cleared only for the full-file rewrite input.
- Stream rewritten batches to the replacement writer: buffer only the
  prefix before the first changed batch, then open the writer lazily;
  unchanged files open no writer, fully-deleted files produce no
  replacement.
- Bind the replacement writer and partition key to the planned
  snapshot's schema so rewrites survive schema evolution.
- Build the replacement parquet writer via
  ParquetWriterBuilder::from_table_properties so replacement files
  honor the table's write.parquet.* properties, carry the table's
  encryption manager so encrypted tables are not downgraded to
  plaintext, and refuse to run when write.format.default is not
  parquet.
- Keep the planner internal; CowRewriteBuilder is the only public
  entry and CowRewriteFile fields sit behind accessors.

Cover delete-style, update-style, no-op, full-file delete, no-match
delete, delete-file planning, writer edge cases, schema evolution,
multi-source replacement path uniqueness, and partition-value
preservation (including the null partition) end-to-end.

Part of apache#2269.

Co-Authored-By: Claude <noreply@anthropic.com>
- derive the effective change flag as `changed || output.is_none()` so a
  rewriter returning {changed: false, output: None} cannot silently drop
  rows while leaving the file marked unchanged; add regression tests
- treat a candidate whose visible rows are all removed by delete files as
  removed with no replacement, so it can be compacted away with its
  delete files instead of landing in unchanged_data_files forever
- honor write.object-storage.enabled when building replacement writers
  (add the missing table property) so object-storage-layout tables keep
  hash-entropy paths
- count output_rows only for rows actually written to replacement files
- derive Debug for CowBatchRewrite
- document the sync rewriter contract, the read-only CowRewriteFile
  surface, and the prefix buffer's whole-file worst case
- drop avoidable FileScanTask/DataFile clones and replace the writer
  expect() with a let-else invariant
A dropped batch could mark a file changed while skipping the Some(output) branch that initialized the writer and flushed kept rows. Open the writer whenever a changed file has buffered or current output, then flush the prefix independently of the current batch output. Cover the keep-first/drop-second data-loss case and full coverage by a real position-delete file; clarify delete-file and schema contracts.
… trade-offs

Address the three non-blocking follow-ups from the approval review of
apache#2752:

- Surface the prefix-buffer memory ceiling in the public rewrite()
  rustdoc: a file that never changes (or whose first change is at EOF)
  buffers its entire decoded contents, one file at a time, which can be
  several GB for a compaction-sized file.
- Add a debug-only partition guard: replacement files record the source
  partition verbatim, so a rewriter that mutates a partition-source
  column would silently write rows under a partition they do not belong
  to. In debug builds every emitted batch is checked against the source
  file's partition; release builds are unchanged.
- Document in the module docs that keeping the source partition spec and
  staying unsorted is, like the snapshot-schema choice, a deliberate
  divergence from Java's RewriteDataFiles / Spark DML.
@u70b3
u70b3 marked this pull request as draft September 29, 2026 06:51
@u70b3 u70b3 closed this Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant