Skip to content

feat(transaction): let a rewrite remove delete files, and fix rewrite conflicts and partition handling - #3

Merged
JanKaul merged 11 commits into
lakekeeper-compaction-basefrom
ct/rewrite-remove-delete-files
Oct 26, 2026
Merged

JanKaul merged 11 commits into
lakekeeper-compaction-basefrom
ct/rewrite-remove-delete-files

Conversation

@c-thiel

@c-thiel c-thiel commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

A rewrite can now remove delete files. The rewrite commit check now refuses a commit only for new deletes that apply to the replaced files. Four fixes make rewrites and scans handle partitions correctly after schema and spec changes. Review commit by commit.

Commits

  • Let a rewrite remove delete files. New RewriteFilesAction::delete_delete_file removes a position delete file, an equality delete file or a deletion vector, as Java's RewriteFiles.deleteFile(DeleteFile) does. A rewrite that only removes delete files commits a replace snapshot. Delete files are matched by path, content offset and content size, so removing one deletion vector keeps the other vectors in the same Puffin file. A delete file given to delete_file, or a data file given to delete_delete_file, is refused with an error that names the right method. Known limitation for format v3: a removed deletion vector is counted in the summary as a position delete file with its Puffin file size. Java counts it under removed-dvs with its blob size. The action comes from feat(transaction): add MergingSnapshotProducer and RewriteFilesAction apache/iceberg-rust#3046 (open); upstream needs delete-only rewrites there first.
  • Reject a rewrite only for new deletes that apply to the replaced files. The check refused a commit when any delete manifest was added anywhere in the table after the starting snapshot. With a streaming writer that commits deletes every minute, almost every long compaction failed. The check now follows Java's validateNoNewDeletesForDataFiles. It refuses only for a new position delete or deletion vector that names a replaced file, or a new partition-scoped delete in the partition of a replaced file. Equality deletes newer than data_sequence_number are skipped when they still apply to the added files. Unlike Java, the check does not prune by column bounds. The rewrite is now always refused when its starting snapshot is not an ancestor of the current snapshot, as in Java.
  • Cherry-pick of fix(expr): widen promoted partition values and bounds to prevent false pruning apache/iceberg-rust#3483 (merged), author Anoop Johnson. Widens partition values and column bounds written before an int to long or float to double promotion. Without it, filtered scans pruned matching files, and deletes did not match data files from before the promotion. A conflict in data_file.rs was resolved by keeping both sides.
  • Rewrite a filtered manifest with the schema it was written with. Removing a file from a manifest written before a partition source promotion, or before its partition source column was dropped, panicked or was refused. Compaction of such a partition never committed. The filter now writes the manifest with its own schema, as row_delta and overwrite already do. Readers widen the values since fix(expr): widen promoted partition values and bounds to prevent false pruning apache/iceberg-rust#3483.
  • Apply equality deletes written under a void-only partition spec as global deletes. A table that dropped a partition field on format v1 keeps it as a void transform. Equality deletes written under that spec were applied only to files of the same spec, so the reader showed deleted rows and a rewrite wrote them back. The delete index now uses PartitionSpec::is_unpartitioned(), as Java does. This follows fix(scan): apply equality deletes from an all-void partition spec globally apache/iceberg-rust#3449 (open) line for line, with JeonDaehong as co-author. One extra test checks that an identity spec with a null value stays partition-scoped.
  • Fill absent identity partition columns before probing equality deletes. Files added from Hive tables often do not store the identity partition column. The equality-delete probe read that column as null, so deletes on it never applied. The probe now uses the file's identity partition value, as Java does. Related: perf(deletes): apply equality deletes via a hashed key set inside the parquet RowFilter apache/iceberg-rust#2961 (the hash-set probe, open) and fix(arrow): evaluate filters on columns missing from a data file against their identity partition value or initial-default apache/iceberg-rust#3302 (the predicate path only, open). The initial-default case (Equality deletes ignore spec projection rules for dropped and newly added delete columns apache/iceberg-rust#3369) is not included.
  • Refuse a rewrite that moves rows into the partition of a new equality delete. A rewrite can move rows from files of an older partition spec into files of the current spec, and it keeps the starting sequence number. An equality delete that a writer committed in the current spec during the rewrite did not apply to the old file, but it applied to the new file. So the moved rows disappeared, while the same delete committed before the rewrite started left them in place. The check now refuses the commit when an added file is in the delete's partition and a replaced file of another spec can hold rows of that partition. A replaced file is ruled out only when both specs have a field with the same source and transform whose values differ, for example another bucket of the same bucket(4, id). The conflict check above already refuses the opposite direction: a new delete in a replaced file's partition when an added file is in another spec. Java's Spark rewrite_data_files commits in both directions with default options. Its check looks up new deletes only for the replaced files, and with the starting sequence number kept it skips equality deletes.

Tests

Each commit builds and passes cargo test -p iceberg --all-features --lib. For each fix, the main tests fail without it. New tests:

  • removing delete files: test_rewrite_files_removes_a_position_delete_file, ..._removes_an_equality_delete_file, ..._removes_one_deletion_vector_of_a_puffin_file, ..._refuses_invalid_delete_file_removals, ..._missing_delete_file, test_validate_duplicate_files_without_added_files_reads_no_manifest;
  • conflict check: test_rewrite_files_rejects_only_new_deletes_for_replaced_files (a table of 14 cases), ..._ignores_delete_manifests_rewritten_since_the_start, ..._rejects_new_deletes_in_rewritten_delete_manifests, ..._rejects_starting_snapshot_off_the_branch, test_may_be_same_partition;
  • manifest schema: test_rewrite_files_rewrites_a_manifest_written_before_a_promotion, test_rewrite_files_after_dropping_an_old_partition_source;
  • void-only spec: test_void_spec_equality_delete_is_global, test_identity_spec_null_partition_equality_delete_stays_scoped;
  • absent partition column: test_eq_delete_on_identity_partition_column_absent_from_data_file, test_eq_delete_on_uuid_partition_column_absent_from_data_file, test_eq_delete_on_absent_partition_column_with_unwidened_value;
  • equality deletes across specs: test_rewrite_files_checks_equality_deletes_across_specs (a table of 10 cases), test_may_hold_rows_of.

The cherry-pick brings its upstream tests.

Compatibility

  • New public method RewriteFilesAction::delete_delete_file. No public item is removed or changed.
  • Rewrites that the old check refused now commit when the new deletes do not apply to the replaced files.
  • A rewrite whose starting snapshot is not an ancestor of the current snapshot is now refused, also without new deletes.
  • The error text for a conflicting delete changed. It now names the delete file and the replaced data file. Callers that match on the old text must be updated.
  • The manifest format does not change.

c-thiel and others added 6 commits October 25, 2026 11:07
`RewriteFilesAction::delete_delete_file` registers a position or equality delete file, or a deletion vector, to be removed from the table, as Java's `RewriteFiles.deleteFile(DeleteFile)` does. A delete-only rewrite commits a `replace` snapshot; the summary counts the removed delete files and lowers the delete totals.

The manifest filter reads delete manifests when a delete file is to be removed. Data files are matched by path in data manifests only; delete files are matched in delete manifests only, by (path, content offset, content size), as Java's `DeleteFileWrapper` does, so removing one deletion vector keeps the other vectors in the same Puffin file. A rewritten manifest keeps its content type, so a filtered delete manifest stays a delete manifest. Survivors keep their snapshot id and sequence numbers, and removed entries are recorded as deleted with their own sequence numbers.

`validate` refuses a delete file given to `delete_file` and a data file given to `delete_delete_file`, naming the file and the right method, so a misplaced file never removes a delete silently. As in Java's `BaseRewriteFiles`, it also refuses added data files when no data file is deleted.

The duplicate check for added files reads no manifest when nothing is added, so a delete-only rewrite does not load every manifest of the table.

Known limitation for format v3: a removed deletion vector is counted in the summary as a position delete file with its Puffin file size, while Java counts it under `removed-dvs` with its blob size.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… the replaced files

The rewrite commit check refused a commit when any delete manifest had been added after the starting snapshot, anywhere in the table. A streaming writer that commits deletes every minute made nearly every longer compaction fail at commit. The check now follows Java's validateNoNewDeletesForDataFiles and refuses only for a new delete that can apply to a replaced data file.

A delete is new when its data sequence number is above the starting snapshot's and it sits in a delete manifest whose sequence number is above the start's. Added and Existing entries both count, so a new delete still counts after a later snapshot rewrote its manifest, as a Java writer's delete-manifest merge does. Deletes that a later manifest rewrite keeps at their old sequence numbers, such as the survivors of delete_delete_file, do not count.

A new delete conflicts when it is a position delete or deletion vector that names a replaced file, a partition-scoped position delete in the partition of a replaced file, or an equality delete in the partition of a replaced file or in an unpartitioned spec. Partitions are compared by spec id and partition values; values of different types count as equal, because after a type promotion the stored values of one partition differ in type.

An equality delete is skipped when the action sets data_sequence_number, the delete is newer than that number, and the delete is in an unpartitioned spec or every added file is in its spec: the delete then still applies to the added files. This is stricter than Java's ignoreEqualityDeletes, which loses such a delete when the rewrite moves files to another spec. A delete whose spec is not in the table metadata is never skipped and counts as applying to the replaced files. Unlike Java, the check does not prune by column bounds.

The rewrite is always refused, with or without new deletes, when its starting snapshot is not an ancestor of the current snapshot, as in Java, because sequence numbers order deletes only along one history. The refusals for a missing or expired starting snapshot are unchanged. A rewrite that only removes delete files is checked only for its starting snapshot. The error names the delete file and the replaced data file.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e pruning (apache#3483)

* fix(expr): widen promoted partition values before comparison

A partition column promoted int->long (or float->double) keeps its
originally-written values in manifests produced before the promotion.
The entry partition tuple is decoded under the manifest's own narrower
type, while the scan predicate is bound to the current wider type.
`Datum::partial_cmp`/`PartialEq` require matching types, so the
`ExpressionEvaluator` compared e.g. an int partition value against a
long predicate literal, got no ordering, and silently pruned the data
file, dropping matching rows from filtered scans.

The manifest-list `ManifestEvaluator` is unaffected: it decodes
`FieldSummary` bounds with `Datum::try_from_bytes`, which already widens
a 4-byte int bound to long. Mirror that on the entry path by widening
the partition value to the reference's type before comparing. Decimals
already compare across precisions, so only int->long and float->double
need widening.

Co-authored-by: Isaac <no-reply@databricks.com>

* fix(expr): widen promoted column bounds in metrics pruning

Companion to the partition-tuple widening: InclusiveMetricsEvaluator
compares a data file's column bounds against the scan predicate, but
those bounds are decoded under the manifest's embedded (narrower) schema
while the predicate is bound to the current (wider) type. For an int
column promoted to long (or float to double), the inequality and IN
checks compared mismatched types, got no ordering, and discarded files
that could match, dropping rows; the eq check failed open and merely
skipped valid pruning.

Widen each stored bound to the reference's type before comparing, via
the same `Datum::promote_to` helper, in the inequality, eq, and IN paths.

* fix(scan): widen promoted partition tuples at manifest read

Replaces the comparison-time partition promotion with a reader-side
projection, per review.

The earlier comparison-time promotion only reached the pruning
evaluators, so a promoted partition column still mismatched in the delete
index: a data file's int partition never matched an eq delete or DV keyed
by long, dropping or mis-applying deletes. Projecting the partition tuple
fixes pruning and delete matching in one place.

* fix(scan): tighten promotion gate, borrow unpromoted bounds, guard tuple length

Addresses review feedback:
- Gate `partition_types_differ` on the type pairs `promoted_partition` actually
  widens (int->long/unknown, float->double/unknown) instead of any primitive
  difference, so a decimal-precision widening or a dropped non-numeric source no
  longer deep-clones every manifest entry on every scan.
- `InclusiveMetricsEvaluator::promote_bound` returns `Cow` and `Datum::promote_to`
  returns `Option`, so a bound that does not widen (e.g. a string or binary) is
  borrowed instead of cloned per data file during planning.
- `DataFile::promoted_partition` returns `DataInvalid` when the stored tuple and
  the spec partition type have different lengths, instead of silently dropping
  trailing values (and the partition-scoped deletes they would match).
- Drop the two delete-index unit tests that only re-exercised `promoted_partition`
  plus `Struct` equality, already covered by the data_file and scan-level tests.

* test(scan): add end-to-end promotion scan regressions for deletes

Adds plan_files scan regressions over a promoted partition column, routing an
old-schema manifest through the reader-side widening:
- A v3 deletion vector written after the promotion against the older data file.
  Without the widening the DV partition check errors (long(7) vs int(7)); the
  test guards that the data file's partition widens so the DV attaches.
- An equality delete written before the promotion against the older data file,
  so both sides widen to long. Confirms the pre-existing match is preserved.

Parameterizes the equality-delete fixture on whether the delete predates the
promotion, and adds a v3 deletion-vector fixture.

* fix(scan): copy dropped partition sources through unchanged

A dropped partition source resolves to the Unknown primitive type.
Widening its stored value to the promotion ceiling keyed on the backing
literal, so a date (stored as an int) was silently promoted to long.
The promotion gate skips a manifest whose only type difference is a
dropped source, so a data manifest that widened a dropped date diverged
from the delete manifest that left it alone, detaching the delete.

Copy dropped-source values through unchanged and restrict the gate to
genuine int to long and float to double promotions on live columns.

* fix(scan): normalize dropped numeric partition sources by written type

* Feedback

---------

Co-authored-by: Isaac <no-reply@databricks.com>
[Conflict in data_file.rs (imports and the test module, which the fork extends) resolved by keeping both sides.]
(cherry picked from commit 5c27157)
…written with

Removing a file from a manifest written before an int->long or float->double promotion of a partition source panicked in the snapshot summary and was then refused by the manifest writer, because the old `int` values were written under the current `long` type. The same refusal hit manifests whose spec uses a source column that was dropped later. Compaction of such a partition, or removal of a delete file listed in such a manifest, never committed.

The filter now writes the rewritten manifest with the schema the manifest was written with, as `row_delta` and `overwrite` already do. Readers widen the values on read since apache#3483 (previous commit), and Java widens them through Avro schema resolution. Java writes under the current schema; for a dropped source column it then stores null and detaches partition-scoped deletes, which this avoids.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… spec as global deletes

The delete file index treated an equality delete as global only when its partition tuple was empty. A table that dropped a partition field while on format v1 keeps it as a void transform, so equality deletes written after its upgrade to v2 have a tuple of one null and were applied only to files of the same spec. Java treats every spec without a non-void field as unpartitioned and applies such deletes to older files of every spec, so the Rust reader showed deleted rows, a rewrite wrote them back, and a dangling-delete pass would have removed such a delete because the planner attached it to no file.

`DeleteFileContext` now carries the delete file's spec, and the index decides with `PartitionSpec::is_unpartitioned()`, the check the rewrite conflict check already uses. Without the spec it falls back to the tuple check. Position deletes stay scoped by spec and partition, as in Java.

The change follows the open upstream PR apache#3449 line for line. When that lands, only the fork additions (three test literals and the null-identity test) remain. An extra test checks that an identity spec with a null partition value stays partition-scoped.

Co-authored-by: JeonDaehong <daehong770@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…uality deletes

Files added from Hive tables often do not store the identity partition column, and Flink upsert writers include partition columns in the equality fields. The equality-delete probe read the absent column as null, so these deletes never applied: the reader showed the deleted rows, and a rewrite wrote them back.

An absent key column now probes as the file's identity partition value, the value the reader returns for that column, and as null only when there is none. Java fills the same constants before it applies equality deletes. A column the file does store is still read from the file, as for the returned rows. A uuid value is probed as fixed[16], the type the delete keys have after the Arrow round trip.

Upstream: the hash-set probe comes from apache#2961 (open); apache#3302 fixes only the predicate path; the `initial-default` case is apache#3369 and is not included here.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
c-thiel and others added 5 commits October 26, 2026 08:48
… of a new equality delete

When a rewrite keeps the starting sequence number, a newer equality delete applies to its added files. The check skipped such a delete when every added file was in the delete's spec. That is right for rows that were already in the delete's partition, but a rewrite across a spec change also moves rows from files of another spec into that partition. The delete did not apply to those rows before, because an equality delete in a partitioned spec only applies to files of its own spec and partition. After the rewrite it applied to them, so the rewrite changed the visible rows, and whether it did depended on whether the delete committed during the run or before it. Example: a table partitioned by `identity(p), bucket(4, id)` drops `p`; compaction rewrites the old files into `bucket(4, id)` while an upsert commits an equality delete in that spec. The moved row with the delete's key disappeared.

The check now also refuses the rewrite when an added file is in the delete's partition and a replaced file of another spec can hold rows of that partition. A replaced file is ruled out only when both specs have a field with the same source and transform whose values certainly differ, for example another bucket of the same `bucket(4, id)`. Without an added file in the delete's partition no row can move into it, so the delete is skipped. The earlier rule stays: a delete in a replaced file's partition is refused when some added file is in another spec. This also closes the same hole when the added files are split across specs.

Java's Spark `rewrite_data_files` shows both directions with default options: its check looks up new deletes only for the replaced files, so a delete in the new spec is never compared with the rows moved under it, and with the starting sequence number kept it skips equality deletes altogether, so a delete in the old spec stops applying to the moved rows.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…moted

A snapshot summary renders each removed file's partition under the current
schema. A value written before an int to long or float to double promotion
keeps its narrower literal, and the datum reached unreachable!() for it, so a
row delta or overwrite given a file read straight from an old manifest
panicked. The value is now widened to the field type before it is rendered.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… with

RewriteManifests wrote every merged manifest with the current schema. A
manifest written before a partition source promotion stores Int values, which
the writer refuses under Long, so the rewrite never committed. Entries are now
grouped by the schema of their manifest as well as by partition, and each new
manifest is written with that schema, as the manifest filter of a rewrite does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ransforms

The cross-spec equality delete check ruled out a replaced file only on a field
with the same source and transform as the delete's. After a spec change between
related transforms, such as bucket(8) to bucket(4) or day to hour, a rewrite
mixing old and new files was refused for any equality delete in a partition of
the new spec. It now also rules a file out when two bucket transforms of the
same source differ modulo the gcd of their widths, or when two of year, month,
day and hour of the same source differ in the coarser unit, and when one
transform of a source is null and another is not. Values before 1970 are not
compared across units, since writers have rounded them differently.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…planned snapshot

The starting snapshot of a rewrite was always the current snapshot of the
table its transaction was created from, and the added files inherited the new
snapshot's sequence number unless the caller set one. A compaction that
commits a later group after refreshing the table, or retries from a refreshed
table, moved the start without noticing, so deletes committed in between were
not checked. And with the inherited sequence number, every equality delete
committed during the rewrite stopped applying to the rewritten rows, so the
commit was refused under any concurrent upsert.

The starting snapshot is now the snapshot the rewrite was planned against, by
default the transaction's current one, and new
RewriteFilesAction::validate_from_snapshot sets it explicitly. When data files
are replaced, the added files get the starting snapshot's sequence number
unless data_sequence_number sets one, so concurrent equality deletes keep
applying and no longer refuse the commit. An explicit data sequence number
below the starting snapshot's is refused: older equality deletes would then
apply to rows of replaced files they did not apply to. V1 tables are
unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JanKaul

JanKaul commented Oct 26, 2026

Copy link
Copy Markdown

I pushed four commits on top (d77380384..81619a6bc) that fix findings from a review of this PR. Each commit builds on its own, and cargo test -p iceberg --all-features --lib passes (2084 tests). Each fix has a test that fails without it.

  • c9bba4a6d fix(spec): render a partition value written before its source was promoted. The snapshot summary renders each removed file's partition under the current schema. A file read from a manifest written before an int → long or float → double promotion still carries Int/Float, and Datum's Display hit unreachable!(). This affected row delta and overwrite when given such a file; files from the scan planner were already widened. Transform::to_human_string now widens the value to the field type first.

  • 410b79339 fix(transaction): rewrite manifests with the schema they were written with. RewriteManifests wrote every merged manifest with the current schema. For manifests written before a partition source promotion, the writer refused Int under Long, so the rewrite never committed. This is the same failure that "rewrite a filtered manifest with the schema it was written with" fixed for the filter. Entries are now grouped by their manifest's schema and partition, and each group is written with that schema.

  • fa5f2c189 fix(transaction): rule out a replaced file across related partition transforms. The cross-spec equality-delete check only ruled out a replaced file on a field with the same transform. After bucket(8) → bucket(4) or day → hour, a rewrite mixing old and new files was refused for any equality delete in a new-spec partition. It now also rules a file out when:

    • two bucket values differ modulo the gcd of their bucket counts;
    • two of year/month/day/hour differ in the coarser unit;
    • one transform of a column is null and the other isn't.

    Values before 1970 are not compared across units, since writers have rounded them differently.

  • 81619a6bc feat(transaction): tie a rewrite's checks and sequence number to its planned snapshot. Before this, the starting snapshot was always the transaction's current snapshot. A compaction that refreshes the table between commit groups therefore moved its start, and stopped checking deletes committed in between. Also, the added files inherited the new snapshot's sequence number, so any concurrent upsert refused the commit.

    • New RewriteFilesAction::validate_from_snapshot(id) sets the planned snapshot. The default stays the transaction's current snapshot.
    • When data files are replaced, the added files now default to the starting snapshot's sequence number: the rewritten rows are data read at that snapshot. Concurrent equality deletes keep applying and no longer refuse the commit. Java leaves this pairing to RewriteDataFilesCommitManager (use-starting-sequence-number); here it's the action's default.
    • data_sequence_number(n) remains as an override. A value below the start's is refused, because older equality deletes would then apply to rows they never applied to.
    • MergingSnapshotProducer no longer stores the sequence number. The action works it out at commit and passes it to commit_snapshot.
    • public-api.txt gets the one new method. I added that line by hand, since cargo-public-api wasn't available, so make check-public-api should confirm it.

Compatibility: a rewrite that never set a sequence number now commits under concurrent equality deletes instead of being refused. The check still refuses every case where a delete would change which rows it applies to. No public item is removed.

Not addressed: a removed deletion vector is still summarized as a position delete file with its Puffin file size (already listed as a known limitation).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants