Repository navigation
feat(transaction): let a rewrite remove delete files, and fix rewrite conflicts and partition handling - #3
Conversation
`RewriteFilesAction::delete_delete_file` registers a position or equality delete file, or a deletion vector, to be removed from the table, as Java's `RewriteFiles.deleteFile(DeleteFile)` does. A delete-only rewrite commits a `replace` snapshot; the summary counts the removed delete files and lowers the delete totals. The manifest filter reads delete manifests when a delete file is to be removed. Data files are matched by path in data manifests only; delete files are matched in delete manifests only, by (path, content offset, content size), as Java's `DeleteFileWrapper` does, so removing one deletion vector keeps the other vectors in the same Puffin file. A rewritten manifest keeps its content type, so a filtered delete manifest stays a delete manifest. Survivors keep their snapshot id and sequence numbers, and removed entries are recorded as deleted with their own sequence numbers. `validate` refuses a delete file given to `delete_file` and a data file given to `delete_delete_file`, naming the file and the right method, so a misplaced file never removes a delete silently. As in Java's `BaseRewriteFiles`, it also refuses added data files when no data file is deleted. The duplicate check for added files reads no manifest when nothing is added, so a delete-only rewrite does not load every manifest of the table. Known limitation for format v3: a removed deletion vector is counted in the summary as a position delete file with its Puffin file size, while Java counts it under `removed-dvs` with its blob size. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… the replaced files The rewrite commit check refused a commit when any delete manifest had been added after the starting snapshot, anywhere in the table. A streaming writer that commits deletes every minute made nearly every longer compaction fail at commit. The check now follows Java's validateNoNewDeletesForDataFiles and refuses only for a new delete that can apply to a replaced data file. A delete is new when its data sequence number is above the starting snapshot's and it sits in a delete manifest whose sequence number is above the start's. Added and Existing entries both count, so a new delete still counts after a later snapshot rewrote its manifest, as a Java writer's delete-manifest merge does. Deletes that a later manifest rewrite keeps at their old sequence numbers, such as the survivors of delete_delete_file, do not count. A new delete conflicts when it is a position delete or deletion vector that names a replaced file, a partition-scoped position delete in the partition of a replaced file, or an equality delete in the partition of a replaced file or in an unpartitioned spec. Partitions are compared by spec id and partition values; values of different types count as equal, because after a type promotion the stored values of one partition differ in type. An equality delete is skipped when the action sets data_sequence_number, the delete is newer than that number, and the delete is in an unpartitioned spec or every added file is in its spec: the delete then still applies to the added files. This is stricter than Java's ignoreEqualityDeletes, which loses such a delete when the rewrite moves files to another spec. A delete whose spec is not in the table metadata is never skipped and counts as applying to the replaced files. Unlike Java, the check does not prune by column bounds. The rewrite is always refused, with or without new deletes, when its starting snapshot is not an ancestor of the current snapshot, as in Java, because sequence numbers order deletes only along one history. The refusals for a missing or expired starting snapshot are unchanged. A rewrite that only removes delete files is checked only for its starting snapshot. The error names the delete file and the replaced data file. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e pruning (apache#3483) * fix(expr): widen promoted partition values before comparison A partition column promoted int->long (or float->double) keeps its originally-written values in manifests produced before the promotion. The entry partition tuple is decoded under the manifest's own narrower type, while the scan predicate is bound to the current wider type. `Datum::partial_cmp`/`PartialEq` require matching types, so the `ExpressionEvaluator` compared e.g. an int partition value against a long predicate literal, got no ordering, and silently pruned the data file, dropping matching rows from filtered scans. The manifest-list `ManifestEvaluator` is unaffected: it decodes `FieldSummary` bounds with `Datum::try_from_bytes`, which already widens a 4-byte int bound to long. Mirror that on the entry path by widening the partition value to the reference's type before comparing. Decimals already compare across precisions, so only int->long and float->double need widening. Co-authored-by: Isaac <no-reply@databricks.com> * fix(expr): widen promoted column bounds in metrics pruning Companion to the partition-tuple widening: InclusiveMetricsEvaluator compares a data file's column bounds against the scan predicate, but those bounds are decoded under the manifest's embedded (narrower) schema while the predicate is bound to the current (wider) type. For an int column promoted to long (or float to double), the inequality and IN checks compared mismatched types, got no ordering, and discarded files that could match, dropping rows; the eq check failed open and merely skipped valid pruning. Widen each stored bound to the reference's type before comparing, via the same `Datum::promote_to` helper, in the inequality, eq, and IN paths. * fix(scan): widen promoted partition tuples at manifest read Replaces the comparison-time partition promotion with a reader-side projection, per review. The earlier comparison-time promotion only reached the pruning evaluators, so a promoted partition column still mismatched in the delete index: a data file's int partition never matched an eq delete or DV keyed by long, dropping or mis-applying deletes. Projecting the partition tuple fixes pruning and delete matching in one place. * fix(scan): tighten promotion gate, borrow unpromoted bounds, guard tuple length Addresses review feedback: - Gate `partition_types_differ` on the type pairs `promoted_partition` actually widens (int->long/unknown, float->double/unknown) instead of any primitive difference, so a decimal-precision widening or a dropped non-numeric source no longer deep-clones every manifest entry on every scan. - `InclusiveMetricsEvaluator::promote_bound` returns `Cow` and `Datum::promote_to` returns `Option`, so a bound that does not widen (e.g. a string or binary) is borrowed instead of cloned per data file during planning. - `DataFile::promoted_partition` returns `DataInvalid` when the stored tuple and the spec partition type have different lengths, instead of silently dropping trailing values (and the partition-scoped deletes they would match). - Drop the two delete-index unit tests that only re-exercised `promoted_partition` plus `Struct` equality, already covered by the data_file and scan-level tests. * test(scan): add end-to-end promotion scan regressions for deletes Adds plan_files scan regressions over a promoted partition column, routing an old-schema manifest through the reader-side widening: - A v3 deletion vector written after the promotion against the older data file. Without the widening the DV partition check errors (long(7) vs int(7)); the test guards that the data file's partition widens so the DV attaches. - An equality delete written before the promotion against the older data file, so both sides widen to long. Confirms the pre-existing match is preserved. Parameterizes the equality-delete fixture on whether the delete predates the promotion, and adds a v3 deletion-vector fixture. * fix(scan): copy dropped partition sources through unchanged A dropped partition source resolves to the Unknown primitive type. Widening its stored value to the promotion ceiling keyed on the backing literal, so a date (stored as an int) was silently promoted to long. The promotion gate skips a manifest whose only type difference is a dropped source, so a data manifest that widened a dropped date diverged from the delete manifest that left it alone, detaching the delete. Copy dropped-source values through unchanged and restrict the gate to genuine int to long and float to double promotions on live columns. * fix(scan): normalize dropped numeric partition sources by written type * Feedback --------- Co-authored-by: Isaac <no-reply@databricks.com> [Conflict in data_file.rs (imports and the test module, which the fork extends) resolved by keeping both sides.] (cherry picked from commit 5c27157)
…written with Removing a file from a manifest written before an int->long or float->double promotion of a partition source panicked in the snapshot summary and was then refused by the manifest writer, because the old `int` values were written under the current `long` type. The same refusal hit manifests whose spec uses a source column that was dropped later. Compaction of such a partition, or removal of a delete file listed in such a manifest, never committed. The filter now writes the rewritten manifest with the schema the manifest was written with, as `row_delta` and `overwrite` already do. Readers widen the values on read since apache#3483 (previous commit), and Java widens them through Avro schema resolution. Java writes under the current schema; for a dropped source column it then stores null and detaches partition-scoped deletes, which this avoids. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… spec as global deletes The delete file index treated an equality delete as global only when its partition tuple was empty. A table that dropped a partition field while on format v1 keeps it as a void transform, so equality deletes written after its upgrade to v2 have a tuple of one null and were applied only to files of the same spec. Java treats every spec without a non-void field as unpartitioned and applies such deletes to older files of every spec, so the Rust reader showed deleted rows, a rewrite wrote them back, and a dangling-delete pass would have removed such a delete because the planner attached it to no file. `DeleteFileContext` now carries the delete file's spec, and the index decides with `PartitionSpec::is_unpartitioned()`, the check the rewrite conflict check already uses. Without the spec it falls back to the tuple check. Position deletes stay scoped by spec and partition, as in Java. The change follows the open upstream PR apache#3449 line for line. When that lands, only the fork additions (three test literals and the null-identity test) remain. An extra test checks that an identity spec with a null partition value stays partition-scoped. Co-authored-by: JeonDaehong <daehong770@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…uality deletes Files added from Hive tables often do not store the identity partition column, and Flink upsert writers include partition columns in the equality fields. The equality-delete probe read the absent column as null, so these deletes never applied: the reader showed the deleted rows, and a rewrite wrote them back. An absent key column now probes as the file's identity partition value, the value the reader returns for that column, and as null only when there is none. Java fills the same constants before it applies equality deletes. A column the file does store is still read from the file, as for the returned rows. A uuid value is probed as fixed[16], the type the delete keys have after the Arrow round trip. Upstream: the hash-set probe comes from apache#2961 (open); apache#3302 fixes only the predicate path; the `initial-default` case is apache#3369 and is not included here. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… of a new equality delete When a rewrite keeps the starting sequence number, a newer equality delete applies to its added files. The check skipped such a delete when every added file was in the delete's spec. That is right for rows that were already in the delete's partition, but a rewrite across a spec change also moves rows from files of another spec into that partition. The delete did not apply to those rows before, because an equality delete in a partitioned spec only applies to files of its own spec and partition. After the rewrite it applied to them, so the rewrite changed the visible rows, and whether it did depended on whether the delete committed during the run or before it. Example: a table partitioned by `identity(p), bucket(4, id)` drops `p`; compaction rewrites the old files into `bucket(4, id)` while an upsert commits an equality delete in that spec. The moved row with the delete's key disappeared. The check now also refuses the rewrite when an added file is in the delete's partition and a replaced file of another spec can hold rows of that partition. A replaced file is ruled out only when both specs have a field with the same source and transform whose values certainly differ, for example another bucket of the same `bucket(4, id)`. Without an added file in the delete's partition no row can move into it, so the delete is skipped. The earlier rule stays: a delete in a replaced file's partition is refused when some added file is in another spec. This also closes the same hole when the added files are split across specs. Java's Spark `rewrite_data_files` shows both directions with default options: its check looks up new deletes only for the replaced files, so a delete in the new spec is never compared with the rows moved under it, and with the starting sequence number kept it skips equality deletes altogether, so a delete in the old spec stops applying to the moved rows. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…moted A snapshot summary renders each removed file's partition under the current schema. A value written before an int to long or float to double promotion keeps its narrower literal, and the datum reached unreachable!() for it, so a row delta or overwrite given a file read straight from an old manifest panicked. The value is now widened to the field type before it is rendered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… with RewriteManifests wrote every merged manifest with the current schema. A manifest written before a partition source promotion stores Int values, which the writer refuses under Long, so the rewrite never committed. Entries are now grouped by the schema of their manifest as well as by partition, and each new manifest is written with that schema, as the manifest filter of a rewrite does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ransforms The cross-spec equality delete check ruled out a replaced file only on a field with the same source and transform as the delete's. After a spec change between related transforms, such as bucket(8) to bucket(4) or day to hour, a rewrite mixing old and new files was refused for any equality delete in a partition of the new spec. It now also rules a file out when two bucket transforms of the same source differ modulo the gcd of their widths, or when two of year, month, day and hour of the same source differ in the coarser unit, and when one transform of a source is null and another is not. Values before 1970 are not compared across units, since writers have rounded them differently. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…planned snapshot The starting snapshot of a rewrite was always the current snapshot of the table its transaction was created from, and the added files inherited the new snapshot's sequence number unless the caller set one. A compaction that commits a later group after refreshing the table, or retries from a refreshed table, moved the start without noticing, so deletes committed in between were not checked. And with the inherited sequence number, every equality delete committed during the rewrite stopped applying to the rewritten rows, so the commit was refused under any concurrent upsert. The starting snapshot is now the snapshot the rewrite was planned against, by default the transaction's current one, and new RewriteFilesAction::validate_from_snapshot sets it explicitly. When data files are replaced, the added files get the starting snapshot's sequence number unless data_sequence_number sets one, so concurrent equality deletes keep applying and no longer refuse the commit. An explicit data sequence number below the starting snapshot's is refused: older equality deletes would then apply to rows of replaced files they did not apply to. V1 tables are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
I pushed four commits on top (
Compatibility: a rewrite that never set a sequence number now commits under concurrent equality deletes instead of being refused. The check still refuses every case where a delete would change which rows it applies to. No public item is removed. Not addressed: a removed deletion vector is still summarized as a position delete file with its Puffin file size (already listed as a known limitation). |
A rewrite can now remove delete files. The rewrite commit check now refuses a commit only for new deletes that apply to the replaced files. Four fixes make rewrites and scans handle partitions correctly after schema and spec changes. Review commit by commit.
Commits
RewriteFilesAction::delete_delete_fileremoves a position delete file, an equality delete file or a deletion vector, as Java'sRewriteFiles.deleteFile(DeleteFile)does. A rewrite that only removes delete files commits areplacesnapshot. Delete files are matched by path, content offset and content size, so removing one deletion vector keeps the other vectors in the same Puffin file. A delete file given todelete_file, or a data file given todelete_delete_file, is refused with an error that names the right method. Known limitation for format v3: a removed deletion vector is counted in the summary as a position delete file with its Puffin file size. Java counts it underremoved-dvswith its blob size. The action comes from feat(transaction): add MergingSnapshotProducer and RewriteFilesAction apache/iceberg-rust#3046 (open); upstream needs delete-only rewrites there first.validateNoNewDeletesForDataFiles. It refuses only for a new position delete or deletion vector that names a replaced file, or a new partition-scoped delete in the partition of a replaced file. Equality deletes newer thandata_sequence_numberare skipped when they still apply to the added files. Unlike Java, the check does not prune by column bounds. The rewrite is now always refused when its starting snapshot is not an ancestor of the current snapshot, as in Java.inttolongorfloattodoublepromotion. Without it, filtered scans pruned matching files, and deletes did not match data files from before the promotion. A conflict indata_file.rswas resolved by keeping both sides.row_deltaandoverwritealready do. Readers widen the values since fix(expr): widen promoted partition values and bounds to prevent false pruning apache/iceberg-rust#3483.PartitionSpec::is_unpartitioned(), as Java does. This follows fix(scan): apply equality deletes from an all-void partition spec globally apache/iceberg-rust#3449 (open) line for line, with JeonDaehong as co-author. One extra test checks that an identity spec with a null value stays partition-scoped.initial-defaultcase (Equality deletes ignore spec projection rules for dropped and newly added delete columns apache/iceberg-rust#3369) is not included.bucket(4, id). The conflict check above already refuses the opposite direction: a new delete in a replaced file's partition when an added file is in another spec. Java's Sparkrewrite_data_filescommits in both directions with default options. Its check looks up new deletes only for the replaced files, and with the starting sequence number kept it skips equality deletes.Tests
Each commit builds and passes
cargo test -p iceberg --all-features --lib. For each fix, the main tests fail without it. New tests:test_rewrite_files_removes_a_position_delete_file,..._removes_an_equality_delete_file,..._removes_one_deletion_vector_of_a_puffin_file,..._refuses_invalid_delete_file_removals,..._missing_delete_file,test_validate_duplicate_files_without_added_files_reads_no_manifest;test_rewrite_files_rejects_only_new_deletes_for_replaced_files(a table of 14 cases),..._ignores_delete_manifests_rewritten_since_the_start,..._rejects_new_deletes_in_rewritten_delete_manifests,..._rejects_starting_snapshot_off_the_branch,test_may_be_same_partition;test_rewrite_files_rewrites_a_manifest_written_before_a_promotion,test_rewrite_files_after_dropping_an_old_partition_source;test_void_spec_equality_delete_is_global,test_identity_spec_null_partition_equality_delete_stays_scoped;test_eq_delete_on_identity_partition_column_absent_from_data_file,test_eq_delete_on_uuid_partition_column_absent_from_data_file,test_eq_delete_on_absent_partition_column_with_unwidened_value;test_rewrite_files_checks_equality_deletes_across_specs(a table of 10 cases),test_may_hold_rows_of.The cherry-pick brings its upstream tests.
Compatibility
RewriteFilesAction::delete_delete_file. No public item is removed or changed.