Skip to content

feat(writer): add DeletionVectorWriter for v3 deletion vectors - #2

Merged
dina-kar merged 3 commits into
loams/dv-writerfrom
dv/writer
Oct 22, 2026
Merged

dina-kar merged 3 commits into
loams/dv-writerfrom
dv/writer

Conversation

@dina-kar

Copy link
Copy Markdown

What

Adds DeletionVectorWriter in writer/base_writer, the writer half of apache#2580. It builds on #1 (DeleteVector::serialize). This PR is stacked on #1 and will be retargeted to loams/dv-writer once #1 merges.

  • Input. It takes the same (file_path, pos) batches as PositionDeleteFileWriter (position_delete_schema), in any order, so a caller can choose a writer by format version. Positions are collected into one DeleteVector per referenced data file.
  • Output. On close, it writes one Puffin file per writer (one partition), with an uncompressed footer and one uncompressed deletion-vector-v1 blob per data file. The blob fields match Iceberg Java's BaseDVFileWriter: field _pos (2147483645), snapshot id and sequence number -1, and the referenced-data-file and cardinality properties. The file's created-by is Apache Iceberg Rust <version>. It returns one DataFile per blob, with content PositionDeletes, format Puffin, record_count set to the cardinality, referenced_data_file, content_offset, content_size_in_bytes and the partition.
  • merge_existing. It folds a data file's current DV into the new one, because a snapshot may hold only one DV per data file. Removing the old DV in the commit is the caller's job (RowDelta, feat(transaction): add RowDelta transaction action for row-level modi… apache/iceberg-rust#2203).
  • Small enablers, crate-internal. PuffinWriter::written_blobs_metadata gives each blob's offset and length. validate_position_delete_batch is now pub(crate) so both writers share the batch checks.

Tests

Unit tests in the module:

  • the blob metadata, offsets and decoded positions, read back with PuffinReader and DeleteVector::deserialize, including a position above 2^32;
  • merging an existing vector;
  • no file when nothing was deleted;
  • the partition and location of a partitioned writer;
  • errors for a negative position and for use after close;
  • ArrowReader applying the written vector to a Parquet file, located only through the returned DataFile.

As a mutation check, shifting content_offset by one fails two of the tests.

Spark-side tests come in the next PR. The round trip through Spark, Trino and Iceberg Java's RowDelta is checked outside this CI; the results are posted on that PR.

DeletionVectorWriter takes the same (file_path, pos) batches as
PositionDeleteFileWriter and, on close, writes one Puffin file with one
uncompressed deletion-vector-v1 blob per referenced data file, with the
blob fields and properties Iceberg Java writes (field _pos, snapshot id
and sequence number -1, referenced-data-file, cardinality). It returns
one DataFile per blob carrying referenced_data_file, content_offset and
content_size_in_bytes.

merge_existing folds a data file's current deletion vector into the new
one, since a snapshot may hold only one deletion vector per data file;
replacing the old one in the commit is left to the caller (RowDelta).

Refs apache#2580

Signed-off-by: Dinakaran <dinakaranvijayakumar@outlook.com>
@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a5f0ab84-dddf-4cff-8a06-6fbbbfbaae1d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@dina-kar

Copy link
Copy Markdown
Author

Round trip with Spark, Trino and iceberg-rust's reader (2026-10-22). Harness: https://github.com/ostrium-labs/loams/tree/he0-dv-writer-interop/spikes/he0-iceberg-v3/dv-writer. It was built against this branch at cace1a1.

Steps:

  1. Spark 4.0.4 with Iceberg 1.12.0 writes three v3 tables with DVs: dv (4 DVs), partitioned dv_part (8 DVs), and upgraded (v2 with position deletes, upgraded to v3, 8 DVs).
  2. Spark computes the expected answer: the rows that remain once positions p % 7 = 3 of every data file are deleted.
  3. DeletionVectorWriter merges each Spark-written DV (merge_existing) with those positions. That gives 25 DVs in 6 Puffin files, one writer per partition.
  4. Iceberg Java's RowDelta commits the new DVs and removes the old ones (validateFromSnapshot set to the snapshot the DVs were merged against).
  5. Spark, Trino 483 and iceberg-rust read each table, with and without a filter.

Results (rows, sum(id), sum(_row_id), sum(_row_id * id)):

case expected (Spark, before) Spark Trino iceberg-rust
dv 686 / 343490 / 343490 / 229202132 same same same
dv, id >= 500 343 / 257530 / 257530 / 200507656 same same same
dv_part 2214 / 3293154 / 3306450 / 6143797921 same same same
dv_part, id >= 1000 1467 / 2919770 / 2645388 / 5797377065 same same same
upgraded 762 / 422018 / 416818 / 268048684 same same same
upgraded, id >= 500 418 / 335846 / 296246 / 230669264 same same same

Notes:

@dina-kar
dina-kar changed the base branch from dv/serialize to loams/dv-writer October 22, 2026 02:33
@dina-kar
dina-kar merged commit f84860f into loams/dv-writer Oct 22, 2026
37 of 42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant