Repository navigation
feat(writer): add DeletionVectorWriter for v3 deletion vectors - #2
Conversation
DeletionVectorWriter takes the same (file_path, pos) batches as PositionDeleteFileWriter and, on close, writes one Puffin file with one uncompressed deletion-vector-v1 blob per referenced data file, with the blob fields and properties Iceberg Java writes (field _pos, snapshot id and sequence number -1, referenced-data-file, cardinality). It returns one DataFile per blob carrying referenced_data_file, content_offset and content_size_in_bytes. merge_existing folds a data file's current deletion vector into the new one, since a snapshot may hold only one deletion vector per data file; replacing the old one in the commit is left to the caller (RowDelta). Refs apache#2580 Signed-off-by: Dinakaran <dinakaranvijayakumar@outlook.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Signed-off-by: Dinakaran <dinakaranvijayakumar@outlook.com>
|
Round trip with Spark, Trino and iceberg-rust's reader (2026-10-22). Harness: https://github.com/ostrium-labs/loams/tree/he0-dv-writer-interop/spikes/he0-iceberg-v3/dv-writer. It was built against this branch at Steps:
Results (rows,
Notes:
|
What
Adds
DeletionVectorWriterinwriter/base_writer, the writer half of apache#2580. It builds on #1 (DeleteVector::serialize). This PR is stacked on #1 and will be retargeted toloams/dv-writeronce #1 merges.(file_path, pos)batches asPositionDeleteFileWriter(position_delete_schema), in any order, so a caller can choose a writer by format version. Positions are collected into oneDeleteVectorper referenced data file.close, it writes one Puffin file per writer (one partition), with an uncompressed footer and one uncompresseddeletion-vector-v1blob per data file. The blob fields match Iceberg Java'sBaseDVFileWriter: field_pos(2147483645), snapshot id and sequence number-1, and thereferenced-data-fileandcardinalityproperties. The file'screated-byisApache Iceberg Rust <version>. It returns oneDataFileper blob, with contentPositionDeletes, formatPuffin,record_countset to the cardinality,referenced_data_file,content_offset,content_size_in_bytesand the partition.merge_existing. It folds a data file's current DV into the new one, because a snapshot may hold only one DV per data file. Removing the old DV in the commit is the caller's job (RowDelta, feat(transaction): add RowDelta transaction action for row-level modi… apache/iceberg-rust#2203).PuffinWriter::written_blobs_metadatagives each blob's offset and length.validate_position_delete_batchis nowpub(crate)so both writers share the batch checks.Tests
Unit tests in the module:
PuffinReaderandDeleteVector::deserialize, including a position above 2^32;ArrowReaderapplying the written vector to a Parquet file, located only through the returnedDataFile.As a mutation check, shifting
content_offsetby one fails two of the tests.Spark-side tests come in the next PR. The round trip through Spark, Trino and Iceberg Java's RowDelta is checked outside this CI; the results are posted on that PR.