You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'd like to work on the deletion-vector writer half of this issue, building on #3232 (DeleteVector::serialize). I'll leave the commit half to #2203 (RowDelta). @vikas-magar, you offered to work on this in July. If you have something in progress, please say so and I'll review it or help with it instead.
Plan: a DeletionVectorWriter in writer/base_writer, next to PositionDeleteFileWriter, with the same (file_path, pos) input, so callers can choose by format version.
It collects the positions per referenced data file into a DeleteVector. On close, it writes one Puffin file with one deletion-vector-v1 blob per data file. The blobs are uncompressed, use field 2147483645 (_pos) and snapshot id and sequence number -1, and carry the referenced-data-file and cardinality properties, as Iceberg Java does.
It returns one DataFile per blob. Each has content PositionDeletes, format Puffin, referenced_data_file, content_offset, content_size_in_bytes, record_count set to the cardinality, and the partition.
The spec allows at most one DV per data file per snapshot, so the writer also takes a data file's existing DV and merges it into the new one. The caller then removes the old DV in the same commit (via RowDelta).
Tests:
Unit tests read the Puffin file back with PuffinReader and DeleteVector::deserialize, and scan a data file through ArrowReader with the written DV.
An integration test uses a Spark-provisioned v3 table with DVs. It reads Spark's blobs, merges new positions into them, and checks the result.
Outside this repository's CI, we check that Spark 4.0.4 (Iceberg 1.12.0) and Trino 483 read tables whose DVs were written this way, committed through Iceberg Java's RowDelta.
The work is on https://github.com/ostrium-labs/iceberg-rust/tree/loams/dv-writer, in small PRs. I'll open upstream PRs against this issue once each piece is green, after #3232 merges, and keep each one under the size guideline. Feedback on the shape of the API is welcome before then.
Activity
vikas-magar commentedon Jul 15, 2026
Hi, I can work on this issue.
dina-kar commentedon Oct 21, 2026
I'd like to work on the deletion-vector writer half of this issue, building on #3232 (
DeleteVector::serialize). I'll leave the commit half to #2203 (RowDelta). @vikas-magar, you offered to work on this in July. If you have something in progress, please say so and I'll review it or help with it instead.Plan: a
DeletionVectorWriterinwriter/base_writer, next toPositionDeleteFileWriter, with the same(file_path, pos)input, so callers can choose by format version.DeleteVector. Onclose, it writes one Puffin file with onedeletion-vector-v1blob per data file. The blobs are uncompressed, use field2147483645(_pos) and snapshot id and sequence number-1, and carry thereferenced-data-fileandcardinalityproperties, as Iceberg Java does.DataFileper blob. Each has contentPositionDeletes, formatPuffin,referenced_data_file,content_offset,content_size_in_bytes,record_countset to the cardinality, and the partition.Tests:
PuffinReaderandDeleteVector::deserialize, and scan a data file throughArrowReaderwith the written DV.The work is on https://github.com/ostrium-labs/iceberg-rust/tree/loams/dv-writer, in small PRs. I'll open upstream PRs against this issue once each piece is green, after #3232 merges, and keep each one under the size guideline. Feedback on the shape of the API is welcome before then.