Skip to content

feat(iceberg): serialize DV-s to the deletion-vector-v1 blob layout - #3232

Closed
laskoviymishka wants to merge 2 commits into
apache:mainfrom
laskoviymishka:feat/dv-v1-serializer
Closed

laskoviymishka wants to merge 2 commits into
apache:mainfrom
laskoviymishka:feat/dv-v1-serializer

Conversation

@laskoviymishka

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Add DeleteVector::serialize, which produces a deletion-vector-v1 blob that DeleteVector::deserialize reads back (#2580). It frames the roaring treemap as [length][magic][vector][crc], matching what deserialize reads: the roaring portable 64-bit directory (a little-endian bitmap count followed by ascending u32-keyed bitmaps), a big-endian length prefix covering the magic and vector, and a big-endian CRC-32 over the same bytes. The vector is written as-is with sparse-key encoding rather than run-length-encoded, so for the same positions the bytes need not match Iceberg-Java's BitmapPositionDeleteIndex, which run-optimizes and writes dense keys; both encodings are spec-conformant and mutually decodable. This emits the raw blob only; wrapping it in a Puffin Blob with the snapshot and sequence-number properties is left to a later change.

Fold the framing into a single implementation shared by serialize and the tests, so the encode paths no longer duplicate the layout: the test-only frame_dv_blob and the encode_dv_blob helpers now delegate to serialize, and the length prefix uses a checked u32 conversion instead of a truncating cast. Round-trip tests cover an empty vector, a single position, a dense single-container run, positions spanning multiple 2^32 containers, and a run-optimized dense range, plus golden assertions on the magic offset, the length prefix, and CRC validation, and a byte-stability test that re-serializes a decoded blob to guard against container-type normalization on the round trip.

…lob layout

Add DeleteVector::serialize, which produces a deletion-vector-v1 blob that
DeleteVector::deserialize reads back (apache#2580). It frames the roaring treemap as
[length][magic][vector][crc], matching what deserialize reads: the roaring
portable 64-bit directory (a little-endian bitmap count followed by ascending
u32-keyed bitmaps), a big-endian length prefix covering the magic and vector,
and a big-endian CRC-32 over the same bytes. The vector is written as-is with
sparse-key encoding rather than run-length-encoded, so for the same positions
the bytes need not match Iceberg-Java's BitmapPositionDeleteIndex, which
run-optimizes and writes dense keys; both encodings are spec-conformant and
mutually decodable. This emits the raw blob only; wrapping it in a Puffin Blob
with the snapshot and sequence-number properties is left to a later change.

Fold the framing into a single implementation shared by serialize and the
tests, so the encode paths no longer duplicate the layout: the test-only
frame_dv_blob and the encode_dv_blob helpers now delegate to serialize, and the
length prefix uses a checked u32 conversion instead of a truncating cast.
Round-trip tests cover an empty vector, a single position, a dense
single-container run, positions spanning multiple 2^32 containers, and a
run-optimized dense range, plus golden assertions on the magic offset, the
length prefix, and CRC validation, and a byte-stability test that re-serializes
a decoded blob to guard against container-type normalization on the round trip.
@laskoviymishka
laskoviymishka marked this pull request as ready for review September 15, 2026 20:03
Comment thread crates/iceberg/src/delete_vector.rs Resolved
Comment thread crates/iceberg/src/delete_vector.rs Resolved
…ad of panicking

Anoop's review noted that the u32 overflow guard in the deletion-vector-v1
framing should not panic. Following the project's preference for avoiding
panics in library code, DeleteVector::serialize and the internal frame_dv_blob
helper now return crate::Result<Vec<u8>>: the u32 length-prefix conversion
yields an ErrorKind::DataInvalid when the body would exceed u32::MAX, and the
roaring treemap write maps its (in practice infallible) error to
ErrorKind::Unexpected instead of expect(). The doc comment's Panics section is
replaced with an Errors section describing these cases. Test-only callers and
the cfg(test) helper are updated for the new Result.
// Nothing in non-test crate code calls this yet (the delete-vector write path wires it in
// later), and `delete_vector` is a private module, so it otherwise reads as dead code.
#[allow(unused)]
pub fn serialize(&self) -> Result<Vec<u8>> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spec says the vector supports positive 64-bit positions ("the most significant bit must be 0"), but insert accepts any u64 and serialize doesn't check, so a position >= 2^63 would be written as a key >= 2^31. Iceberg Java rejects that on read (RoaringPositionBitmap.readKey requires key <= Integer.MAX_VALUE - 1), so the "mutually decodable" note wouldn't hold for that range.

Real row positions won't get there, so this is mostly defensive, but would it make sense for serialize to return an error when the highest key has its top bit set, like the length check below? That would keep us from writing a blob other implementations can't read.

@dina-kar

dina-kar commented Oct 21, 2026 •

Copy link
Copy Markdown

Thanks for working on this. We need deletion-vector writes in iceberg-rust, and we'd like to build a DeletionVectorWriter on top of this PR rather than duplicate it. The writer would emit the Puffin file and its DataFile entries (referenced_data_file, content_offset, content_size_in_bytes). I've described the plan on #2580.

Some things that might help here:

  • We have DV tables written by Spark 4.0.4 with Iceberg 1.12.0: unpartitioned and partitioned tables, DELETE, merge-on-read UPDATE and MERGE, and a v2 table with position deletes upgraded to v3. For every Spark-written blob in them, we can check that deserialize → serialize → deserialize returns the same positions, and that Spark and Trino read the blobs this PR writes once they're committed. If useful, I can contribute those checks as tests, either here or in a follow-up PR, whichever you prefer.
  • The branch currently conflicts with main: crates/iceberg/src/test_utils.rs was moved in refactor(scan) move test utils to test_utils.rs #3107.
  • Since feat(delete-vector): make the delete_vector module public #3461 made delete_vector public, cargo doc with -D warnings rejects the intra-doc link from serialize's docs to the private decode_roaring_directory. Plain backticks fix it. public-api.txt also needs the new serialize line.
  • Your suggestion to optimize() before serializing matches what we saw: Spark's blobs are run-optimized.

Our own builds carry this PR's two commits unchanged, so nothing here needs to change on our account.

dina-kar added a commit to ostrium-labs/iceberg-rust that referenced this pull request Oct 22, 2026
feat(iceberg): carry apache#3232, serialize deletion vectors
@dina-kar

Copy link
Copy Markdown

@laskoviymishka, would you mind if we finished this PR? It hasn't changed since mid-September, and the deletion-vector writer in #2580 builds on it.

If you're fine with that, I'd open a PR that keeps your two commits as they are, with you as their author. On top of them it would add:

I'd also handle the open review point about positions of 2^63 and above. If you'd rather keep going yourself, that's great too. Just say so and I'll help with review and tests instead.

@laskoviymishka

Copy link
Copy Markdown
Contributor Author

@dina-kar - i'm fine with that

@dina-kar

Copy link
Copy Markdown

Thanks @laskoviymishka

wasifaleem added a commit to Supermetal-Inc/iceberg-rust that referenced this pull request Nov 2, 2026
Upstream reads v3 deletion vectors but cannot write them. Add the write
side.

- `DeletionVectorWriter` collects deleted positions per data file, from
  `PositionDeleteInput`s or whole `DeleteVector`s (`write_bitmap`), and on
  close writes each data file's vector as a `deletion-vector-v1` blob in its
  own Puffin file. The returned position-delete `DataFile` carries the
  referenced data file, the blob's content offset and size, and the
  cardinality as record count.
- `DeleteVector::serialize` produces the `deletion-vector-v1` framing that
  `deserialize` reads (the signature of apache#3232, closed unmerged),
  and `to_puffin_blob` wraps it in a Puffin blob with the `cardinality` and
  `referenced-data-file` properties and snapshot id and sequence number -1,
  as the spec requires. As in Java's `BaseDVFileWriter`, the blob is computed
  for the row position metadata column and the Puffin file sets
  `created-by`.
- `PuffinWriter::blobs_metadata` exposes the offsets and lengths of the
  blobs written so far.

Writers picked from risingwavelabs/iceberg-rust. A test reads a data file
through `ArrowReader` with a written deletion vector applied.
wasifaleem added a commit to Supermetal-Inc/iceberg-rust that referenced this pull request Nov 3, 2026
Upstream reads v3 deletion vectors and has no writer for them. This commit adds one.

- `DeletionVectorWriter` collects deleted positions per data file, from `PositionDeleteInput`s or
  from whole `DeleteVector`s through `write_bitmap`. On close it writes each data file's vector
  as a `deletion-vector-v1` blob in its own Puffin file. The returned position delete `DataFile`
  carries the referenced data file, the blob's content offset and size, and the cardinality as
  its record count.
- `DeleteVector::serialize` produces the `deletion-vector-v1` framing that `deserialize` reads,
  with the signature apache#3232 proposed before it closed unmerged.
  `to_puffin_blob` wraps it in a Puffin blob with the `cardinality` and `referenced-data-file`
  properties, and with snapshot id and sequence number set to -1 as the spec requires. The blob
  covers the row position metadata column and the Puffin file sets `created-by`, as Java's
  `BaseDVFileWriter` does.
- `PuffinWriter::blobs_metadata` exposes the offsets and lengths of the blobs written so far.

The writers come from risingwavelabs/iceberg-rust. A test reads a data file through
`ArrowReader` with a written deletion vector applied.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants