Repository navigation
feat(iceberg): serialize DV-s to the deletion-vector-v1 blob layout - #3232
laskoviymishka wants to merge 2 commits into
Conversation
…lob layout Add DeleteVector::serialize, which produces a deletion-vector-v1 blob that DeleteVector::deserialize reads back (apache#2580). It frames the roaring treemap as [length][magic][vector][crc], matching what deserialize reads: the roaring portable 64-bit directory (a little-endian bitmap count followed by ascending u32-keyed bitmaps), a big-endian length prefix covering the magic and vector, and a big-endian CRC-32 over the same bytes. The vector is written as-is with sparse-key encoding rather than run-length-encoded, so for the same positions the bytes need not match Iceberg-Java's BitmapPositionDeleteIndex, which run-optimizes and writes dense keys; both encodings are spec-conformant and mutually decodable. This emits the raw blob only; wrapping it in a Puffin Blob with the snapshot and sequence-number properties is left to a later change. Fold the framing into a single implementation shared by serialize and the tests, so the encode paths no longer duplicate the layout: the test-only frame_dv_blob and the encode_dv_blob helpers now delegate to serialize, and the length prefix uses a checked u32 conversion instead of a truncating cast. Round-trip tests cover an empty vector, a single position, a dense single-container run, positions spanning multiple 2^32 containers, and a run-optimized dense range, plus golden assertions on the magic offset, the length prefix, and CRC validation, and a byte-stability test that re-serializes a decoded blob to guard against container-type normalization on the round trip.
…ad of panicking Anoop's review noted that the u32 overflow guard in the deletion-vector-v1 framing should not panic. Following the project's preference for avoiding panics in library code, DeleteVector::serialize and the internal frame_dv_blob helper now return crate::Result<Vec<u8>>: the u32 length-prefix conversion yields an ErrorKind::DataInvalid when the body would exceed u32::MAX, and the roaring treemap write maps its (in practice infallible) error to ErrorKind::Unexpected instead of expect(). The doc comment's Panics section is replaced with an Errors section describing these cases. Test-only callers and the cfg(test) helper are updated for the new Result.
| // Nothing in non-test crate code calls this yet (the delete-vector write path wires it in | ||
| // later), and `delete_vector` is a private module, so it otherwise reads as dead code. | ||
| #[allow(unused)] | ||
| pub fn serialize(&self) -> Result<Vec<u8>> { |
There was a problem hiding this comment.
The spec says the vector supports positive 64-bit positions ("the most significant bit must be 0"), but insert accepts any u64 and serialize doesn't check, so a position >= 2^63 would be written as a key >= 2^31. Iceberg Java rejects that on read (RoaringPositionBitmap.readKey requires key <= Integer.MAX_VALUE - 1), so the "mutually decodable" note wouldn't hold for that range.
Real row positions won't get there, so this is mostly defensive, but would it make sense for serialize to return an error when the highest key has its top bit set, like the length check below? That would keep us from writing a blob other implementations can't read.
|
Thanks for working on this. We need deletion-vector writes in iceberg-rust, and we'd like to build a Some things that might help here:
Our own builds carry this PR's two commits unchanged, so nothing here needs to change on our account. |
feat(iceberg): carry apache#3232, serialize deletion vectors
|
@laskoviymishka, would you mind if we finished this PR? It hasn't changed since mid-September, and the deletion-vector writer in #2580 builds on it. If you're fine with that, I'd open a PR that keeps your two commits as they are, with you as their author. On top of them it would add:
I'd also handle the open review point about positions of 2^63 and above. If you'd rather keep going yourself, that's great too. Just say so and I'll help with review and tests instead. |
|
@dina-kar - i'm fine with that |
|
Thanks @laskoviymishka |
Upstream reads v3 deletion vectors but cannot write them. Add the write side. - `DeletionVectorWriter` collects deleted positions per data file, from `PositionDeleteInput`s or whole `DeleteVector`s (`write_bitmap`), and on close writes each data file's vector as a `deletion-vector-v1` blob in its own Puffin file. The returned position-delete `DataFile` carries the referenced data file, the blob's content offset and size, and the cardinality as record count. - `DeleteVector::serialize` produces the `deletion-vector-v1` framing that `deserialize` reads (the signature of apache#3232, closed unmerged), and `to_puffin_blob` wraps it in a Puffin blob with the `cardinality` and `referenced-data-file` properties and snapshot id and sequence number -1, as the spec requires. As in Java's `BaseDVFileWriter`, the blob is computed for the row position metadata column and the Puffin file sets `created-by`. - `PuffinWriter::blobs_metadata` exposes the offsets and lengths of the blobs written so far. Writers picked from risingwavelabs/iceberg-rust. A test reads a data file through `ArrowReader` with a written deletion vector applied.
Upstream reads v3 deletion vectors and has no writer for them. This commit adds one. - `DeletionVectorWriter` collects deleted positions per data file, from `PositionDeleteInput`s or from whole `DeleteVector`s through `write_bitmap`. On close it writes each data file's vector as a `deletion-vector-v1` blob in its own Puffin file. The returned position delete `DataFile` carries the referenced data file, the blob's content offset and size, and the cardinality as its record count. - `DeleteVector::serialize` produces the `deletion-vector-v1` framing that `deserialize` reads, with the signature apache#3232 proposed before it closed unmerged. `to_puffin_blob` wraps it in a Puffin blob with the `cardinality` and `referenced-data-file` properties, and with snapshot id and sequence number set to -1 as the spec requires. The blob covers the row position metadata column and the Puffin file sets `created-by`, as Java's `BaseDVFileWriter` does. - `PuffinWriter::blobs_metadata` exposes the offsets and lengths of the blobs written so far. The writers come from risingwavelabs/iceberg-rust. A test reads a data file through `ArrowReader` with a written deletion vector applied.
Which issue does this PR close?
Add DeleteVector::serialize, which produces a deletion-vector-v1 blob that DeleteVector::deserialize reads back (#2580). It frames the roaring treemap as [length][magic][vector][crc], matching what deserialize reads: the roaring portable 64-bit directory (a little-endian bitmap count followed by ascending u32-keyed bitmaps), a big-endian length prefix covering the magic and vector, and a big-endian CRC-32 over the same bytes. The vector is written as-is with sparse-key encoding rather than run-length-encoded, so for the same positions the bytes need not match Iceberg-Java's BitmapPositionDeleteIndex, which run-optimizes and writes dense keys; both encodings are spec-conformant and mutually decodable. This emits the raw blob only; wrapping it in a Puffin Blob with the snapshot and sequence-number properties is left to a later change.
Fold the framing into a single implementation shared by serialize and the tests, so the encode paths no longer duplicate the layout: the test-only frame_dv_blob and the encode_dv_blob helpers now delegate to serialize, and the length prefix uses a checked u32 conversion instead of a truncating cast. Round-trip tests cover an empty vector, a single position, a dense single-container run, positions spanning multiple 2^32 containers, and a run-optimized dense range, plus golden assertions on the magic offset, the length prefix, and CRC validation, and a byte-stability test that re-serializes a decoded blob to guard against container-type normalization on the round trip.