Skip to content

Deletion-vector writer + RowDelta action #2580

Open

Description

@Shekharrajak

Is your feature request related to a problem or challenge?

iceberg-rust cannot produce v3 deletes today; only FastAppend exists.

Describe the solution you'd like

No response

Willingness to contribute

Yes

Activity

  1. vikas-magar commented on Jul 15, 2026

    @vikas-magar

    Hi, I can work on this issue.

  2. dina-kar commented on Oct 21, 2026

    @dina-kar

    I'd like to work on the deletion-vector writer half of this issue, building on #3232 (DeleteVector::serialize). I'll leave the commit half to #2203 (RowDelta). @vikas-magar, you offered to work on this in July. If you have something in progress, please say so and I'll review it or help with it instead.

    Plan: a DeletionVectorWriter in writer/base_writer, next to PositionDeleteFileWriter, with the same (file_path, pos) input, so callers can choose by format version.

    • It collects the positions per referenced data file into a DeleteVector. On close, it writes one Puffin file with one deletion-vector-v1 blob per data file. The blobs are uncompressed, use field 2147483645 (_pos) and snapshot id and sequence number -1, and carry the referenced-data-file and cardinality properties, as Iceberg Java does.
    • It returns one DataFile per blob. Each has content PositionDeletes, format Puffin, referenced_data_file, content_offset, content_size_in_bytes, record_count set to the cardinality, and the partition.
    • The spec allows at most one DV per data file per snapshot, so the writer also takes a data file's existing DV and merges it into the new one. The caller then removes the old DV in the same commit (via RowDelta).

    Tests:

    • Unit tests read the Puffin file back with PuffinReader and DeleteVector::deserialize, and scan a data file through ArrowReader with the written DV.
    • An integration test uses a Spark-provisioned v3 table with DVs. It reads Spark's blobs, merges new positions into them, and checks the result.
    • Outside this repository's CI, we check that Spark 4.0.4 (Iceberg 1.12.0) and Trino 483 read tables whose DVs were written this way, committed through Iceberg Java's RowDelta.

    The work is on https://github.com/ostrium-labs/iceberg-rust/tree/loams/dv-writer, in small PRs. I'll open upstream PRs against this issue once each piece is green, after #3232 merges, and keep each one under the size guideline. Feedback on the shape of the API is welcome before then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions