Skip to content

feat: add table-aware position delete updates - #919

Open
wirybeaver wants to merge 3 commits into
apache:mainfrom
wirybeaver:feature/v3-dv-write-pipeline
Open

wirybeaver wants to merge 3 commits into
apache:mainfrom
wirybeaver:feature/v3-dv-write-pipeline

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Sep 5, 2026 •

Copy link
Copy Markdown

Motivation

DV-PR-1 in #916 lets callers delete a row by data file and position. A second delete on the same v3 file must preserve positions deleted by the first commit.

Modification

Resolve each live file's partition and spec, and check the requested positions against its record count before creating output. Commit through RowDelta, which validates that the referenced data files still exist.

file + positions -> resolve live file -> v2: write sorted Parquet deletes
                                      -> v3: merge prior deletes into a Puffin DV
                 -> commit new entries and supersede rewritten deletes

A definite commit failure cleans up generated files. An unknown commit outcome preserves them because the catalog may already reference them.

Test

The in-library end-to-end test uses real Parquet and Puffin encoding with Arrow MockFileSystem and InMemoryCatalog.

  • It writes five rows, deletes positions 3 and 1, commits, and reloads the table. The scan returns rows 0, 2, and 4.
  • After partition evolution, a second delete at position 2 leaves exactly rows 0 and 4 on reload.
  • An upgrade scenario verifies that a v3 DV incorporates earlier v2 Parquet deletes.

@zhjwpku

zhjwpku commented Nov 20, 2026

Copy link
Copy Markdown
Member

@wirybeaver is this ready for review?

@wirybeaver
wirybeaver marked this pull request as ready for review November 20, 2026 07:25
@wirybeaver
wirybeaver force-pushed the feature/v3-dv-write-pipeline branch from f43d28c to 2bd9494 Compare November 20, 2026 07:32
@wirybeaver

Copy link
Copy Markdown
Author

Hi, @zhjwpku feel free to take a review. I rebase the most recent and updated the PR body

@zhjwpku

zhjwpku commented Nov 20, 2026

Copy link
Copy Markdown
Member

Hi, @zhjwpku feel free to take a review. I rebase the most recent and updated the PR body

CI failed, please take a look when available, thanks.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants