Skip to content

feat: add deletion vector compaction planning and execution - #921

Draft
wirybeaver wants to merge 6 commits into
apache:mainfrom
wirybeaver:feature/dv-compaction-executor
Draft

wirybeaver wants to merge 6 commits into
apache:mainfrom
wirybeaver:feature/dv-compaction-executor

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Sep 5, 2026 •

Copy link
Copy Markdown

Note

This PR depends on #919.

Motivation

Compaction should combine small files and reclaim deleted rows while preserving the identity of surviving rows and any delete entries still needed by other files.

Modification

snapshot scan -> group candidates within partition and spec
              -> read surviving rows -> write replacement files
              -> atomically replace inputs and obsolete delete entries

The planner uses file size and file-scoped delete counts to select candidates and packs them deterministically. The executor validates the source snapshot and preserves row lineage. On hierarchical filesystems, a missing output directory is created when the replacement file is opened.

Test

The in-library compaction test writes partitioned Parquet files, commits deletes, and compares surviving rows and lineage before and after the rewrite. Compaction succeeds with initially absent output partition directories, emits no replacement for a fully deleted group, and retains the shared Puffin file when an uncompacted input still needs it.

@wirybeaver
wirybeaver force-pushed the feature/dv-compaction-executor branch 5 times, most recently from 55c6b56 to b3834d9 Compare September 6, 2026 00:19
@wirybeaver
wirybeaver force-pushed the feature/dv-compaction-executor branch from b3834d9 to a7d3361 Compare October 1, 2026 19:57
@wirybeaver wirybeaver changed the title feat: add deletion vector compaction execution feat: add deletion vector compaction planning and execution Oct 1, 2026
@wirybeaver
wirybeaver force-pushed the feature/dv-compaction-executor branch from a7d3361 to 877ff3e Compare October 1, 2026 21:45
@wirybeaver
wirybeaver force-pushed the feature/dv-compaction-executor branch from 5ef2c7a to 7bb76b3 Compare November 20, 2026 07:32

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant