Skip to content

fix: support all non-negative int64 deletion vector positions - #917

Draft
wirybeaver wants to merge 3 commits into
apache:mainfrom
wirybeaver:fix/dv-full-positive-int64
Draft

wirybeaver wants to merge 3 commits into
apache:mainfrom
wirybeaver:fix/dv-full-positive-int64

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Sep 5, 2026 •

Copy link
Copy Markdown

Motivation

DV-PR-2 in #916 supports every non-negative int64_t row position. Two distant deleted rows should require storage only for the bitmap buckets they occupy.

Modification

Store populated Roaring buckets by their high 32 bits and serialize them in key order using the existing wire format. Handle runs ending at INT64_MAX without overflowing the exclusive endpoint. Before opening the output file, check the complete blob size, including the length prefix and CRC, against the reader's limit.

Test

The in-memory writer-to-reader test writes delete positions for two data files into one Puffin file. Reading the two blob ranges restores the expected positions for each file. A sparse index containing INT64_MAX also round-trips with its original cardinality.

Earlier PR description and validation

Summary

Use sparse ordered high-32-bit buckets to represent every non-negative int64_t deletion-vector position without storage proportional to the largest key.

  • preserve portable Roaring/Puffin encoding and serialize populated buckets in order

  • avoid overflow when a position-delete range ends at INT64_MAX; document the half-open endpoint limitation

  • retain fixed framing-size prechecks before allocation/output creation, without adding a configurable test-only writer factory

  • make emptiness checks constant-time under the nonempty-bucket invariant and reuse the computed framing length during serialization

DV-PR-2 in #916. Based directly on main at 3c5715c; no fixture-only or footer-validation dependency. Existing independent raw Roaring fixtures remain, but the standalone fixtures from #913 are not included.

Test plan

GCC 14 focused checks on the four-feature integration tree based on 3c5715c; the final changed branch sources were verified to match the tested tree.

  • util_test --gtest_filter='RoaringPositionBitmap*:*PositionDeleteIndex*:*PositionDeleteRangeConsumer*': 64 passed
  • data_test --gtest_filter='DVWriterTest.*': 11 passed

Existing sparse roundtrip coverage also checks restored cardinality; no new exhaustive boundary matrix was introduced.

  • Repository pre-commit checks and git diff --check: passed

@wirybeaver
wirybeaver force-pushed the fix/dv-full-positive-int64 branch 2 times, most recently from 0a6cc1a to 5c494b6 Compare September 5, 2026 23:16
@wirybeaver
wirybeaver force-pushed the fix/dv-full-positive-int64 branch 2 times, most recently from 948bfeb to 002ccd9 Compare October 1, 2026 21:45
@wirybeaver
wirybeaver force-pushed the fix/dv-full-positive-int64 branch from 659c200 to b05e3e8 Compare November 20, 2026 07:32

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant