Repository navigation
fix: support all non-negative int64 deletion vector positions - #917
Draft
wirybeaver wants to merge 3 commits into
Draft
wirybeaver wants to merge 3 commits into
wirybeaver wants to merge 3 commits into
Conversation
4 tasks
wirybeaver
force-pushed
the
fix/dv-full-positive-int64
branch
2 times, most recently
from
September 5, 2026 23:16
0a6cc1a to
5c494b6
Compare
wirybeaver
force-pushed
the
fix/dv-full-positive-int64
branch
2 times, most recently
from
October 1, 2026 21:45
948bfeb to
002ccd9
Compare
wirybeaver
force-pushed
the
fix/dv-full-positive-int64
branch
from
November 20, 2026 07:32
659c200 to
b05e3e8
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
DV-PR-2 in #916 supports every non-negative
int64_trow position. Two distant deleted rows should require storage only for the bitmap buckets they occupy.Modification
Store populated Roaring buckets by their high 32 bits and serialize them in key order using the existing wire format. Handle runs ending at
INT64_MAXwithout overflowing the exclusive endpoint. Before opening the output file, check the complete blob size, including the length prefix and CRC, against the reader's limit.Test
The in-memory writer-to-reader test writes delete positions for two data files into one Puffin file. Reading the two blob ranges restores the expected positions for each file. A sparse index containing
INT64_MAXalso round-trips with its original cardinality.Earlier PR description and validation
Summary
Use sparse ordered high-32-bit buckets to represent every non-negative
int64_tdeletion-vector position without storage proportional to the largest key.preserve portable Roaring/Puffin encoding and serialize populated buckets in order
avoid overflow when a position-delete range ends at
INT64_MAX; document the half-open endpoint limitationretain fixed framing-size prechecks before allocation/output creation, without adding a configurable test-only writer factory
make emptiness checks constant-time under the nonempty-bucket invariant and reuse the computed framing length during serialization
DV-PR-2 in #916. Based directly on
mainat3c5715c; no fixture-only or footer-validation dependency. Existing independent raw Roaring fixtures remain, but the standalone fixtures from #913 are not included.Test plan
GCC 14 focused checks on the four-feature integration tree based on
3c5715c; the final changed branch sources were verified to match the tested tree.util_test --gtest_filter='RoaringPositionBitmap*:*PositionDeleteIndex*:*PositionDeleteRangeConsumer*': 64 passeddata_test --gtest_filter='DVWriterTest.*': 11 passedExisting sparse roundtrip coverage also checks restored cardinality; no new exhaustive boundary matrix was introduced.
git diff --check: passed