fix(deletion_vector): read Java bitmap64 deletion vectors - #785
Open
jackylee-ch wants to merge 1 commit into
Open
fix(deletion_vector): read Java bitmap64 deletion vectors#785jackylee-ch wants to merge 1 commit into
jackylee-ch wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A deletion vector written by Java in the 64-bit format (
Bitmap64DeletionVector)cannot be read:
read_from_bytesaccepts only the v1 magic, so every scan touchingsuch a file fails with
Invalid magic number, naming the v1 magic alone.Java's
DeletionVector.readdispatches on both.Two things differ beyond the value. The v2 magic is written little-endian, and
DeletionFile.length()counts the length prefix and the CRC for v2 but neither forv1. The second also breaks the read range: the factory asks for
length + 8, pastthe end of a v2 entry, so a vector ending its index file is rejected by the storage
layer before parsing.
Fix: detect either magic, apply each format's length convention, and decode
the payload as portable 64-bit roaring. Java run-length encodes before writing and
emits every bucket key densely, so tests cover both shapes.
Read side only; the writer still emits v1. Positions above
u32::MAXare rejectedrather than truncated — every API here is already roaring32-bound, and reaching
one needs a data file with over 4.29e9 rows. A negative
bitmapLengthread fromthe file now errors instead of wrapping the size guard into passing.