Skip to content

fix(deletion_vector): read Java bitmap64 deletion vectors - #785

Open
jackylee-ch wants to merge 1 commit into
apache:mainfrom
jackylee-ch:fix/deletion-vector-bitmap64
Open

fix(deletion_vector): read Java bitmap64 deletion vectors#785
jackylee-ch wants to merge 1 commit into
apache:mainfrom
jackylee-ch:fix/deletion-vector-bitmap64

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

A deletion vector written by Java in the 64-bit format (Bitmap64DeletionVector)
cannot be read: read_from_bytes accepts only the v1 magic, so every scan touching
such a file fails with Invalid magic number, naming the v1 magic alone.
Java's DeletionVector.read dispatches on both.

Two things differ beyond the value. The v2 magic is written little-endian, and
DeletionFile.length() counts the length prefix and the CRC for v2 but neither for
v1. The second also breaks the read range: the factory asks for length + 8, past
the end of a v2 entry, so a vector ending its index file is rejected by the storage
layer before parsing.

Fix: detect either magic, apply each format's length convention, and decode
the payload as portable 64-bit roaring. Java run-length encodes before writing and
emits every bucket key densely, so tests cover both shapes.

Read side only; the writer still emits v1. Positions above u32::MAX are rejected
rather than truncated — every API here is already roaring32-bound, and reaching
one needs a data file with over 4.29e9 rows. A negative bitmapLength read from
the file now errors instead of wrapping the size guard into passing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant