Repository navigation
Read deletion vectors whose first bitmaps are empty - #4425
rahulsmahadev wants to merge 1 commit into
Conversation
DeletionVector.to_vector() built its chunked array without a type, so an empty first bitmap (positions only at or above 2**32) or a vector with no bitmaps raised ArrowInvalid, and a vector of only empty bitmaps came back null-typed. Build it as int64. Co-authored-by: Isaac <no-reply@databricks.com>
|
@kevinjqliu, this fixes |
Fokko
left a comment
There was a problem hiding this comment.
Thanks @rahulsmahadev! Nice find, and the fix is as small as it gets.
I checked out the branch and ran tests/table/test_deletion_vector.py locally: all 14 pass, and the empty and all-above-2**32 cases now come back as a proper empty int64 chunked array instead of an ArrowInvalid.
LGTM!
| ([BitMap(), BitMap([5])], [2**32 + 5]), | ||
| ([BitMap(), BitMap(), BitMap([0, 7])], [2 * 2**32, 2 * 2**32 + 7]), | ||
| ([BitMap([2]), BitMap(), BitMap([1])], [2, 2 * 2**32 + 1]), | ||
| ([BitMap()], []), | ||
| ([BitMap(), BitMap()], []), |
There was a problem hiding this comment.
If one case fails, the output shows only the assert on raw BitMaps and large integers, so it's hard to tell which scenario broke. Not blocking, since the project prefers a for loop over parametrize for many simple cases.
Rationale for this change
Bug:
DeletionVector.to_vector()builds the PyArrow array without a type, so PyArrow infers it from the first bitmap. A deletion vector has one bitmap per high 32 bits of the position. When all deleted positions are at or above 232 (a data file with more than 232 rows), the first bitmap is empty, PyArrow types it as null and the read fails:A vector with no bitmaps, like
64mapempty.binintests/table/bitmaps, fails withArrowInvalid: cannot construct ChunkedArray from empty vector and omitted type. Java writes an empty bitmap for key 0 in the first case (RoaringPositionBitmap.serialize) and reads both fine.Fix: pass
type=pa.int64(). This also coversfrom_positions()in #3474, which pads missing keys with empty bitmaps the same way.Are these changes tested?
Yes, added
test_to_vector_with_empty_bitmaps_for_low_keysand extendedtest_map_emptyintests/table/test_deletion_vector.py. Both fail on main with the errors above and pass with this change.Are there any user-facing changes?
Yes, these deletion vectors can be read now. A vector with only empty bitmaps now converts to an empty int64 array instead of a null-typed one.
Please add the
changeloglabel.This pull request and its description were written by Isaac.