You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[C++] "replace_with_mask" kernel produces invalid result (wrong validity bitmap) when array is a boolean ChunkedArray #34634
Describe the bug, including details regarding any error messages, version, and platform.
Passing a boolean ChunkedArray to pc.replace_with_mask seems to produce an invalid result. Replacing the ChunkedArray with a regular Array seems to work fine. Other types seem to work fine with ChunkedArray. The invalid result appears that it may be specific to the boolean type.
import pyarrow as pa
import pyarrow.compute as pc
arr = pa.chunked_array([[True, True]])
mask = pa.array([False, True])
replacements = pa.array([False, False])
pc.replace_with_mask(arr, mask, replacements)
output:
<pyarrow.lib.ChunkedArray object at 0x7fb57a7a6900>
[
<Invalid array: Buffer #1 too small in array of type bool and length 2: expected at least 1 byte(s), got 0>
]
changed the title [-]pc.replace_with_mask produces invalid return when array is boolean ChunkedArray[/-][+]pc.replace_with_mask produces invalid result when array is a boolean ChunkedArray[/+]on Mar 19, 2023
changed the title [-]pc.replace_with_mask produces invalid result when array is a boolean ChunkedArray[/-][+][Python] pc.replace_with_mask produces invalid result when array is a boolean ChunkedArray[/+]on Mar 19, 2023
According to documentation the following should be fulfilled: len(replacements) == sum(mask == true).
In this case, len(replacements) == 2 and sum(mask==True) == 1.
However, error talks about something yet different.
According to documentation the following should be fulfilled: len(replacements) == sum(mask == true).
In this case, len(replacements) == 2 and sum(mask==True) == 1.
Indeed, the usage in the top-post reproducer is actually wrong, but generally we currently ignore if the replacements array is too long (see also #32436). And indeed, also if you provide the correct length, the same issue occurs:
In [12]: import pyarrow as pa
...: import pyarrow.compute as pc
...:
...: arr = pa.chunked_array([[True, True]])
...: mask = pa.array([False, True])
...: replacements = pa.array([False])
In [13]: pc.replace_with_mask(arr, mask, replacements)
Out[13]:
<pyarrow.lib.ChunkedArray object at 0x7f027c727650>
[
<Invalid array: Buffer #1 too small in array of type bool and length 2: expected at least 1 byte(s), got 0
]
The support for chunked arrays in this kernel is generally limited (see #31665), but providing a chunked array for the input array should now work. The above example also works if I use a different type (eg int64) instead of boolean. So indeed this seems to be an issue specifically with chunked boolean input array.
changed the title [-][Python] pc.replace_with_mask produces invalid result when array is a boolean ChunkedArray[/-][+][C++] "replace_with_mask" kernel produces invalid result (wrong validity bitmap) when array is a boolean ChunkedArray[/+]on Mar 21, 2023
The same issue happens for the fill_forward/backward kernels:
In [37]: pc.fill_null_forward(pa.array([True, False, None]))
Out[37]:
<pyarrow.lib.BooleanArray object at 0x7f0276f33820>
[
true,
false,
false
]
In [38]: pc.fill_null_forward(pa.chunked_array([[True, False, None]]))
Out[38]:
<pyarrow.lib.ChunkedArray object at 0x7f0276ff56c0>
[
<Invalid array: Buffer #1 too small in array of type bool and length 3: expected at least 1 byte(s), got 0
/home/joris/scipy/repos/arrow/cpp/src/arrow/array/validate.cc:116 ValidateLayout(*data.type)>
]
Looking at the implementation, I think the issue is with pre-allocating the result arrays for fixed width types. This uses ->byte_width(), but that only works for fixed width types with at least 1 byte per element, while boolean uses bits, and so this ->byte_width() returns 0 (basically that should never be used if the type can be boolean):
@lukemanley or @pstorozenko if someone of you would like to try fixing this, I would be happy to provide some guidance
Thanks for the offer. I'm not sure I will get around to this as I'm not all that familiar with cpp. Based on a quick glance, I assume something like the following (from vector_selection.cpp) is needed:
Activity
[-]pc.replace_with_mask produces invalid return when array is boolean ChunkedArray[/-][+]pc.replace_with_mask produces invalid result when array is a boolean ChunkedArray[/+][-]pc.replace_with_mask produces invalid result when array is a boolean ChunkedArray[/-][+][Python] pc.replace_with_mask produces invalid result when array is a boolean ChunkedArray[/+]pstorozenko commentedon Mar 20, 2023
According to documentation the following should be fulfilled:
len(replacements) == sum(mask == true).In this case,
len(replacements) == 2andsum(mask==True) == 1.However, error talks about something yet different.
pstorozenko commentedon Mar 20, 2023
Anyway, the following code in R yields the same error.
jorisvandenbossche commentedon Mar 21, 2023
Indeed, the usage in the top-post reproducer is actually wrong, but generally we currently ignore if the
replacementsarray is too long (see also #32436). And indeed, also if you provide the correct length, the same issue occurs:The support for chunked arrays in this kernel is generally limited (see #31665), but providing a chunked array for the input array should now work. The above example also works if I use a different type (eg int64) instead of boolean. So indeed this seems to be an issue specifically with chunked boolean input array.
[-][Python] pc.replace_with_mask produces invalid result when array is a boolean ChunkedArray[/-][+][C++] "replace_with_mask" kernel produces invalid result (wrong validity bitmap) when array is a boolean ChunkedArray[/+]jorisvandenbossche commentedon Mar 21, 2023
The same issue happens for the fill_forward/backward kernels:
Looking at the implementation, I think the issue is with pre-allocating the result arrays for fixed width types. This uses
->byte_width(), but that only works for fixed width types with at least 1 byte per element, while boolean uses bits, and so this->byte_width()returns 0 (basically that should never be used if the type can be boolean):arrow/cpp/src/arrow/compute/kernels/vector_replace.cc
Lines 424 to 426 in 69118b2
So in case of boolean arrays, we are allocating here a data buffer of length 0.
@lukemanley or @pstorozenko if someone of you would like to try fixing this, I would be happy to provide some guidance
lukemanley commentedon Mar 28, 2023
Thanks for the offer. I'm not sure I will get around to this as I'm not all that familiar with cpp. Based on a quick glance, I assume something like the following (from vector_selection.cpp) is needed:
arrow/cpp/src/arrow/compute/kernels/vector_selection.cc
Lines 252 to 256 in 69118b2
thisisnic commentedon Nov 18, 2026
fixed by #50843