Skip to content

[C++] "replace_with_mask" kernel produces invalid result (wrong validity bitmap) when array is a boolean ChunkedArray #34634

Closed

Description

@lukemanley

Describe the bug, including details regarding any error messages, version, and platform.

Passing a boolean ChunkedArray to pc.replace_with_mask seems to produce an invalid result. Replacing the ChunkedArray with a regular Array seems to work fine. Other types seem to work fine with ChunkedArray. The invalid result appears that it may be specific to the boolean type.

import pyarrow as pa
import pyarrow.compute as pc

arr = pa.chunked_array([[True, True]])
mask = pa.array([False, True])
replacements = pa.array([False, False])

pc.replace_with_mask(arr, mask, replacements)

output:

<pyarrow.lib.ChunkedArray object at 0x7fb57a7a6900>
[
<Invalid array: Buffer #1 too small in array of type bool and length 2: expected at least 1 byte(s), got 0>
]

Component(s)

Python

Activity

  1. changed the title [-]pc.replace_with_mask produces invalid return when array is boolean ChunkedArray[/-] [+]pc.replace_with_mask produces invalid result when array is a boolean ChunkedArray[/+] on Mar 19, 2023
  2. changed the title [-]pc.replace_with_mask produces invalid result when array is a boolean ChunkedArray[/-] [+][Python] pc.replace_with_mask produces invalid result when array is a boolean ChunkedArray[/+] on Mar 19, 2023
  3. pstorozenko commented on Mar 20, 2023

    @pstorozenko

    According to documentation the following should be fulfilled: len(replacements) == sum(mask == true).
    In this case, len(replacements) == 2 and sum(mask==True) == 1.
    However, error talks about something yet different.

  4. pstorozenko commented on Mar 20, 2023

    @pstorozenko

    Anyway, the following code in R yields the same error.

    library(arrow)
    arr <- chunked_array(c(TRUE, TRUE))
    mask <- Array$create(c(FALSE, TRUE))
    replacements <- Array$create(c(FALSE, FALSE))
    
    call_function("replace_with_mask", arr, mask, replacements)
    ChunkedArray
    <bool>
    [
    <Invalid array: Buffer #1 too small in array of type bool and length 2: expected at least 1 byte(s), got 0>
    ]
    
  5. jorisvandenbossche commented on Mar 21, 2023

    @jorisvandenbossche
    Member

    According to documentation the following should be fulfilled: len(replacements) == sum(mask == true).
    In this case, len(replacements) == 2 and sum(mask==True) == 1.

    Indeed, the usage in the top-post reproducer is actually wrong, but generally we currently ignore if the replacements array is too long (see also #32436). And indeed, also if you provide the correct length, the same issue occurs:

    In [12]: import pyarrow as pa
        ...: import pyarrow.compute as pc
        ...: 
        ...: arr = pa.chunked_array([[True, True]])
        ...: mask = pa.array([False, True])
        ...: replacements = pa.array([False])
    
    In [13]: pc.replace_with_mask(arr, mask, replacements)
    Out[13]: 
    <pyarrow.lib.ChunkedArray object at 0x7f027c727650>
    [
    <Invalid array: Buffer #1 too small in array of type bool and length 2: expected at least 1 byte(s), got 0
    ]
    

    The support for chunked arrays in this kernel is generally limited (see #31665), but providing a chunked array for the input array should now work. The above example also works if I use a different type (eg int64) instead of boolean. So indeed this seems to be an issue specifically with chunked boolean input array.

  6. changed the title [-][Python] pc.replace_with_mask produces invalid result when array is a boolean ChunkedArray[/-] [+][C++] "replace_with_mask" kernel produces invalid result (wrong validity bitmap) when array is a boolean ChunkedArray[/+] on Mar 21, 2023
  7. jorisvandenbossche commented on Mar 21, 2023

    @jorisvandenbossche
    Member

    The same issue happens for the fill_forward/backward kernels:

    In [37]: pc.fill_null_forward(pa.array([True, False, None]))
    Out[37]: 
    <pyarrow.lib.BooleanArray object at 0x7f0276f33820>
    [
      true,
      false,
      false
    ]
    
    In [38]: pc.fill_null_forward(pa.chunked_array([[True, False, None]]))
    Out[38]: 
    <pyarrow.lib.ChunkedArray object at 0x7f0276ff56c0>
    [
    <Invalid array: Buffer #1 too small in array of type bool and length 3: expected at least 1 byte(s), got 0
    /home/joris/scipy/repos/arrow/cpp/src/arrow/array/validate.cc:116  ValidateLayout(*data.type)>
    ]
    

    Looking at the implementation, I think the issue is with pre-allocating the result arrays for fixed width types. This uses ->byte_width(), but that only works for fixed width types with at least 1 byte per element, while boolean uses bits, and so this ->byte_width() returns 0 (basically that should never be used if the type can be boolean):

    const int64_t slot_width = out->type()->byte_width();
    ARROW_ASSIGN_OR_RAISE(chunk_out->buffers[1],
    ctx->Allocate(slot_width * chunk->length()));

    So in case of boolean arrays, we are allocating here a data buffer of length 0.


    @lukemanley or @pstorozenko if someone of you would like to try fixing this, I would be happy to provide some guidance

  8. lukemanley commented on Mar 28, 2023

    @lukemanley
    Author

    @lukemanley or @pstorozenko if someone of you would like to try fixing this, I would be happy to provide some guidance

    Thanks for the offer. I'm not sure I will get around to this as I'm not all that familiar with cpp. Based on a quick glance, I assume something like the following (from vector_selection.cpp) is needed:

    if (bit_width == 1) {
    ARROW_ASSIGN_OR_RAISE(out->buffers[1], ctx->AllocateBitmap(length));
    } else {
    ARROW_ASSIGN_OR_RAISE(out->buffers[1], ctx->Allocate(length * bit_width / 8));
    }

  9. thisisnic commented on Nov 18, 2026

    @thisisnic
    Member

    fixed by #50843

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions