Repository navigation
Description
Describe the bug, including details regarding any error messages, version, and platform.
When a struct column has struct-level nulls and a non-zero array offset, the ORC writer writes the struct's validity and child values starting at physical row 0 instead of at the offset. No error is raised. The struct column ends up holding values from the wrong rows, and is misaligned with the other columns in the same file.
A non-zero offset comes up in everyday use: Table.slice() / table[a:b], or any chunk produced by to_batches(max_chunksize=...) or by slicing a larger array.
import pyarrow as pa
import pyarrow.orc as orc
t = pa.table({
"id": [0, 1, 2, 3, 4, 5],
"s": pa.array([{"a": 0}, None, {"a": 2}, {"a": 3}, None, {"a": 5}]),
})
# 1. Table.slice()
orc.write_table(t.slice(1, 4), "sliced.orc")
print(orc.read_table("sliced.orc").to_pylist())
# 2. Re-chunking a table (second chunk is a zero-copy slice with offset 4)
t2 = pa.Table.from_batches(t.to_batches(max_chunksize=4))
orc.write_table(t2, "rechunked.orc")
print(orc.read_table("rechunked.orc").column("s").to_pylist())Actual:
[{'id': 1, 's': {'a': 0}}, {'id': 2, 's': None}, {'id': 3, 's': {'a': 2}}, {'id': 4, 's': {'a': 3}}]
[{'a': 0}, None, {'a': 2}, {'a': 3}, {'a': 0}, None]
Expected:
[{'id': 1, 's': None}, {'id': 2, 's': {'a': 2}}, {'id': 3, 's': {'a': 3}}, {'id': 4, 's': None}]
[{'a': 0}, None, {'a': 2}, {'a': 3}, None, {'a': 5}]
id is written correctly but s is not, so the rows get mixed up. The file itself is wrong, not just the Arrow reader: the Apache ORC C++ reader (via pyorc) returns the same values for sliced.orc, [(1, (0,)), (2, None), (3, (2,)), (4, (3,))].
The same thing happens when the struct has offset 0 but a child has its own offset (e.g. pa.StructArray.from_arrays([pa.array([10, 11, 12, 13, 14, 15]).slice(3, 3)], names=["a"], mask=pa.array([False, True, False])) is written as [{'a': 10}, None, {'a': 12}] instead of [{'a': 13}, None, {'a': 15}]).
Not affected: sliced structs with no struct-level nulls, and sliced flat/list columns (the list case was fixed in GH-15212).
Root cause
NormalizeArray() in cpp/src/arrow/adapters/orc/util.cc, STRUCT branch (L395-L427 on main), which runs only when null_count() != 0. It rebuilds each child to AND in the struct's validity, and loses every offset along the way:
struct_array->field(i)returns the child already sliced to the struct's range (child offset = child offset + struct offset), butBitmapAnd(pool, bitmap->data(), 0, child_bitmap->data(), 0, length, 0)reads both bitmaps from bit 0,ArrayData::Make(child->type(), child->length(), child_buffers, ...)rebuilds the child without passingchild->offset(), so it defaults to 0 over the original, unsliced buffers, andstd::make_shared<StructArray>(struct_type, length, new_children, bitmap)dropsstruct_array->offset()while reusing the unsliced validity buffer.
ORC writer doesn't work on sliced list arrays #15212 (PR GH-15212: [C++] fix sliced list array writing in ORC #15213) fixed this same class of bug for the list and map branches of this function by passing the offset through when the arrays are rebuilt. The STRUCT branch was left as it was.
The existing regression test from that fix, TestORCWriterTrivialNoConversion.writeSlicedBatch, slices a table that includes a struct column, but generates it with probability=0. With no nulls, NormalizeArray takes the null_count() == 0 early return and never reaches this branch.
Suggested fix
StructArray::GetFlattenedField() already does exactly this correctly: it pushes the parent slice down to the child and ANDs the bitmaps at the correct offsets. The SPARSE_UNION branch of the same function already uses its union counterpart. Something like:
case Type::type::STRUCT: {
if (array->null_count() == 0) {
return array;
}
auto struct_array = checked_pointer_cast<StructArray>(array);
std::vector<std::shared_ptr<Array>> new_children(struct_array->num_fields());
for (int i = 0; i < struct_array->num_fields(); ++i) {
ARROW_ASSIGN_OR_RAISE(auto child, struct_array->GetFlattenedField(i));
ARROW_ASSIGN_OR_RAISE(new_children[i], NormalizeArray(child));
}
// Children are now relative to the struct's logical start, so rebase the
// struct's own validity bitmap to offset 0 as well.
ARROW_ASSIGN_OR_RAISE(
auto bitmap, ::arrow::internal::CopyBitmap(default_memory_pool(),
struct_array->null_bitmap_data(),
struct_array->offset(),
struct_array->length()));
return std::make_shared<StructArray>(struct_array->type(), struct_array->length(),
new_children, std::move(bitmap),
struct_array->null_count());
}The regression test could be writeSlicedBatch with a non-zero null probability, or a dedicated sliced struct column with nulls.
Related: GH-55618 is a separate struct-normalization bug in the same function. It is caused by the null_count() == 0 early return (nested struct nulls are never propagated), so it happens without any slicing. This issue is the null_count() > 0 branch and only needs a plain top-level struct plus an offset. A fix for one does not fix the other.
Workaround: copy affected columns to a zero-offset array before writing, e.g. pa.concat_arrays([arr, pa.array([], arr.type)]) or arr.take(...). Note that table.combine_chunks() does not help for a single sliced chunk, because it keeps the offset.
Environment: pyarrow 26.0.1 (also reproduced on 18.1.0), Python 3.13, Linux x86_64. The code is unchanged on main at a092214.
Component(s)
C++, Python
Activity
[-][C++][ORC] Writer ignores the array offset of struct columns with nulls and writes the wrong rows[/-][+][Python][C++][ORC] Writer ignores the array offset of struct columns with nulls and writes the wrong rows[/+]