Repository navigation
Description
Describe the bug, including details regarding any error messages, version, and platform.
When a nested builder contains a DictionaryBuilder whose index type was promoted while building (e.g. int8 -> int16 after more than 128 distinct values), the finished parent array can declare the original int8 index type while its child data holds int16 indices.
Minimal reproducer (pyarrow 26.0.1 from PyPI, Linux x86_64; also reproduced with C++ built from main at 55803660f015f9945fe1b9bc84f0c138ae70ce98):
import pyarrow as pa
vals = [str(i).encode() for i in range(200)]
ty = pa.map_(pa.int32(), pa.dictionary(pa.int8(), pa.binary()))
arr = pa.array([list(zip(range(200), vals))], type=ty)
print(arr.type.item_type) # dictionary<values=binary, indices=int8, ordered=0>
print(arr.items.type) # dictionary<values=binary, indices=int16, ordered=0>
arr.validate(full=True) # passes
sink = pa.BufferOutputStream()
table = pa.table({"m": arr})
with pa.ipc.new_stream(sink, table.schema) as w:
w.write_table(table)
back = pa.ipc.open_stream(sink.getvalue()).read_all()
print(back["m"].to_pylist() == arr.to_pylist()) # FalseExpected: arr.type.item_type == arr.items.type (both int16 indices) and the IPC round trip returns equal values.
Actual: the declared type says int8 while the child data is int16. validate(full=True) does not flag it for map/list containers. After an IPC round trip the values differ (the int16 index buffer is read as int8). With 300 distinct values the same script aborts in our runs (std::length_error, SIGABRT) during the round trip.
The same mismatch occurs for list, large_list, list_view, large_list_view, fixed_size_list and struct when the dictionary sits one level deeper (e.g. list<list<dictionary>>), and for dense/sparse unions with a list<dictionary> child (C++ only). For struct, ValidateFull does report the mismatch.
Cause: in cpp/src/arrow/array/builder_nested.h / builder_nested.cc / builder_union.cc, the parent's FinishInternal computes its output type (via type(), or for MapBuilder by recomputing from the key/item builders) after finishing its children. Finishing a child list calls Reset() on its value builder, and a reset DictionaryBuilder reports int8 indices again. The parent therefore reads the reset type instead of the type of the finished child data:
BaseListBuilder<T>::FinishInternal,BaseListViewBuilder<T>::FinishInternal(builder_nested.h)FixedSizeListBuilder::FinishInternal,StructBuilder::FinishInternal,MapBuilder::FinishInternal(builder_nested.cc)BasicUnionBuilder::FinishInternal(builder_union.cc)
A proposed fix (build the output type from the finished child ArrayData types, preserving field names/nullability/metadata) with C++ and Python regression tests is on a fork branch: https://github.com/tianhaogutum/arrow/tree/nested-builder-finished-child-types
Related but different: #4930 (ARROW-5935, ArrayBuilder::type() accuracy), #25719 (dictionary keys in maps over IPC), #49797 (ordered flag in DictionaryBuilder).
AI disclosure: this issue was investigated and drafted with the help of an AI coding assistant; the reproducer above was run as written.
Component(s)
C++, Python
Activity