Skip to content

[C++][Python] Nested builders declare stale dictionary index type after Finish (map, list, struct, union) #55640

Open

Description

@tianhaogutum

Describe the bug, including details regarding any error messages, version, and platform.

When a nested builder contains a DictionaryBuilder whose index type was promoted while building (e.g. int8 -> int16 after more than 128 distinct values), the finished parent array can declare the original int8 index type while its child data holds int16 indices.

Minimal reproducer (pyarrow 26.0.1 from PyPI, Linux x86_64; also reproduced with C++ built from main at 55803660f015f9945fe1b9bc84f0c138ae70ce98):

import pyarrow as pa

vals = [str(i).encode() for i in range(200)]
ty = pa.map_(pa.int32(), pa.dictionary(pa.int8(), pa.binary()))
arr = pa.array([list(zip(range(200), vals))], type=ty)

print(arr.type.item_type)  # dictionary<values=binary, indices=int8, ordered=0>
print(arr.items.type)      # dictionary<values=binary, indices=int16, ordered=0>
arr.validate(full=True)    # passes

sink = pa.BufferOutputStream()
table = pa.table({"m": arr})
with pa.ipc.new_stream(sink, table.schema) as w:
    w.write_table(table)
back = pa.ipc.open_stream(sink.getvalue()).read_all()
print(back["m"].to_pylist() == arr.to_pylist())  # False

Expected: arr.type.item_type == arr.items.type (both int16 indices) and the IPC round trip returns equal values.

Actual: the declared type says int8 while the child data is int16. validate(full=True) does not flag it for map/list containers. After an IPC round trip the values differ (the int16 index buffer is read as int8). With 300 distinct values the same script aborts in our runs (std::length_error, SIGABRT) during the round trip.

The same mismatch occurs for list, large_list, list_view, large_list_view, fixed_size_list and struct when the dictionary sits one level deeper (e.g. list<list<dictionary>>), and for dense/sparse unions with a list<dictionary> child (C++ only). For struct, ValidateFull does report the mismatch.

Cause: in cpp/src/arrow/array/builder_nested.h / builder_nested.cc / builder_union.cc, the parent's FinishInternal computes its output type (via type(), or for MapBuilder by recomputing from the key/item builders) after finishing its children. Finishing a child list calls Reset() on its value builder, and a reset DictionaryBuilder reports int8 indices again. The parent therefore reads the reset type instead of the type of the finished child data:

  • BaseListBuilder<T>::FinishInternal, BaseListViewBuilder<T>::FinishInternal (builder_nested.h)
  • FixedSizeListBuilder::FinishInternal, StructBuilder::FinishInternal, MapBuilder::FinishInternal (builder_nested.cc)
  • BasicUnionBuilder::FinishInternal (builder_union.cc)

A proposed fix (build the output type from the finished child ArrayData types, preserving field names/nullability/metadata) with C++ and Python regression tests is on a fork branch: https://github.com/tianhaogutum/arrow/tree/nested-builder-finished-child-types

Related but different: #4930 (ARROW-5935, ArrayBuilder::type() accuracy), #25719 (dictionary keys in maps over IPC), #49797 (ordered flag in DictionaryBuilder).

AI disclosure: this issue was investigated and drafted with the help of an AI coding assistant; the reproducer above was run as written.

Component(s)

C++, Python

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions