Skip to content

[C++][ORC] Writing struct<struct<…>> with a null inner struct shifts the child values #55618

Open

Description

@mhatipog

Writing a nested struct to ORC loses data when the inner struct has a null but the outer struct has none. Every child value after the null row is shifted by one, and the first shifted value is written as 0 (or "" for strings).

import pyarrow as pa, pyarrow.orc as orc
inner = pa.struct([("c", pa.int32())])
t = pa.table({"o": pa.array([{"i": {"c": 1}}, {"i": None}, {"i": {"c": 3}}], pa.struct([("i", inner)]))})
orc.write_table(t, "/tmp/a.orc")
orc.read_table("/tmp/a.orc").to_pylist()
# [{'o': {'i': {'c': 1}}}, {'o': {'i': None}}, {'o': {'i': {'c': 0}}}]   expected c = 3

The ORC Java reader (Spark) returns the same wrong values, so the file itself is wrong.

The same happens wherever a struct with nulls sits below another struct that has no nulls in the batch, also through a list or a map:

S = pa.struct([("c", pa.int32())])
pa.array([{"l": [None, {"c": 1}, {"c": 2}]}], pa.struct([("l", pa.list_(S))]))   # reads back [None, {c: 0}, {c: 1}]
pa.array([{"m": [(1, None), (2, {"c": 2})]}], pa.struct([("m", pa.map_(pa.int32(), S))]))   # reads back (2, {c: 0})
pa.array([[{"i": {"c": 1}}, {"i": None}, {"i": {"c": 3}}]], pa.list_(pa.struct([("i", S)])))   # reads back c = 0

Not affected: a top-level list or map<k, struct> with null elements, list<list<…>> and struct<list> with null lists, and any case where the outer struct also has a null in the batch. A round trip over a few thousand random nested schemas outside that shape found no other differences.

Seen on pyarrow 12.0.1 through 26.0.1 (Linux and Windows).

It looks like a struct's null mask is not passed on to its children when the enclosing struct reports no nulls, so the child gets an extra value for each null row. The writer side is WriteStructBatch in cpp/src/arrow/adapters/orc/util.cc.

Found while cross-checking the same files across several readers.

Activity

  1. emadleo13 commented on Nov 17, 2026

    @emadleo13

    Hello,
    I would like to work on this issue, if it is still open.
    From what I have gathered, the presence of null values ​​within nested structures can cause incorrect values in the ORC output, specifically affecting subsequent child values.
    My plan is to first examine how the WriteStructBatch function (located in cpp/src/arrow/adapters/orc/util.cc) handles null values ​​and child offsets, and then add regression tests covering these structures.
    Before I begin, could you please let me know if anyone else is already working on this issue? Also, does my proposed approach seem reasonable to you?
    if the issue is available i would be happy to be assigned to it...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions