Skip to content

GH-55650: [C++][ORC] Preserve offsets when normalizing structs - #55662

Open
SabyaXSachi wants to merge 1 commit into
apache:mainfrom
SabyaXSachi:fix/orc-sliced-struct-offset
Open

SabyaXSachi wants to merge 1 commit into
apache:mainfrom
SabyaXSachi:fix/orc-sliced-struct-offset

Conversation

@SabyaXSachi

@SabyaXSachi SabyaXSachi commented Nov 17, 2026 •

Copy link
Copy Markdown

GH-55650: [C++][ORC] Preserve offsets when normalizing structs

Summary

The ORC writer serializes a sliced struct column with struct-level nulls incorrectly: it writes the struct's validity and the child values starting at physical row 0 instead of at the array's offset. No error is raised — the file itself is wrong, and the struct column ends up misaligned with the other columns in the same file. This affects everyday operations such as Table.slice() / table[a:b], or writing chunks produced by to_batches(max_chunksize=...). The same happens when the struct has offset 0 but a child array has its own independent offset.

Example (pyarrow): writing t.slice(1, 4) of [{a:0}, None, {a:2}, {a:3}, None, {a:5}] produces [{'a': 0}, None, {'a': 2}, {'a': 3}] instead of [None, {'a': 2}, {'a': 3}, None], while the id column is written correctly. The Apache ORC C++ reader returns the same wrong values, confirming the file is corrupt, not just the Arrow reader.

Root cause

NormalizeArray() in cpp/src/arrow/adapters/orc/util.cc (STRUCT branch) runs when null_count() != 0 and rebuilds each child to AND in the struct's validity. In doing so it drops every offset:

  1. StructArray::field(i) returns the child already re-sliced to the struct's range (child offset = child offset + struct offset), but
  2. BitmapAnd() reads both validity buffers from bit 0 instead of from the parent's and child's offsets,
  3. ArrayData::Make(child->type(), child->length(), ...) rebuilds the child without passing child->offset(), so it defaults to 0 over the original, unsliced buffers, and
  4. std::make_shared<StructArray>(...) drops the struct's own offset while reusing the unsliced validity buffer.

Additionally, when a child has no validity bitmap, the parent's raw, unoffset bitmap buffer was assigned directly as the child's validity buffer, which only aligns by coincidence.

GH-15212 / PR #15213 fixed the same class of bug for the LIST/LARGE_LIST/FIXED_SIZE_LIST/MAP branches by passing offsets through; the STRUCT branch was left as it was. The existing writeSlicedBatch regression test does not reach this branch because it generates its table with probability=0 (no nulls), so NormalizeArray takes the null_count() == 0 early return.

Fix

Use the existing StructArray::GetFlattenedField() abstraction — the same one the SPARSE_UNION branch of this function already uses. It pushes the parent slice down to each child (child offset = child offset + parent offset) and merges parent and child validity at the correct bit positions, including the case where the child has no validity bitmap. Each flattened child is normalized recursively. The struct's own validity bitmap is then rebased to offset 0 with AllocateEmptyBitmap() + CopyBitmap(), and the struct is rebuilt at offset 0 with its correct length and null count, matching the now offset-free children. The null_count() == 0 early return is unchanged.

Tests

Deterministic regression tests added to cpp/src/arrow/adapters/orc/adapter_test.cc:

  • writeSlicedBatchWithStructNulls: a struct column with struct-level nulls, sliced with a non-zero offset (table->Slice(1, 4)), round-tripped through the ORC writer/reader.
  • writeStructWithOffsetChild: struct at offset 0 with a child that has its own offset (logical values [13, 14, 15] over physical [10, 11, 12, 13, 14, 15]) and a struct-level null at row 1, round-tripped.
  • writeSlicedStructWithOffsetChild: combines a non-zero struct offset with an independently offset child.

Results against the unpatched implementation (built locally at the same revision):

[  FAILED  ] TestORCWriterTrivialNoConversion.writeSlicedBatchWithStructNulls
[  FAILED  ] TestORCWriterTrivialNoConversion.writeStructWithOffsetChild
[  FAILED  ] TestORCWriterTrivialNoConversion.writeSlicedStructWithOffsetChild
[  PASSED  ] 0 tests.

The failure diffs reproduce the corruption reported in the issue exactly (e.g. child values [10, null, 12] written instead of [13, 14, 15], validity shifted from [false, true, true, false] to [true, false, true, true]). After this patch, all three pass:

[       OK ] TestORCWriterTrivialNoConversion.writeSlicedBatchWithStructNulls
[       OK ] TestORCWriterTrivialNoConversion.writeStructWithOffsetChild
[       OK ] TestORCWriterTrivialNoConversion.writeSlicedStructWithOffsetChild
[  PASSED  ] 3 tests.

Validation

  • Build: cmake -G Ninja with -DARROW_ORC=ON -DARROW_BUILD_TESTS=ON -DARROW_COMPUTE=ON -DARROW_JSON=ON -DARROW_DEPENDENCY_SOURCE=BUNDLED (Release, MSVC x64), target arrow-orc-adapter-test.
  • Full ORC adapter suite: ctest -R arrow-orc-adapter-test — 100% tests passed, 0 failed (44/44) with this patch.
  • Without the patch, the same suite passes everything except the three new tests (the remaining failures were a local environment issue: the ORC writer needs an IANA time zone database on Windows, fixed locally by pointing TZDIR at an existing zoneinfo tree; those tz-dependent tests fail identically with and without this patch).
  • git diff --check: clean. No unrelated tests were modified.

Scope

This PR addresses GH-55650 only. GH-55618 (a struct-normalization bug on the null_count() == 0 early-return path of the same function) is related but distinct and is intentionally not addressed here.

AI disclosure

This contribution was developed with the assistance of an AI coding agent. The complete diff was reviewed by the author, who verified the root-cause analysis against the checked-out sources, and the build and regression tests were run locally with the results reported above.

Closes #55650.

NormalizeArray()'s STRUCT branch rebuilt each child of a struct with
struct-level nulls without propagating offsets: the child and parent
ArrayData were reconstructed at offset 0 and the validity bitmaps were
combined starting at bit 0. Writing a sliced struct column therefore
serialized the wrong rows, misaligning the column with the rest of the
file.

Fix by reusing StructArray::GetFlattenedField() (already used by the
SPARSE_UNION branch of the same function) to push the parent slice down
to each child and merge parent and child validity at the correct bit
positions, then rebasing the struct's own validity bitmap to offset 0
before rebuilding the struct. The null_count() == 0 early return is
unchanged.

Adds deterministic regression tests covering a sliced struct column
with struct-level nulls, a struct whose child has an independent
offset, and both combined.

Closes apache#55650.
@SabyaXSachi
SabyaXSachi requested a review from wgtmac as a code owner November 17, 2026 19:50
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #55650 has been automatically assigned in GitHub to PR creator.

@wgtmac

wgtmac commented Nov 18, 2026

Copy link
Copy Markdown
Member

Could you please respect the PR template?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Python][C++][ORC] Writer ignores the array offset of struct columns with nulls and writes the wrong rows

2 participants