Repository navigation
GH-55650: [C++][ORC] Preserve offsets when normalizing structs - #55662
Open
SabyaXSachi wants to merge 1 commit into
Open
SabyaXSachi wants to merge 1 commit into
SabyaXSachi wants to merge 1 commit into
Conversation
NormalizeArray()'s STRUCT branch rebuilt each child of a struct with struct-level nulls without propagating offsets: the child and parent ArrayData were reconstructed at offset 0 and the validity bitmaps were combined starting at bit 0. Writing a sliced struct column therefore serialized the wrong rows, misaligning the column with the rest of the file. Fix by reusing StructArray::GetFlattenedField() (already used by the SPARSE_UNION branch of the same function) to push the parent slice down to each child and merge parent and child validity at the correct bit positions, then rebasing the struct's own validity bitmap to offset 0 before rebuilding the struct. The null_count() == 0 early return is unchanged. Adds deterministic regression tests covering a sliced struct column with struct-level nulls, a struct whose child has an independent offset, and both combined. Closes apache#55650.
|
|
Member
|
Could you please respect the PR template? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GH-55650: [C++][ORC] Preserve offsets when normalizing structs
Summary
The ORC writer serializes a sliced
structcolumn with struct-level nulls incorrectly: it writes the struct's validity and the child values starting at physical row 0 instead of at the array's offset. No error is raised — the file itself is wrong, and the struct column ends up misaligned with the other columns in the same file. This affects everyday operations such asTable.slice()/table[a:b], or writing chunks produced byto_batches(max_chunksize=...). The same happens when the struct has offset 0 but a child array has its own independent offset.Example (pyarrow): writing
t.slice(1, 4)of[{a:0}, None, {a:2}, {a:3}, None, {a:5}]produces[{'a': 0}, None, {'a': 2}, {'a': 3}]instead of[None, {'a': 2}, {'a': 3}, None], while theidcolumn is written correctly. The Apache ORC C++ reader returns the same wrong values, confirming the file is corrupt, not just the Arrow reader.Root cause
NormalizeArray()incpp/src/arrow/adapters/orc/util.cc(STRUCT branch) runs whennull_count() != 0and rebuilds each child to AND in the struct's validity. In doing so it drops every offset:StructArray::field(i)returns the child already re-sliced to the struct's range (child offset = child offset + struct offset), butBitmapAnd()reads both validity buffers from bit 0 instead of from the parent's and child's offsets,ArrayData::Make(child->type(), child->length(), ...)rebuilds the child without passingchild->offset(), so it defaults to 0 over the original, unsliced buffers, andstd::make_shared<StructArray>(...)drops the struct's own offset while reusing the unsliced validity buffer.Additionally, when a child has no validity bitmap, the parent's raw, unoffset bitmap buffer was assigned directly as the child's validity buffer, which only aligns by coincidence.
GH-15212 / PR #15213 fixed the same class of bug for the LIST/LARGE_LIST/FIXED_SIZE_LIST/MAP branches by passing offsets through; the STRUCT branch was left as it was. The existing
writeSlicedBatchregression test does not reach this branch because it generates its table withprobability=0(no nulls), soNormalizeArraytakes thenull_count() == 0early return.Fix
Use the existing
StructArray::GetFlattenedField()abstraction — the same one the SPARSE_UNION branch of this function already uses. It pushes the parent slice down to each child (child offset = child offset + parent offset) and merges parent and child validity at the correct bit positions, including the case where the child has no validity bitmap. Each flattened child is normalized recursively. The struct's own validity bitmap is then rebased to offset 0 withAllocateEmptyBitmap()+CopyBitmap(), and the struct is rebuilt at offset 0 with its correct length and null count, matching the now offset-free children. Thenull_count() == 0early return is unchanged.Tests
Deterministic regression tests added to
cpp/src/arrow/adapters/orc/adapter_test.cc:writeSlicedBatchWithStructNulls: a struct column with struct-level nulls, sliced with a non-zero offset (table->Slice(1, 4)), round-tripped through the ORC writer/reader.writeStructWithOffsetChild: struct at offset 0 with a child that has its own offset (logical values[13, 14, 15]over physical[10, 11, 12, 13, 14, 15]) and a struct-level null at row 1, round-tripped.writeSlicedStructWithOffsetChild: combines a non-zero struct offset with an independently offset child.Results against the unpatched implementation (built locally at the same revision):
The failure diffs reproduce the corruption reported in the issue exactly (e.g. child values
[10, null, 12]written instead of[13, 14, 15], validity shifted from[false, true, true, false]to[true, false, true, true]). After this patch, all three pass:Validation
cmake -G Ninjawith-DARROW_ORC=ON -DARROW_BUILD_TESTS=ON -DARROW_COMPUTE=ON -DARROW_JSON=ON -DARROW_DEPENDENCY_SOURCE=BUNDLED(Release, MSVC x64), targetarrow-orc-adapter-test.ctest -R arrow-orc-adapter-test— 100% tests passed, 0 failed (44/44) with this patch.TZDIRat an existing zoneinfo tree; those tz-dependent tests fail identically with and without this patch).git diff --check: clean. No unrelated tests were modified.Scope
This PR addresses GH-55650 only. GH-55618 (a struct-normalization bug on the
null_count() == 0early-return path of the same function) is related but distinct and is intentionally not addressed here.AI disclosure
This contribution was developed with the assistance of an AI coding agent. The complete diff was reviewed by the author, who verified the root-cause analysis against the checked-out sources, and the build and regression tests were run locally with the results reported above.
Closes #55650.