Skip to content

fix(python): Respect union slice offsets and lengths when iterating - #1010

Merged
paleolimbot merged 1 commit into
apache:mainfrom
peterolkhov:codex/fix-sparse-union-slice-offsets
Nov 11, 2026
Merged

paleolimbot merged 1 commit into
apache:mainfrom
peterolkhov:codex/fix-sparse-union-slice-offsets

Conversation

@peterolkhov

Copy link
Copy Markdown
Contributor

Sliced sparse unions currently select child values from earlier rows because their child-run cursor omits the parent offset. For example, slicing [10, 'twenty', 30, 'forty'] at offset 2 produces [10, 'twenty'] instead of [30, 'forty']. Start sparse child runs at the effective parent offset and read exactly the requested number of type IDs in both union iterators. The exact-length correction also prevents an extra row when a dense union is a child of a sliced struct.

Adds regression coverage for sparse/dense slices, custom type codes, nullable children, child offsets, empty slices and nested list/struct/RecordBatch conversion.

Tested locally on macOS arm64/Python 3.14/PyArrow 26: 276 Python tests passed (4 skipped), 118 doctests passed, stubtest passed for 8 modules, applicable configured pre-commit hooks passed. Built and installed the changed source wheel with two compilation jobs. Ubuntu/Python-version matrix and sdist/twine CI were not run locally.

AI assistance: OpenAI Codex generated the patch and tests.

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@paleolimbot
paleolimbot merged commit af80853 into apache:main Nov 11, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants