Skip to content

fix(python): Honor element slices in struct-backed buffer iteration - #1009

Open
peterolkhov wants to merge 1 commit into
apache:mainfrom
peterolkhov:codex/honor-struct-buffer-slices
Open

peterolkhov wants to merge 1 commit into
apache:mainfrom
peterolkhov:codex/honor-struct-buffer-slices

Conversation

@peterolkhov

Copy link
Copy Markdown
Contributor

Sliced fixed-width binary, half-float, decimal and interval arrays can currently read earlier values or return extra rows because CBufferView._iter_struct() ignores its offset and length. Apply those bounds to a byte memoryview before unpacking, preserving the existing scalar and tuple output. Regression tests cover direct buffer slices, empty slices, bounds and lifetime, plus nullable PyArrow Array and RecordBatch conversion.

Tested locally on macOS arm64/Python3.14: Python suite (284 passed, 4 skipped), doctests (118 passed), stubtest (8 modules), configured applicable pre-commit hooks and a 56-case external PyArrow matrix. Ubuntu and the full CI Python matrix remain for CI.

The code change and regression tests were prepared with OpenAI Codex assistance (GPT-6.1 Sol).

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

One slight simplification that I believe is equivalent (but check!)

Comment on lines +354 to +355
view = memoryview(self).cast("B")
view = view[offset * self.itemsize:(offset + length) * self.itemsize]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can just do this?

Suggested change
view = memoryview(self).cast("B")
view = view[offset * self.itemsize:(offset + length) * self.itemsize]
view = memoryview(self)[offset:(offset + length)]

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants