Skip to content

[C++][Python] sort_by produces incorrect result #45686

Closed

Description

@roosephu

Describe the bug, including details regarding any error messages, version, and platform.

To reproduce:

import pyarrow
import numpy as np

print(pyarrow.__version__)

N = 2**30 // 4

data = {
    "id": [4, 3, 2, 1],
    "data": [np.zeros(N, dtype=np.int64) + i for i in range(4)],
}
table = pyarrow.Table.from_pydict(data)

table2 = table.sort_by("id")
print(table2)

Actual output:

pyarrow.__version__ = '19.0.1'
pyarrow.Table
id: int64
data: list<item: int64>
  child 0, item: int64
----
id: [[1,2,3,4]]
data: [[[1,1,1,1,1,...,1,1,1,1,1],[0,0,0,0,0,...,0,0,0,0,0],[1,1,1,1,1,...,1,1,1,1,1],[0,0,0,0,0,...,0,0,0,0,0]]]

Changing dtype to np.int8 gives the correct output:

pyarrow.__version__ = '19.0.1'
pyarrow.Table
id: int64
data: list<item: int8>
  child 0, item: int8
----
id: [[1,2,3,4]]
data: [[[3,3,3,3,3,...,3,3,3,3,3],[2,2,2,2,2,...,2,2,2,2,2],[1,1,1,1,1,...,1,1,1,1,1],[0,0,0,0,0,...,0,0,0,0,0]]]

My guess is that it overflows when calculating offsets during sorting, although I have no idea how pyarrow works internally.

Component(s)

Python

Activity

  1. changed the title [-]sort_by produces incorrect result[/-] [+][C++][Python] sort_by produces incorrect result[/+] on Mar 7, 2025
  2. added
    Critical FixBugfixes for security vulnerabilities, crashes, or invalid data.
    on Mar 7, 2025
  3. thisisnic commented on Nov 18, 2026

    @thisisnic
    Member

    Could not reproduce on 26.0.0 with a lower-memory version of the reprex, which does fail on 19.0.1. Appears to have been fixed in
    #49602.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions