Skip to content

[Python] Bug in pyarrow.from_pandas() when input has MultiIndex index columns having non-string names #38983

Closed

Description

@qsourav

Describe the bug, including details regarding any error messages, version, and platform.

When the input dataframe has MultiIndex index columns having non-string names, the output is not as expected:

import pyarrow as pa
print(f"version: {pa.__version__}")

import pandas as pd
index = pd.MultiIndex.from_tuples([(10,20), (30,40), (50,60), (70,80)], names = [1,2])
df = pd.DataFrame({"a": [1,2,3,4], "b": [5,6,7,8]}, index=index)
print(df)
print(pa.Table.from_pandas(df))

Outputs:

version: 14.0.1
sys:1: UserWarning: The DataFrame has non-str index name `[1, 1]` which will be converted to string and not roundtrip correctly.
pyarrow.Table
a: int64
b: int64
1: int64
1: int64
----
a: [[1,2,3,4]]
b: [[5,6,7,8]]
1: [[10,30,50,70]]
1: [[10,30,50,70]]

Component(s)

Python

Activity

  1. qsourav commented on Nov 29, 2023

    @qsourav
    Author

    The same seems to be fixed from the below workaround:

    n = len(getattr(index, 'levels', [index]))

    def _get_index_level_values(index):
        n = len(getattr(index, 'levels', [index]))
        if isinstance(index, _pandas_api.pd.MultiIndex):
            return [index.levels[i] for i in range(n)]
        else:
            return [index.get_level_values(i) for i in range(n)]
    
  2. qsourav commented on Nov 29, 2023

    @qsourav
    Author

    Some issues with the "levels" parameter when the index columns have non-string names. Even names = ["1", 2] works fine!

  3. qsourav commented on Nov 29, 2023

    @qsourav
    Author
  4. added
    Status: needs championHigh impact issues which aren't being worked on but require a volunteer to move the task forward.
    on Oct 26, 2026
  5. changed the title [-]Bug in pyarrow.from_pandas() when input has MultiIndex index columns having non-string names[/-] [+][Python] Bug in pyarrow.from_pandas() when input has MultiIndex index columns having non-string names[/+] on Nov 10, 2026
  6. added a commit that references this issue on Nov 12, 2026
  7. added this to the 27.0.0 milestone on Nov 12, 2026
  8. AlenkaF commented on Nov 12, 2026

    @AlenkaF
    Member

    Issue resolved by pull request 55485
    #55485

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Component: PythonStatus: needs championHigh impact issues which aren't being worked on but require a volunteer to move the task forward.Type: bug

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions