Skip to content

[BUG]: cudf.read_parquet with columns=[] gives a shape-(0, 0) dataframe #24523

Closed

Description

@TomAugspurger

Describe the bug

cudf.read_parquet with columns=[] returns an empty dataframe with shape (0, 0). The correct shape should (probably) be (nrows, 0).

Steps/Code to reproduce bug

import io

import cudf
import pandas as pd

df = pd.DataFrame({"a": [1, 2, 3]})
b = io.BytesIO()
df.to_parquet(b)
empty = cudf.read_parquet(b, columns=[])
assert empty.shape == (3, 0), empty.shape  # true for pyarrow=25.x, false for pyarrow=26.x

which gives

AssertionError                            Traceback (most recent call last)
Cell In[12], line 10
      8 df.to_parquet(b)
      9 empty = cudf.read_parquet(b, columns=[])
---> 10 assert empty.shape == (3, 0), empty.shape  # true for pyarrow=25.x, false for pyarrow=26.x

AssertionError: (0, 0)

Expected behavior

No assertion error; the shape should be (3, 0).

Environment overview (please complete the following information)

Environment details

In [13]: cudf.__version__
Out[13]: '26.12.00a342.post261009155743'

Additional context

The behavior of pandas depends on the runtime version of pyarrow. prior to apache/arrow#51151 (released in pyarrow 26) pandas gives (0, 0), and we match that. With pyarrow>=26, pandas gives (3, 0).

I think the pandas behavior was a bug (apache/arrow#51151 (comment)) and the new result of (3, 0) is correct, so that's what we should try to meet.

#24520 pinned pyarrow<26 for now. We can remove / bump that pin once this fixed.

Activity

  1. akhil9tiet commented on Oct 9, 2026

    @akhil9tiet

    I'd like to work on this — please assign it to me. I'll open a PR shortly.

  2. TomAugspurger commented on Oct 9, 2026

    @TomAugspurger
    ContributorAuthor

    Thanks @akhil9tiet . One thing that needs to be determined is how to handle pandas compatibility: do we try to match the version of pandas/pyarrow installed at runtime? Right now, we're leaning towards trying to match pandas/pyarrow.

    So the expected result will depend on what version of arrow you have installed and it'll always match what pandas happens to give you.

  3. vyasr commented on Oct 15, 2026

    @vyasr
    Contributor

    Yes, let's stick with that plan.

  4. bdice commented on Oct 17, 2026

    @bdice
    Contributor

    I'm hitting this bug right now in the 26.10 hotfix. I don't want to add an upper bound pin on pyarrow<26.0.0 for a 26.10.1 hotfix release when no upper bound pin existed in the 26.10.0 release.

    I'm going to file a proper fix for this targeting main. I will match the behavior of the installed PyArrow version and will backport it for the hotfix PR #24652.

  5. added a commit that references this issue on Oct 19, 2026
    3068f8a
  6. added a commit that references this issue on Oct 19, 2026
    5c8dc06
  7. added a commit that references this issue on Oct 19, 2026
    900023b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcudf.pandasIssues specific to cudf.pandas

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions