Skip to content

BUG: groupby/factorize on pyarrow dictionary dtype uses stale dictionary entries (GH#71828) - #71828

Merged
jbrockmendel merged 3 commits into
pandas-dev:mainfrom
jbrockmendel:fu-731
Nov 13, 2026
Merged

jbrockmendel merged 3 commits into
pandas-dev:mainfrom
jbrockmendel:fu-731

Conversation

@jbrockmendel

Copy link
Copy Markdown
Member

ArrowExtensionArray.factorize returned a pyarrow dictionary array's stored dictionary as its uniques. That dictionary is not a factorization of the values: it can keep entries no row references (e.g. after a boolean filter), hold duplicate or null entries, and be in any order. So on a dictionary-typed column, such as one read via read_parquet(..., dtype_backend="pyarrow") from a file holding a Categorical, groupby invented groups for filtered-out values, split one value across two groups, and ordered sort=False groups by dictionary order.

enc = pa.array(["a", "b", "c", "a"]).dictionary_encode()
df = pd.DataFrame({"k": pd.array(enc, dtype=pd.ArrowDtype(enc.type)), "v": [1, 2, 3, 4]})
df[df["v"] > 2].groupby("k", sort=False, observed=True)["v"].sum()
# main: a 4, b 0, c 3    branch: c 3, a 4

The fix factorizes the (small) dictionaries' values, maps the indices through those codes, and renumbers by first appearance, so the N values are never hashed. When the chunks' dictionaries together are longer than the array (e.g. many parquet row groups each storing every category), it decodes and factorizes the values instead. Value types with no hash or take kernel (list, struct, string_view) keep the stored dictionary as before, raising where that would mishandle a null (GH-66490).

1M rows / 1000 string keys: factorize 2.2 ms, vs 13-15 ms for decode + re-encode.

Supersedes GH-69024, a different approach written before GH-66490.

AI disclosure: drafted with Claude Code (claude opus 5.5 (high)), which wrote the fix and tests and ran the review loop; I reviewed the result.

…y contents (GH#71827)

The stored dictionary of a pyarrow DictionaryArray can hold unreferenced,
duplicate or null entries, so factorize the dictionary values, map the
indices through those codes, and renumber by first appearance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jbrockmendel jbrockmendel added Bug Groupby Algos Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff Arrow pyarrow functionality labels Nov 1, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jbrockmendel jbrockmendel changed the title BUG: groupby/factorize on pyarrow dictionary dtype uses stale dictionary entries (GH#71827) BUG: groupby/factorize on pyarrow dictionary dtype uses stale dictionary entries (GH#71828) Nov 1, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jbrockmendel
jbrockmendel marked this pull request as ready for review November 2, 2026 21:41
@jbrockmendel jbrockmendel added this to the 3.2 milestone Nov 10, 2026

@fangchenli fangchenli left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a blocker: string_view tests will fail after apache/arrow#50164 land.

@jbrockmendel
jbrockmendel merged commit b650df0 into pandas-dev:main Nov 13, 2026
55 checks passed
@jbrockmendel
jbrockmendel deleted the fu-731 branch November 13, 2026 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Algos Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff Arrow pyarrow functionality Bug Groupby

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants