Skip to content

GH-51293: [Python] Reject a null Expression in two public APIs - #51294

Merged
AlenkaF merged 9 commits into
apache:mainfrom
1fanwang:1fannnw/gh-51293-partition-keys-none
Nov 16, 2026
Merged

AlenkaF merged 9 commits into
apache:mainfrom
1fanwang:1fannnw/gh-51293-partition-keys-none

Conversation

@1fanwang

@1fanwang 1fanwang commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Passing None to partition-key extraction or expression equality terminates Python instead of raising an exception. Applications cannot catch that failure.

Fixes #51293.

What changes are included in this PR?

Both expression parameters reject None at the Cython boundary. Regression tests cover these two entry points.

Are these changes tested?

Built from source on macOS arm64 with Python 3.12.14. Both calls crash on upstream main and raise TypeError on the rebased branch.

$ python -c 'import pyarrow.dataset as ds; ds.get_partition_keys(None)'
Before:
returncode=-10
After:
TypeError: Argument 'partition_expression' has incorrect type (expected pyarrow._compute.Expression, got NoneType)
returncode=1

$ python -c "import pyarrow.compute as pc; pc.field('a').equals(None)"
Before:
returncode=-10
After:
TypeError: Argument 'other' has incorrect type (expected pyarrow._compute.Expression, got NoneType)
returncode=1

Are there any user-facing changes?

Null expressions raise TypeError instead of terminating Python. Valid calls keep their existing behavior.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

@1fanwang
1fanwang requested a review from AlenkaF as a code owner September 10, 2026 18:43
Copilot AI lite review requested due to automatic review settings September 10, 2026 18:43
@1fanwang
1fanwang requested review from raulcd and rok as code owners September 10, 2026 18:43
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #51293 has been automatically assigned in GitHub to PR creator.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is minimal, directly addresses a crash, and includes a focused regression test validating the new behavior.

Pull request overview

This PR fixes a hard crash in the PyArrow Dataset Python API by rejecting a None partition expression at the Cython boundary, converting an interpreter “bus error” into a normal Python TypeError and adding a regression test.

Changes:

  • Mark get_partition_keys’s partition_expression parameter as not None in python/pyarrow/_dataset.pyx so None is rejected before dereferencing.
  • Add a regression test ensuring ds.get_partition_keys(None) raises TypeError.
File summaries
File Description
python/pyarrow/_dataset.pyx Adds not None to the typed Expression parameter to prevent null from reaching Cython internals.
python/pyarrow/tests/test_dataset.py Adds a regression test asserting TypeError is raised for None input.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI review requested due to automatic review settings September 10, 2026 18:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new subprocess-based sweep test can hang indefinitely and may miss post-“DONE” crashes unless it asserts returncode == 0 and uses a timeout.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread python/pyarrow/tests/test_misc.py Resolved
Comment thread python/pyarrow/tests/test_misc.py Resolved
Copilot AI review requested due to automatic review settings September 10, 2026 20:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new subprocess-based sweep tests need a timeout (to avoid hanging CI) and a couple of small documentation/comment fixes for correctness and debuggability.

Review details

Suppressed comments (4)

Previously missed (1) — in code that hasn't changed since the last review.

python/pyarrow/tests/test_misc.py:319

  • This subprocess sweep has no timeout, so a single blocking callable can hang the entire test run. Consider adding a reasonable timeout (and include stderr in the failure) to make CI failures deterministic and easier to debug.

This issue also appears on line 377 of the same file.

python/pyarrow/tests/test_misc.py:280

  • The comment says passing None to any public callable must produce a Python exception, but the test intentionally allows callables to accept None (it only asserts the subprocess doesn't crash). Please update the comment to match the actual contract (no crash).
    # GH-51293: a typed Cython parameter that is not declared "not None"
    # lets None reach code that dereferences it, killing the interpreter
    # instead of raising. Passing None to any public callable must produce
    # a Python exception, never a fatal signal.

python/pyarrow/tests/test_misc.py:381

  • Same issue as the callable sweep above: this subprocess.run has no timeout and the failure message drops stderr, which can lead to hung CI jobs and harder debugging when something goes wrong.
    res = subprocess.run([sys.executable, "-c", code],
                         capture_output=True, text=True)
    lines = res.stdout.splitlines()
    if not lines or lines[-1] != "DONE":
        culprit = lines[-1] if lines else "<no output>"

python/pyarrow/_compute.pyx:2687

  • The equals() docstring parameter type references pyarrow.dataset.Expression, but this is pyarrow.compute.Expression (and the method signature enforces that). This mismatch can confuse users reading the docs/help().
        Parameters
        ----------
        other : pyarrow.dataset.Expression

  • Files reviewed: 4/4 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings September 10, 2026 20:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The functional changes directly address the reported interpreter-crash bug and are covered by targeted and guard-style tests; remaining feedback is limited to minor doc/test robustness improvements.

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

python/pyarrow/_compute.pyx:2687

  • The Expression.equals docstring references pyarrow.dataset.Expression, but this type lives in pyarrow.compute (and this method is defined in pyarrow/_compute.pyx). Updating the docstring avoids confusing API users.
    def equals(self, Expression other not None):
        """
        Parameters
        ----------
        other : pyarrow.dataset.Expression

python/pyarrow/tests/test_misc.py:363

  • The subprocess method sweep uses inspect._empty, which is a private inspect implementation detail. Prefer the public inspect.Parameter.empty to reduce risk of breakage across Python versions.
  • Files reviewed: 4/4 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@1fanwang 1fanwang changed the title GH-51293: [Python] Reject a null partition expression GH-51293: [Python] Reject a null Expression in two public APIs Sep 10, 2026
@AlenkaF

AlenkaF commented Sep 11, 2026

Copy link
Copy Markdown
Member

Thanks for the PR.

Since a hand-audit goes stale, this adds two guard tests that pass None to public
callables in a child process and fail naming the offender if the interpreter dies.
One covers module-level functions, the other methods reached from a live object. The
method sweep found Expression.equals after the function sweep came back clean.

I am not sure this is needed. @raul what do you think?

I would rather see a missing test for the Expression.equals(None) to be added.

Copilot AI review requested due to automatic review settings September 12, 2026 05:18
@1fanwang

Copy link
Copy Markdown
Contributor Author

Done in 4415d41. This adds the focused Expression.equals(None) regression.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The subprocess guard can exercise unrelated crashing methods and may skip the intended Cython methods, so it does not reliably validate the fix.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

python/pyarrow/tests/test_misc.py:360

  • This method guard is effectively vacuous for the Cython methods it is meant to cover. The child process uses stdlib inspect, but this repository's Cython build does not reliably expose __text_signature__; dev/archery/archery/lang/python.py:169-184 adds a docstring parser specifically because inspect.signature fails. This except therefore skips Expression.equals and similar methods, so the test can pass without exercising the regression; use the repository's Cython signature parser or explicitly include the methods under test.
                    sig = inspect.signature(attr)
                except BaseException:
                    continue
  • Files reviewed: 5/5 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread python/pyarrow/tests/test_misc.py Resolved
Copilot AI review requested due to automatic review settings September 22, 2026 01:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

All reviewed changes address the null-expression crash with regression coverage.

Review effort: Lite
Findings: None

Resolved since last review (1)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The reviewed changes address null-expression handling with regression coverage and have no blocking issues.

Review effort: Lite
Findings: None

get_partition_keys took an Expression without rejecting None, so passing
None dereferenced a null pointer and terminated the interpreter.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
…lables

Sweeps public module-level callables with None in a child process and
fails with the offending name if the interpreter dies by signal, so a
typed Cython parameter missing "not None" is caught as a test failure
rather than a segfault in someone else's code.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
The sweep in the previous commit only covered module-level functions.
Extending it to methods reached from a live object found Expression.equals
taking a typed Expression without rejecting None, which segfaulted.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
…cisely

Add a timeout so a blocking callable cannot hang CI, fail on a non-zero
returncode so a crash during interpreter shutdown is caught, and correct
the comment: some public APIs accept None, so the invariant is only that
none of them terminate the interpreter.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
Signed-off-by: 1fanwang <1fannnw@gmail.com>
The two Expression guards should not depend on unrelated APIs accepting None.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
@1fanwang
1fanwang force-pushed the 1fannnw/gh-51293-partition-keys-none branch from e96bab5 to 94e0f89 Compare September 25, 2026 20:49
Copilot AI review requested due to automatic review settings September 25, 2026 20:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The reviewed changes address the null-expression crashes and include regression coverage.

Review effort: Lite
Findings: None

-partition-keys-none

Signed-off-by: 1fanwang <1fannnw@gmail.com>
Copilot AI lite review requested due to automatic review settings November 10, 2026 17:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues

Review effort: Lite

Comment thread python/pyarrow/tests/test_misc.py Resolved
Comment thread python/pyarrow/tests/test_misc.py Resolved
AlenkaF pushed a commit that referenced this pull request Nov 13, 2026
### Rationale for this change

Passing None where pyarrow requires an object can abort the Python process or raise an unrelated AttributeError, because the value reaches native Arrow code that cannot handle it. The original report covered some constructors, and later comments found the same failure in ScanNodeOptions, ParquetSchema, ColumnSchema, SubTreeFileSystem and ProjectNodeOptions, so this PR fixes those too.

Fixes #51043.

get_partition_keys(None) is in #51294.

### What changes are included in this PR?

The Python layer now checks required arguments before native code runs, so None raises a TypeError that names the argument instead of crashing. This covers schemas, formats, types, dictionaries, fragments, filesystems and child arrays, plus the classes listed above. Array buffer constructors also require a data type.

### Are these changes tested?

A native Arrow 26.0.0-SNAPSHOT build reproduced the failures. The same commands then raised TypeError.

```console
$ python -c 'from pyarrow import acero; acero.ScanNodeOptions(None)'
Before: process aborted (SIGBUS)
After: TypeError: Argument 'dataset' has incorrect type (expected pyarrow._dataset.Dataset, got NoneType)

$ python -c 'import pyarrow.parquet as pq; pq.ParquetSchema(None)'
Before: process aborted (SIGSEGV)
After: TypeError: Argument 'container' has incorrect type (expected pyarrow._parquet.FileMetaData, got NoneType)

$ python -c 'import pyarrow as pa; pa.Array.from_buffers(type=None, length=0, buffers=[])'
Before: AttributeError: 'NoneType' object has no attribute 'num_fields'
After: TypeError: Argument 'type' has incorrect type (expected pyarrow.lib.DataType, got NoneType)
```

ProjectNodeOptions([None]) and SubTreeFileSystem("", None) aborted with SIGBUS. ColumnSchema(None, 0) aborted on .name with SIGSEGV. All three now raise TypeError.

### Are there any user-facing changes?

Missing required objects raise TypeError. Valid null buffers keep their existing behavior.

[New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request) |
[Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html) |
[AI-generated Code Guidance](https://arrow.apache.org/docs/dev/developers/overview.html#ai-generated-code)

### Was AI used for this PR?

AI assisted with the code, tests, and description.

**PR code and description written by:**

- [ ] Human
- [x] AI

**Reviewed before submission by:**

- [x] Human
- [ ] AI
- [ ] Not reviewed

* GitHub Issue: #51043

Authored-by: 1fanwang <1fannnw@gmail.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
Signed-off-by: 1fanwang <1fannnw@gmail.com>
Copilot AI lite review requested due to automatic review settings November 16, 2026 02:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported, and the change includes targeted regression coverage for safe TypeError handling in both affected APIs.

No open issues 🎉

2 resolved since last review

Review effort: Lite

Comment thread python/pyarrow/tests/test_misc.py Outdated
("ds.get_partition_keys(None)", "partition_expression"),
("pc.field('a').equals(None)", "other"),
])
def test_expression_apis_reject_none_without_crashing(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we simply remove this test? (as in all other PRs before, pytest.raises should be enough)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e12af6a.

The pytest.raises tests in test_compute.py and test_dataset.py already cover
both entry points, so the child-process test is redundant.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
Copilot AI lite review requested due to automatic review settings November 16, 2026 05:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active issues remain, with the reviewed API guards and regression tests addressing null-expression handling in both affected PyArrow APIs.

No open issues 🎉

Review effort: Lite

@AlenkaF
AlenkaF merged commit dc7ee12 into apache:main Nov 16, 2026
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Python] get_partition_keys segfaults when the partition expression is None

4 participants