Repository navigation
GH-51043: [Python] Reject null required Arrow objects - #51161
Conversation
|
|
There was a problem hiding this comment.
🟡 Changes recommended
DictionaryArray.from_buffers() still allows dictionary=None while dereferencing it unconditionally, so a None argument can still trigger a crash.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR hardens several PyArrow Cython entry points by rejecting None for required Arrow extension objects, preventing null C++ pointer dereferences that can segfault the Python interpreter (GH-51043).
Changes:
- Mark required
Schema,FileFormat, andDataTypeparameters as non-nullable (not None) at the Cython boundary to raiseTypeErrorinstead of crashing. - Add an explicit
Nonecheck forFileSystemDatasetfragments before unwrapping. - Add focused regression tests covering the newly rejected
Nonearguments.
File summaries
| File | Description |
|---|---|
| python/pyarrow/_dataset.pyx | Reject None fragments and make schema/format non-nullable in FileSystemDataset. |
| python/pyarrow/_parquet.pyx | Make SortingColumn conversion helpers reject schema=None at the boundary. |
| python/pyarrow/array.pxi | Make DictionaryArray.from_buffers reject type=None at the boundary. |
| python/pyarrow/tests/test_dataset.py | Add regression assertions for FileSystemDataset(..., schema=None/format=None) and [None] fragments. |
| python/pyarrow/tests/parquet/test_metadata.py | Add regression assertions for SortingColumn.* with schema=None. |
| python/pyarrow/tests/test_array.py | Add regression assertion for DictionaryArray.from_buffers(type=None, ...). |
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
There was a problem hiding this comment.
🟢 Approval recommended
The changes directly address the null-dereference crash at the Cython boundary and are covered by targeted regression tests for the reported cases.
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 0 new
- Review effort level: Lite
AlenkaF
left a comment
There was a problem hiding this comment.
Are there any other instances of this pattern anywhere else in the Cython bindings? They might produce other errors (like AttributeError) but would still benefit from similar change.
Ah, ok, there is another PR aiming at other files in PyArrow: #51163 For future work we could try keeping the number of PRs down and tackle similar issues in one. |
|
|
1 similar comment
|
|
846a3ac to
bb913b6
Compare
Signed-off-by: 1fanwang <1fannnw@gmail.com>
Signed-off-by: 1fanwang <1fannnw@gmail.com>
Signed-off-by: 1fanwang <1fannnw@gmail.com>
…tors Raise TypeError instead of aborting the interpreter when ScanNodeOptions, ProjectNodeOptions, ParquetSchema, ColumnSchema, or SubTreeFileSystem receive None for a required argument. Signed-off-by: 1fanwang <1fannnw@gmail.com>
bb913b6 to
cafded2
Compare
|
|
…-none-segfaults Signed-off-by: 1fanwang <1fannnw@gmail.com>
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
3 open issues
These tests match exact Cython-generatedTypeErrorphrasing (e.g., \"has incorrect type\"). That… · New Thefs/SubTreeFileSystemsubprocess case isn’t marked with a feature marker (whileacero,… · New The newTypeErroris correct, but it’s not very actionable whenfragmentsis long (it doesn’t… · New
Review effort: Lite
|
Thank you for updating and including other similar checks. Added one more comment on |
…sses The test now calls the constructors in process under pytest.raises, as other tests do, and skips a case when its module is not built. Signed-off-by: 1fanwang <1fannnw@gmail.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Approval should wait until the cross-cutting Cython boundary changes are validated in generated extensions and the documented crash regressions are confirmed across supported build configurations.
No open issues 🎉
3 resolved since last review
Review effort: Lite
Put the Acero, Parquet schema and SubTreeFileSystem checks in the tests for those modules, and drop the string buffer test, which covers behavior this PR leaves unchanged. Signed-off-by: 1fanwang <1fannnw@gmail.com>


Rationale for this change
Passing None where pyarrow requires an object can abort the Python process or raise an unrelated AttributeError, because the value reaches native Arrow code that cannot handle it. The original report covered some constructors, and later comments found the same failure in ScanNodeOptions, ParquetSchema, ColumnSchema, SubTreeFileSystem and ProjectNodeOptions, so this PR fixes those too.
Fixes #51043.
get_partition_keys(None) is in #51294.
What changes are included in this PR?
The Python layer now checks required arguments before native code runs, so None raises a TypeError that names the argument instead of crashing. This covers schemas, formats, types, dictionaries, fragments, filesystems and child arrays, plus the classes listed above. Array buffer constructors also require a data type.
Are these changes tested?
A native Arrow 26.0.0-SNAPSHOT build reproduced the failures. The same commands then raised TypeError.
ProjectNodeOptions([None]) and SubTreeFileSystem("", None) aborted with SIGBUS. ColumnSchema(None, 0) aborted on .name with SIGSEGV. All three now raise TypeError.
Are there any user-facing changes?
Missing required objects raise TypeError. Valid null buffers keep their existing behavior.
New Contributor's Guide |
Contributing Overview |
AI-generated Code Guidance
Was AI used for this PR?
AI assisted with the code, tests, and description.
PR code and description written by:
Reviewed before submission by:
None#51043