Repository navigation
Conversation
|
|
|
Thanks for continuing to push this forward Rok! I think this PR is a great next step and is quite clear to review. I have checked out the PR locally and started with running type checkers ( The reason for these warnings and errors in I propose to push the changes directly if we agree the additions makes sense. Meanwhile I will continue with running the |
|
I have run I have also run mypy's stubtest with Besides the reports mentioned above which will be fixed in subsequent PRs, stubtest did found one bug in pyarrow: Note missing "L" in "Decimal". I will work on the fix for this now and would suggest to add it as part of this PR. Can commit directly, if that is ok. |
|
I've run Running on editable dev build 27, with pyright==1.1.414 in my conda env, I also hit errors |
That makes sense and please go for it. I think we should aim for
Huh. I've never noticed these were missing. Given we're not really using them as types I'd skip them if we can.
Huh. Let's do the fix as separate PR, but also include a fix here? (And we'll just rebase over the actual fix on main)
I'm not sure we want (can) bring in ruff as a CI dependency. However we should autoformat before we merge, so we don't do it later and change history for no good reason. So I propose one of you runs the ruff formatter and commits the change into this branch. |
I think
Sounds good. Will open an issue and a PR and add a fix here also. |
Yeah, they are internal and exposed through |
Ok, I'll do that reformatting in a bit. |
|
I have started with reviewing pyarrow tests with running click for output$ mypy --check-untyped-defs --warn-unused-ignores pyarrow/tests/test_array.py
pyarrow/tests/test_array.py:21: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:22: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:25: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:80: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:790: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:986: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:1300: error: List item 3 has incompatible type "None"; expected "int" [list-item]
pyarrow/tests/test_array.py:1302: error: List item 1 has incompatible type "None"; expected "list[tuple[bytes, int]]" [list-item]
pyarrow/tests/test_array.py:1313: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:1320: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:1324: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:1348: error: List item 0 has incompatible type "None"; expected "list[tuple[bytes, int]]" [list-item]
pyarrow/tests/test_array.py:1348: error: List item 1 has incompatible type "None"; expected "list[tuple[bytes, int]]" [list-item]
pyarrow/tests/test_array.py:1362: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:2513: error: Need type annotation for "buffers" (hint: "buffers: list[<type>] = ...") [var-annotated]
pyarrow/tests/test_array.py:2539: error: "object" has no attribute "dtype" [attr-defined]
pyarrow/tests/test_array.py:2541: error: Value of type "object" is not indexable [index]
pyarrow/tests/test_array.py:2542: error: Value of type "object" is not indexable [index]
pyarrow/tests/test_array.py:2543: error: Value of type "object" is not indexable [index]
pyarrow/tests/test_array.py:2885: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:2917: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:2938: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:2939: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:3183: error: Value of type "object" is not indexable [index]
pyarrow/tests/test_array.py:3299: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:3720: error: Incompatible types in assignment (expression has type "None", variable has type "ndarray[tuple[Any, ...], dtype[Any]]") [assignment]
pyarrow/tests/test_array.py:3729: error: Value of type "object" is not indexable [index]
pyarrow/tests/test_array.py:3740: error: Value of type "object" is not indexable [index]
pyarrow/tests/test_array.py:3817: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:4004: error: Incompatible types in assignment (expression has type "MyArrayInvalid", variable has type "MyArray") [assignment]
pyarrow/tests/test_array.py:4017: error: Incompatible types in assignment (expression has type "MyArray2", variable has type "MyArray") [assignment]
pyarrow/tests/test_array.py:4031: error: Incompatible types in assignment (expression has type "MyArray3", variable has type "MyArray") [assignment]
pyarrow/tests/test_array.py:4095: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:4099: error: Unused "type: ignore" comment [unused-ignore]
pyarrow/tests/test_array.py:4315: error: List item 3 has incompatible type "None"; expected "int" [list-item]
pyarrow/tests/test_array.py:4319: error: List item 5 has incompatible type "None"; expected "int" [list-item]
pyarrow/tests/test_array.py:4324: error: List item 2 has incompatible type "None"; expected "int" [list-item]
pyarrow/tests/test_array.py:4324: error: List item 4 has incompatible type "None"; expected "int" [list-item]
pyarrow/tests/test_array.py:4395: error: List item 1 has incompatible type "None"; expected "int" [list-item]
pyarrow/tests/test_array.py:4407: error: List item 0 has incompatible type "None"; expected "int" [list-item]
Found 40 errors in 1 file (checked 1 source file)Type checkers are currently configured to skip I propose to temporarily add the |
|
Another thing: I can't run |
|
Oh, I was too fast here. Unfortunately Would it make sense to remove I will leave this for discussion and start with reading the stubs and comparing them with the APIs, starting with |
|
Ignore statements not being needed is more likely due to changes to typecheckers than due to changes in stubs. Or they might be needed once we import other parts of the library. Also note camel cased (e.g. reportArgumentType) ignores are likely coming from pyright. So if mypy thinks it's a redundant ignore you might want to check pyright too. |
Let's rather remove |
|
OK, agree, when reviewing I plan to do that later and am focusing on reading the stubs currently. |
|
Ehhh... I've fallen into this interesting rabbit hole - There are over 1.5k mypy errors % mypy --check-untyped-defs --warn-unused-ignores pyarrow/tests/test_convert_builtin.py
pyarrow/tests/test_convert_builtin.py:78: error: Module not callable [operator]
pyarrow/tests/test_convert_builtin.py:79: error: Module not callable [operator]
...
pyarrow/tests/test_convert_builtin.py:168: error: Module not callable [operator]
...
Found 321 errors in 1 file (checked 1 source file)
% mypy --check-untyped-defs --warn-unused-ignores pyarrow/tests/test_*.py
Found 1866 errors in 35 files (checked 44 source files)
% ... | grep -c "Module not callable"
1562test_convert_builtin.py:168 : pa.array(... A stub So I propose we reintroduce subfolder |
|
Just to note: with commit eda3425 stubs |
|
@tadeja I suppose the |
|
@jorisvandenbossche oh, yes, I think |
|
@tadeja @rok I have checked typed.pyi and decided to push a commit (4b97c69) after all =) Instead of suggestions I will post the reasoning behind the changes here:
Future TODO:
I have run
|
Co-authored-by: Rok Mihevc <rok@mihevc.org>
There was a problem hiding this comment.
🟡 Changes recommended
Incomplete and overly restrictive stubs reject supported Buffer operations, array operators and constructors, timezone inputs, and DataType arguments.
5 open issues
What changed in this PR
This PR advances PyArrow’s type-checking support by combining the interdependent internal type-system and core data-structure stubs from #49103 and #49193. The stubs remain excluded from distributed wheels while coverage is incomplete.
Changes:
- Add core API stubs, shared typing helpers, and public re-exports.
- Add static and runtime annotation tests, stubtest validation, and pinned checking tools.
- Integrate checks into development workflows and update documentation and existing tests.
| File | Description |
|---|---|
python/scripts/update_stub_docstrings.py |
Resolve internal stubs through pyarrow.lib. |
python/scripts/run_stubtest.py |
Compare stubs against installed runtime signatures. |
python/requirements-stubs.txt |
Centralize annotation-checking dependencies. |
python/pyproject.toml |
Configure type checkers, annotation tests, and stub linting. |
python/pyarrow/tests/test_types.py |
Add missing assertions and typing suppressions. |
python/pyarrow/tests/test_table.py |
Annotate optional NumPy import fallback. |
python/pyarrow/tests/test_sparse_tensor.py |
Adjust optional imports and SciPy skip conditions. |
python/pyarrow/tests/test_schema.py |
Annotate optional NumPy import fallback. |
python/pyarrow/tests/test_convert_builtin.py |
Annotate optional NumPy import fallback. |
python/pyarrow/tests/test_array.py |
Update scalar assertion and add missing offset assertion. |
python/pyarrow/tensor.pxi |
Correct sparse-tensor parameter documentation. |
python/pyarrow/scalar.pxi |
Correct documented parameter name. |
python/pyarrow/array.pxi |
Fix spacing in a documentation example. |
python/pyarrow-stubs/types.pyi |
Annotate type predicates and type identifiers. |
python/pyarrow-stubs/lib.pyi |
Re-export internal stubs and define MonthDayNano. |
python/pyarrow-stubs/_stubs_typing.pyi |
Define shared aliases, protocols, and synthetic types. |
python/pyarrow-stubs/_compute.pyi |
Provide a compute placeholder. |
python/pyarrow-stubs/__lib/types.pyi |
Annotate data types, fields, schemas, and factories. |
python/pyarrow-stubs/__lib/tensor.pyi |
Annotate dense and sparse tensor APIs. |
python/pyarrow-stubs/__lib/table.pyi |
Annotate tables, record batches, and chunked arrays. |
python/pyarrow-stubs/__lib/scalar.pyi |
Provide scalar placeholders. |
python/pyarrow-stubs/__lib/memory.pyi |
Annotate memory pools and allocation controls. |
python/pyarrow-stubs/__lib/ipc.pyi |
Provide a record-batch reader placeholder. |
python/pyarrow-stubs/__lib/io.pyi |
Add a buffer placeholder. |
python/pyarrow-stubs/__lib/error.pyi |
Annotate exceptions and signal handling. |
python/pyarrow-stubs/__lib/device.pyi |
Annotate devices and memory managers. |
python/pyarrow-stubs/__lib/config.pyi |
Annotate build and runtime information. |
python/pyarrow-stubs/__lib/builder.pyi |
Annotate string builders. |
python/pyarrow-stubs/__lib/array.pyi |
Annotate arrays, constructors, and conversions. |
python/pyarrow-stubs/__lib/__init__.pyi |
Separate internal stubs from public factory names. |
python/pyarrow-stubs/__init__.pyi |
Expose annotated public APIs. |
python/pyarrow-stubs-tests/test_types.py |
Check type factories, schemas, and narrowing. |
python/pyarrow-stubs-tests/test_tensor.py |
Check tensor conversions and annotations. |
python/pyarrow-stubs-tests/test_table.py |
Check tabular APIs and input protocols. |
python/pyarrow-stubs-tests/test_device.py |
Check device properties and hashability. |
python/pyarrow-stubs-tests/test_array.py |
Check array operations, constructors, and invalid inputs. |
python/pyarrow-stubs-tests/stubtest_allowlist.txt |
Record expected runtime/stub mismatches. |
python/pyarrow-stubs-tests/conftest.py |
Supply stub-only types during runtime tests. |
python/pyarrow-stubs-tests/_check.py |
Pair static assertions with runtime checks. |
python/CMakeLists.txt |
Correct stub paths in commented-out installation guidance. |
docs/source/developers/python/development.rst |
Document annotation development and validation. |
ci/scripts/python_wheel_validate_contents.py |
Update the stub source path used for validation. |
ci/scripts/python_test_type_annotations.sh |
Run annotation tests and stubtest in CI. |
.pre-commit-config.yaml |
Add stub linting, formatting, and manual type checks. |
.github/workflows/dev.yml |
Run manual stub-checking hooks in lint CI. |
Review effort: Balanced
| index_type: IntegerType | _DataTypeAliasInput, | ||
| value_type: _DataTypeLike, | ||
| ordered: bool = False, | ||
| ) -> DictionaryType: ... | ||
| def struct( | ||
| fields: _StructFieldsInput, | ||
| ) -> StructType: ... | ||
| def sparse_union( | ||
| child_fields: Iterable[Field], type_codes: Sequence[int] | None = None | ||
| ) -> SparseUnionType: ... | ||
| def dense_union( | ||
| child_fields: Iterable[Field], type_codes: Sequence[int] | None = None | ||
| ) -> DenseUnionType: ... | ||
| def union( | ||
| child_fields: Iterable[Field], | ||
| mode: Literal["sparse", "dense"] | int, | ||
| type_codes: Sequence[int] | None = None, | ||
| ) -> SparseUnionType | DenseUnionType: ... | ||
| def run_end_encoded( | ||
| run_end_type: _RunEndType | _DataTypeAliasInput, |
There was a problem hiding this comment.
Requiring the stub-only
IntegerTypeunion rejects integer types obtained through APIs annotated as returningDataType
That's a real problem
Please accept
DataType | strfor these inputs, matching their runtime contract, rather than requiring callers to cast to synthetic subclasses.
This isn't a good solution though.
The typing-only fix is to not return DataType from wherever the problem originated.
But the downside is you'll likely need overloads, which use the stub-only types.
Alternatively:
- You could make runtime changes, where each unique
DataTypehas it's own class which would be available at runtime - You could keep the runtime classes the same, but expose
NewTypes and export those somewhere which would be available at runtime
Those alternatives solve the runtime-availability aspect - but the reality is that you will run into issues like this without overloads
There was a problem hiding this comment.
The typing-only fix is to not return
DataTypefrom wherever the problem originated.
But that is not possible in all cases, though (even with overloads). So I think the only solution is to actually be less strict in the typing of the input arguments (we already had a similar discussion and change regarding input arguments typed with eg Int64Array instead of Array)
There was a problem hiding this comment.
I would like to agree with Joris and update the PR to be less strict here.
|
Only a note on 7fa510f - this is to fix strict type checking errors "Missing type arguments" ( a) Details
a) python % mypy --strict and numpy 2.5.5 b) pyright (strict) and numpy-2.1.3 pyright (strict) and numpy-2.5.5 errors in CI: For now we don't have strict type checking but I think it's good to fix errors ahead for potential switch to "strict" sometime later. Line 138 in eac50b3 |
|
@github-actions crossbow submit preview-docs |
|
Revision: d085957 Submitted crossbow builds: ursacomputing/crossbow @ actions-fa2fdfd6a0
|
Detailed AI generated open questionsOpen questionsSummary of what is still open on this PR after going through all comments and review threads (head Needs an answer or a decision
Deferred, tracked in an issue
Deferred, no issue yet
Additional findings from a local passChecked against the current head with mypy and pyright; not raised in review so far.
Details |
|
I suppose this is now ready for a wider review! (See #55264 (comment) for potentially useful context) |
@tadeja I think those can partly also be solved by not actually using concrete types like (another point here is that, if we add a specific element type, we could probably say the elements should be |
paddyroddy
left a comment
There was a problem hiding this comment.
No blockers, just listed some thoughts.
| # The previous step already ran the ruff hooks. | ||
| SKIP: ruff-check,ruff-format | ||
| run: | | ||
| pre-commit run --hook-stage manual python-stubs --all-files --color=always --show-diff-on-failure |
There was a problem hiding this comment.
Would recommend prek over pre-commit. More feature rich, friendlier community and faster. Acts as a drop-in replacement.
| pre-commit run --hook-stage manual python-stubs --all-files --color=always --show-diff-on-failure | |
| prek run --hook-stage manual python-stubs --all-files --color=always --show-diff-on-failure |
| # modules that aren't annotated yet, e.g. pyarrow.compute, are treated as Any | ||
| # (see python/pyproject.toml). | ||
| - repo: https://github.com/astral-sh/ruff-pre-commit | ||
| rev: v0.17.2 |
There was a problem hiding this comment.
Are we keeping these up-to-date anywhere? @renovatebot or @dependabot as options, I guess @pre-commit-ci too (although not a fan of anything @pre-commit personally).
| @@ -0,0 +1,20 @@ | |||
| # Type checkers and the type stubs of PyArrow's dependencies, pinned so that | |||
There was a problem hiding this comment.
How will these versions be maintained?
There was a problem hiding this comment.
Could this instead be done through a lock file (uv.lock for example)?
|
|
||
|
|
||
| def check(actual: _T, klass: type[Any] | tuple[type[Any], ...]) -> _T: | ||
| """Assert that ``actual`` is an instance of ``klass`` and return it.""" |
There was a problem hiding this comment.
ha, that seems to be just a funny spelling instead of class which is a reserved word. There are about 250 calls check(actual, klass) within this PR.
We could also rename klass here to expected_type or similar?
There was a problem hiding this comment.
I had seen it was listed everywhere and was wondering if I'd missed something lol.
OK, just remembered to post this - verified and fixed to a more suitable |
| def value_counts(self) -> StructArray: ... | ||
| @staticmethod | ||
| def from_pandas( | ||
| obj: pd.Series | NDArray[Any] | ArrayLike, |
There was a problem hiding this comment.
I propose to remove ArrayLike alias as it is Any today and used only once in array.pyi. This alias is shared with compute stubs (#49201) where the meaning of ArrayLike is a bit different (probably what _pack_compute_args accepts) so I would define a private type alias in compute stubs separately.
We could define _ArrayConstructorInput here though, similar to _ChunkedArrayConstructorInput, and use it in Array.from_pandas, array and asarray stubs.
Happy to push the changes if this sounds reasonable.

Rationale for this change
This is the next step in adding type annotations to PyArrow (#32609). It merges #49103 (internal type system stubs) and #49193 (core data structures) because the two depend on each other and can't be type checked separately.
What changes are included in this PR?
This PR adds stubs for core data structures and types. Annotations live in
pyarrow-stubs/and can be used at development time. Annotations are not packaged into wheels. Annotations are tested in CI withci/scripts/python_test_type_annotations.shand can be linted and tested locally withpre-commit.During review we agreed on no generics for now (could be follow-up) and overloads are used sparingly.
Detailed AI generated description of changes
**Type stubs** (`python/pyarrow-stubs/`)pyarrow.libthat hold data types and core data structures:types,array,table,tensor,builder,memory,device,configanderror.pyarrow.types(theis_*predicates, withTypeIs/TypeGuardnarrowing).pyarrow/__init__.pyinow re-exports the annotated public names explicitly, sopa.array,pa.Table,pa.schemaetc. are typed.io,ipc,scalarand_compute. Their names resolve toAny, exceptBufferandExtensionScalar, which are empty classes for now.pyarrow/__init__.pyiandlib.pyialso keep a module-level__getattr__returningAnyfor names that aren't annotated yet (to be removed before the stubs ship, [CI] Remove all __getattr__(name) -> Any from *.pyi files before next release #48970).Layout
pyarrow-stubs/is now laid out as a PEP 561 stub-only package (pyarrow-stubs/__init__.pyiinstead ofpyarrow-stubs/pyarrow/__init__.pyi), so type checkers and IDEs pick it up fromarrow/python..pxifiles compiled intopyarrow.liblive in a stub-onlypyarrow-stubs/__lib/package and are re-exported fromlib.pyi. A flat layout madepa.array,pa.tableandpa.scalarresolve to stub modules instead of functions.Annotation conventions agreed during review
DataTypesubclasses in_stubs_typing.pyidescribe the results of the type factories.Selfwhen the runtime returns the receiver's class and a concrete class otherwise. Parameters never useSelf.Arrayrather than a specific subclass such asBooleanArrayorInt32Array, because user variables are usually inferred asArray.pa.array()is annotated as returningArray. The cases where it returns aChunkedArrayare tracked in [Python] Let array() return aChunkedArrayonly when asked #55499.asarray()has overloads.__getitem__andasarray).unit,null_placement, ...) areLiteraltypes, as in NumPy and pandas-stubs.NDArray[...].__eq__without__hash__are marked unhashable.Type checking setup
python/pyproject.tomlto checkpyarrow-stubsandpyarrow-stubs-tests. Modules that aren't annotated yet are treated asAny.python/requirements-stubs.txtpins the type checkers and the stub packages of the dependencies. CI and pre-commit both install from it.python-stubspre-commit hooks: ruff lint and format for the.pyifiles run on commit; mypy, pyright and ty run on request (pre-commit run --hook-stage manual python-stubs --all-files) and in the Lint CI job.ci/scripts/python_test_type_annotations.shnow also runs the type stub tests and stubtest against the PyArrow built in CI.python/scripts/run_stubtest.pyrunsmypy.stubtestwith the__libmodules mapped topyarrow.lib, using a small allowlist.Other changes
array.pxi,scalar.pxi,tensor.pxiandtypes.pxi, and comparisons inpyarrow/teststhat were missingassert.# type: ignorecomments on the optional-import fallbacks in a fewpyarrow/testsfiles.CMakeLists.txt(commented-out install block),python_wheel_validate_contents.pyandupdate_stub_docstrings.py.Are these changes tested?
Yes.
The PyArrow test suite (
pyarrow/tests) is not type checked in CI yet.Yes CI runs (
ci/scripts/python_test_type_annotations.sh) which in runs typecheckers and tests inpython/pyarrow-stubs-tests/.Changes are also teste locally via
pre-commit. Once we package annotations into wheels we should expand testing.Are there any user-facing changes?
No. The stubs are still not packaged into wheels.
Was AI used for this PR?
In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.
PR code and description written by:
Reviewed before submission by: