Repository navigation
GH-55263: [Python][Annotations] Add internal types and core data structures #55264
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
rok
wants to merge
132
commits into
apache:main
Choose a base branch
from
rok:pyarrow-stubs-pr3-core-data-sans-generic
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+4,719
−116
Open
Changes from all commits
Commits
Show all changes
132 commits
Select commit
Hold shift + click to select a range
9404659
Add internal types and helpers
rok 5f7cdc8
Update python/pyarrow-stubs/pyarrow/_stubs_typing.pyi
rok 610c48d
post rebase changes
rok 7bd5d7c
linting and formatting
rok 2f1a0ba
add pyi check to pre-commit
rok 42ab12d
ruff format
rok 675ae64
Apply suggestions from code review
rok 14b924b
review feedback
rok 4ec84e4
Decimals should be covariant
rok 235c90d
fix editable install collisions
rok 046fd11
review feedback
rok 0454fb9
move placeholder
rok e6bc69b
post rebase
rok a33f327
Apply suggestions from code review
rok ca9441c
Update python/pyarrow-stubs/pyarrow/_types.pyi
rok dbc4e40
update
rok d2cc882
lint
rok dce40ae
more covariant types
rok f278750
Apply suggestions from code review
rok ca7c54e
lint
rok c032596
Add type stubs for core data structures
rok 3df0567
temporary
rok c990f32
post rebase
rok cffdf24
Use plain classes and unions instead of generics in stubs
rok bb8681e
Add tests for the type annotations of public APIs
rok 376407d
Add placeholder stub for pyarrow._ipc
rok bc27123
Skip type checking of modules that aren't annotated yet
rok ddd9cb4
Fix stub annotations that don't match the runtime
rok 92250c5
Fix test_timedelta_overflow for sub-unit truncation
rok 9636e5c
Keep RecordBatchReader as Any until it gets annotated
rok 4abbecc
Use TypeGuard for is_fixed_size_binary()
rok 325ed1d
Test indexing of arrays, chunked arrays and tables
rok 26fa0e0
Add manual pre-commit hooks for linting and type checking the stubs
rok 7d7a9fd
Add _compute.pyi and reportMissingModuleSource
AlenkaF 12ea05f
Fix linter error
AlenkaF 25a866b
Stubs ruff-format
tadeja 94021e0
Move to __lib/ to fix Module errors for pa.array/table/scalar
tadeja ae3cad1
Move remaining ipc/types.pyi to __lib/
tadeja ca5eb2e
Review type.pyi
AlenkaF d18e949
Add typing_extensions to dependency-groups test
tadeja cf32c2e
Array staticmethod _import_from_c_capsule/device and test
tadeja 516e8a9
Add hash stubs to ExtensionType and KeyValueMetadata
AlenkaF 401c3c9
Simplified array inputs, ignore opt imports, fix DLPack signature and…
tadeja 368a6dc
Change Array.equals Any to Array and add tests
tadeja 91c3f0b
Review table.pyi
AlenkaF 1589c4c
Update ci/scripts/python_test_type_annotations.sh
AlenkaF f6892cd
Remove sys.version_info >= (3, 11) check
AlenkaF a1163e2
Assert struct array
AlenkaF e51b18e
Add test for list_
AlenkaF d297339
Fix CI ty check failure latest v0.99
tadeja 9dcdac6
More cases for test_table.py
tadeja 2f49f15
table.pyi: Allow array protocols for inputs. Specify append_column
tadeja 3301e77
Update python/pyarrow-stubs/tests/test_types.py
AlenkaF bf4a592
Add test for Field.metadata
AlenkaF a9cdd20
Remove sys.version_info >= (3, 11) on all files
AlenkaF 905aa33
Update python/pyarrow-stubs/pyarrow/__lib/error.pyi
AlenkaF a94a96e
More cases for test_array.py
tadeja 319b5c5
Review array.pyi
tadeja b642a3b
Apply AlenkaF's suggestions and add tests
tadeja 4b7dc90
Apply batched suggestions from code review
rok 41cc576
Move stubs to a flat pyarrow-stubs package and tests to pyarrow-stubs…
rok 82f4e53
Address review comments on type checking setup and stubs
rok 79a8f15
Re-export annotated pyarrow.lib names in pyarrow and use pa.* in tests
rok 046f15f
Run the ruff hooks for the type stubs on commit
rok 5caf254
Install the stub type checkers with uvx from requirements-stubs.txt
rok c093920
Mark MonthDayNano as final in the type stubs
rok cce4648
Remove the partial marker so that pyright checks the type stubs
rok 61c7c26
Remove the import ignores from the type stubs
rok 6ed844e
Add tensor.pyi updates and tensor test file
AlenkaF 6a4e614
Update stubtest allowlist
AlenkaF da3465a
Update _KeyValueMetadataInput
AlenkaF 86fe422
Update __contains__ and __getitem__ in KeyValueMetadata
AlenkaF 5ea9439
test_array.py unused type-ignore
tadeja 42927be
Update mask arg in array()
AlenkaF 724d480
Update Mask alias
AlenkaF 1335885
Update python/pyarrow-stubs/__lib/array.pyi
AlenkaF bb913a8
pyarrow/tests/test_*.py remove/update ignores
tadeja 17e3a08
Clean up the type stubs
rok e054197
Tidy the type checking setup
rok 4be907a
Remove the type checker accommodations from the PyArrow tests
rok 2d3f85c
Import scipy directly in the tensor type stub test
rok 6ae817e
Fix the type stub mismatches that stubtest skipped
rok ff91bcf
Make the optional import fallbacks in the PyArrow tests pass mypy
rok c7d1d97
Apply suggestion from @rok
rok d027724
Order imports and unify importing from pyarrow.lib
AlenkaF 9fc46cd
Fix the CI due to changes in imports
AlenkaF a8cc1ae
Update Indiced type alias
AlenkaF 96ba7b9
array.pyi: from_pandas type to accept str
tadeja 0e446fc
test_array.py: test from_pandas str and take Array indices
tadeja faf04a9
array.pyi: from_buffers accept Collection instead of Sequence
tadeja 4a899eb
Use TYPE_CHECKING for invalid usage in stub tests and fix review find…
rok 91ae655
Fix type checker errors in stub tests
rok d2462bd
array.pyi: nulls/repeat return Array instead of ArrayLike
tadeja 7f59731
Update test_import_from_c_capsule test
AlenkaF 7c1d685
Add two ree.from_buffers tests that fail currently but should pass (f…
AlenkaF dce7c2a
Ensure we accept objects from external projects in protocol that are …
AlenkaF 3675813
Review error.pyi
AlenkaF 8e26b41
array.pyi: array->Array, asarray overload for ChunkedArray
tadeja 08bccc8
array.pyi: widen list to Iterable in from_arrays, add mask TODO
tadeja ceaf8c8
Review and update device.pyi
AlenkaF 248bb04
Move types import into __init__.pyi so that pa.types.is_* do not reso…
AlenkaF 157090e
Apply suggestion from @jorisvandenbossche
tadeja fc6fb83
Add 2 overloads to _Tabular __getitem__
AlenkaF 4f0e1d1
Remove note from ty section in the docs
AlenkaF 247ec99
Update docs
AlenkaF eb7485d
Update .pre-commit-config.yaml
AlenkaF fbf55e6
Use Array next to Iterable for clarity
AlenkaF 5a026a0
Fix an issue found in RecordBatch.set_column
AlenkaF bc94a18
Accept plain Array for offsets, type codes, masks and run ends
rok 66d8661
Limit list from_arrays/from_pandas masks, align with runtime
tadeja abd654d
Move array() constructor cases to own test
tadeja 97a1f74
Two more overloads for Array/ChunkedArray.__getitem__
tadeja d3db98c
Minor mypy docs change
tadeja 72146cb
array.pyi: StructArrays fix. Accept iterables as runtime
tadeja 5a31e48
Update Type Checking docs section
AlenkaF 739800d
Add missing info about ty config
AlenkaF f808d64
Update python/pyarrow-stubs/__lib/table.pyi
AlenkaF 1616969
Update python/pyarrow-stubs/__lib/table.pyi
AlenkaF cf459f4
TEST: CI run with typeCheckingMode strict
tadeja 1531f93
Revert "TEST: CI run with typeCheckingMode strict"
tadeja 69b549c
Change np.ndarray to NDArray[Any]
tadeja 7fa510f
Fix mypy/pyright strict errors without narrowing types
tadeja f2a746d
Update docs/source/developers/python/development.rst
AlenkaF eac50b3
Fix Sphinx linter error
AlenkaF ccf5a11
Order imports in table.pyi
AlenkaF 3d17546
Update timestamp() stubs
AlenkaF c4b5754
Import Expression from stubs not from compute module
AlenkaF 374093b
Remove have_signal_refcycle from the stubs, see PR #55502
AlenkaF d085957
array.pyi: from_dlpack and FixedShapeTensorArray.from_tensor added
tadeja cd5e79c
array.pyi: arithmetic and bitwise operators added (since GH-48085)
tadeja 7e9de7d
array.pyi: Correct to_pandas categories to Collection[str]
tadeja 0f9b12e
array.pyi: replace ArrayLike (=Any) with _ArrayConstructorInput alias
tadeja File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -145,6 +145,70 @@ repos: | |
| ( | ||
| ?^python/pyarrow/vendored/| | ||
| ) | ||
| # Linting, formatting and type checking of the type stubs. The type checkers | ||
| # don't run on commit, only on request (the Lint CI job runs them too): | ||
| # pre-commit run --hook-stage manual python-stubs --all-files | ||
| # The type checkers run with uvx, in the environment of | ||
| # python/requirements-stubs.txt. They don't need PyArrow installed: the | ||
| # modules that aren't annotated yet, e.g. pyarrow.compute, are treated as Any | ||
| # (see python/pyproject.toml). | ||
| - repo: https://github.com/astral-sh/ruff-pre-commit | ||
| rev: v0.17.2 | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Are we keeping these up-to-date anywhere? @renovatebot or @dependabot as options, I guess @pre-commit-ci too (although not a fan of anything @pre-commit personally). |
||
| hooks: | ||
| - id: ruff-check | ||
| alias: python-stubs | ||
| name: Python Stubs Lint (ruff) | ||
| args: | ||
| - "--fix" | ||
| files: >- | ||
| ^python/pyarrow-stubs/ | ||
| types: | ||
| - pyi | ||
| - id: ruff-format | ||
| alias: python-stubs | ||
| name: Python Stubs Format (ruff) | ||
| files: >- | ||
| ^python/pyarrow-stubs/ | ||
| types: | ||
| - pyi | ||
| - repo: local | ||
| hooks: | ||
| - id: mypy | ||
| alias: python-stubs | ||
| name: Python Stubs Type Check (mypy) | ||
| stages: | ||
| - manual | ||
| language: python | ||
| entry: bash -c "cd python && exec uvx --with-requirements requirements-stubs.txt mypy" | ||
| pass_filenames: false | ||
| files: >- | ||
| ^python/(pyarrow-stubs(-tests)?/|(pyproject\.toml|requirements-stubs\.txt)$) | ||
| additional_dependencies: &uv | ||
| - uv==0.13.7 | ||
| - id: pyright | ||
| alias: python-stubs | ||
| name: Python Stubs Type Check (pyright) | ||
| stages: | ||
| - manual | ||
| language: python | ||
| entry: uvx --with-requirements python/requirements-stubs.txt pyright -p python/pyproject.toml | ||
| pass_filenames: false | ||
| files: >- | ||
| ^python/(pyarrow-stubs(-tests)?/|(pyproject\.toml|requirements-stubs\.txt)$) | ||
| additional_dependencies: *uv | ||
| - id: ty | ||
| alias: python-stubs | ||
| name: Python Stubs Type Check (ty) | ||
| stages: | ||
| - manual | ||
| language: python | ||
| # Unset the VIRTUAL_ENV of the hook, so that ty uses the environment | ||
| # of uvx. | ||
| entry: env -u VIRTUAL_ENV -u CONDA_PREFIX uvx --with-requirements python/requirements-stubs.txt ty check --project python | ||
| pass_filenames: false | ||
| files: >- | ||
| ^python/(pyarrow-stubs(-tests)?/|(pyproject\.toml|requirements-stubs\.txt)$) | ||
| additional_dependencies: *uv | ||
| - repo: https://github.com/MarcoGorelli/cython-lint | ||
| rev: v0.21.1 | ||
| hooks: | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would recommend prek over pre-commit. More feature rich, friendlier community and faster. Acts as a drop-in replacement.