Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,12 @@ jobs:
- name: Run pre-commit
run: |
pre-commit run --all-files --color=always --show-diff-on-failure
- name: Run pre-commit for type stubs
env:
# The previous step already ran the ruff hooks.
SKIP: ruff-check,ruff-format
run: |
pre-commit run --hook-stage manual python-stubs --all-files --color=always --show-diff-on-failure
release:
name: Source Release and Merge Script on ${{ matrix.runs-on }}
Expand Down
64 changes: 64 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,70 @@ repos:
(
?^python/pyarrow/vendored/|
)
# Linting, formatting and type checking of the type stubs. The type checkers
# don't run on commit, only on request (the Lint CI job runs them too):
# pre-commit run --hook-stage manual python-stubs --all-files
# The type checkers run with uvx, in the environment of
# python/requirements-stubs.txt. They don't need PyArrow installed: the
# modules that aren't annotated yet, e.g. pyarrow.compute, are treated as Any
# (see python/pyproject.toml).
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.17.2
hooks:
- id: ruff-check
alias: python-stubs
name: Python Stubs Lint (ruff)
args:
- "--fix"
files: >-
^python/pyarrow-stubs/
types:
- pyi
- id: ruff-format
alias: python-stubs
name: Python Stubs Format (ruff)
files: >-
^python/pyarrow-stubs/
types:
- pyi
- repo: local
hooks:
- id: mypy
alias: python-stubs
name: Python Stubs Type Check (mypy)
stages:
- manual
language: python
entry: bash -c "cd python && exec uvx --with-requirements requirements-stubs.txt mypy"
pass_filenames: false
files: >-
^python/(pyarrow-stubs(-tests)?/|(pyproject\.toml|requirements-stubs\.txt)$)
additional_dependencies: &uv
- uv==0.13.7
- id: pyright
alias: python-stubs
name: Python Stubs Type Check (pyright)
stages:
- manual
language: python
entry: uvx --with-requirements python/requirements-stubs.txt pyright -p python/pyproject.toml
pass_filenames: false
files: >-
^python/(pyarrow-stubs(-tests)?/|(pyproject\.toml|requirements-stubs\.txt)$)
additional_dependencies: *uv
- id: ty
alias: python-stubs
name: Python Stubs Type Check (ty)
stages:
- manual
language: python
# Unset the VIRTUAL_ENV of the hook, so that ty uses the environment
# of uvx.
entry: env -u VIRTUAL_ENV -u CONDA_PREFIX uvx --with-requirements python/requirements-stubs.txt ty check --project python
pass_filenames: false
files: >-
^python/(pyarrow-stubs(-tests)?/|(pyproject\.toml|requirements-stubs\.txt)$)
additional_dependencies: *uv
- repo: https://github.com/MarcoGorelli/cython-lint
rev: v0.21.1
hooks:
Expand Down
16 changes: 11 additions & 5 deletions ci/scripts/python_test_type_annotations.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,20 @@ if [ -n "${ARROW_PYTHON_VENV:-}" ]; then
. "${ARROW_PYTHON_VENV}/bin/activate"
fi

# Install library stubs. Note some libraries contain their own type hints so they need to be installed.
pip install fsspec pandas-stubs scipy-stubs types-cffi types-psutil types-requests types-python-dateutil
cd "${pyarrow_dir}"

# Install type checkers
pip install mypy pyright ty
# Install type checkers and library stubs. Note some libraries contain their
# own type hints so they need to be installed.
pip install -r requirements-stubs.txt

# Run type checkers
cd "${pyarrow_dir}"
mypy
pyright --stats
ty check --verbose --output-format concise

# The type annotation tests are type checked above; also run them to verify
# the expected annotated types against the runtime types.
pytest -r s pyarrow-stubs-tests

# Check that the stubs match the signatures of the installed PyArrow.
python scripts/run_stubtest.py
4 changes: 2 additions & 2 deletions ci/scripts/python_wheel_validate_contents.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ def validate_wheel(path):
), "pyarrow/py.typed is present in the wheel."

source_root = Path(__file__).resolve().parents[2]
stubs_dir = source_root / "python" / "pyarrow-stubs" / "pyarrow"
stubs_dir = source_root / "python" / "pyarrow-stubs"
assert stubs_dir.exists(), f"Stub source directory not found: {stubs_dir}"

expected_stub_files = {
Expand All @@ -75,7 +75,7 @@ def validate_wheel(path):
}

assert not (wheel_stub_files == expected_stub_files), (
"Wheel .pyi files do not differ from python/pyarrow-stubs/pyarrow.\n"
"Wheel .pyi files do not differ from python/pyarrow-stubs.\n"
f"Missing in wheel: {sorted(expected_stub_files - wheel_stub_files)}\n"
f"Unexpected in wheel: {sorted(wheel_stub_files - expected_stub_files)}"
)
Expand Down
183 changes: 147 additions & 36 deletions docs/source/developers/python/development.rst
Original file line number Diff line number Diff line change
Expand Up @@ -105,69 +105,180 @@ Type Checking
=============

PyArrow provides type stubs (``*.pyi`` files) for static type checking. These
stubs are located in the ``pyarrow-stubs/`` directory and are automatically
included in the distributed wheel packages.
stubs are located in the ``pyarrow-stubs/`` directory, which is laid out as a
:pep:`561` stub-only package for ``pyarrow``, so type checkers and IDEs pick
it up from the ``arrow/python`` directory.

.. note::
The stubs aren't included in the distributed wheel packages until they are
complete (see `GH-49831 <https://github.com/apache/arrow/issues/49831>`_).

Run all the commands in this section from the ``arrow/python`` directory,
which holds the type checker configurations in ``pyproject.toml``:

.. code-block::

$ cd arrow/python

Maintaining Type Stubs
-----------------------

Type stubs for PyArrow are maintained in the ``pyarrow-stubs/``
directory. These stubs mirror the structure of the main ``pyarrow/`` package.

When adding or modifying public APIs:

1. **Update the corresponding .pyi stub file** in ``pyarrow-stubs/``
to reflect the new or changed function/class signatures.

2. **Run type checkers** to ensure the stubs are correct and complete.

The stub files will be automatically copied into the built wheel during the build
process and will be included when users install PyArrow, enabling type checking
in downstream projects and for users' IDEs (once
`issue 32609 <https://github.com/apache/arrow/issues/32609>`_ is sufficiently
resolved).

.. Note::

``py.typed`` marker file in the ``pyarrow/`` directory indicates to type
checkers that PyArrow supports type checking according to :pep:`561`.

Running Type Checkers
---------------------

We support multiple type checkers. Their configurations are in
``pyproject.toml``.
pre-commit
##########

**mypy**

To run mypy on the PyArrow codebase:
We support multiple type checkers. The easiest way to run all of them is with
``pre-commit``, which installs the type checkers and stub packages pinned in
``requirements-stubs.txt`` (with uv). This also lints and formats the stubs
with ruff.

.. code-block::

$ cd arrow/python
$ mypy
$ pre-commit run --hook-stage manual python-stubs --all-files

The mypy configuration is in the ``[tool.mypy]`` section of ``pyproject.toml``.
Type checker hooks run only on request (``--hook-stage manual``), while ruff
hooks run on every commit if the pre-commit git hook is set up locally
(with ``pre-commit install``).

**pyright**
Individual Type Checks
######################

To run pyright:
To run a type checker directly instead, install the pinned type checkers and
stub packages in your PyArrow development environment:

.. code-block::

$ cd arrow/python
$ pyright
$ pip install -r requirements-stubs.txt

The pyright configuration is in the ``[tool.pyright]`` section of ``pyproject.toml``.
Checking the PyArrow tests also needs their dependencies from
``requirements-test.txt``.

**ty**
.. tab-set::

To run ty (note: currently only partially configured):
.. tab-item:: mypy

.. code-block::
To run mypy on the PyArrow codebase:

$ cd arrow/python
$ ty check
.. code-block::

Maintaining Type Stubs
-----------------------
$ mypy

Type stubs for PyArrow are maintained in the ``pyarrow-stubs/``
directory. These stubs mirror the structure of the main ``pyarrow/`` package.
The mypy configuration is in the ``[tool.mypy]`` section of ``pyproject.toml``.
With an editable install, mypy finds ``pyarrow-stubs`` automatically. With a
non-editable install (``pip install .`` or a wheel built with
``python -m build``), the stubs aren't installed yet, so pass ``pyarrow-stubs``
along with the files you check:

When adding or modifying public APIs:
.. code-block::

1. **Update the corresponding ``.pyi`` stub file** in ``pyarrow-stubs/``
to reflect the new or changed function/class signatures.
$ mypy pyarrow-stubs pyarrow/tests/test_array.py

2. **Include type annotations** where possible. For Cython modules or
dynamically generated APIs such as compute kernels add the corresponding
stub in ``pyarrow-stubs/``.
.. tab-item:: pyright

3. **Run type checkers** to ensure the stubs are correct and complete.
To run pyright on the PyArrow codebase:

The stub files are automatically copied into the built wheel during the build
process and will be included when users install PyArrow, enabling type checking
in downstream projects and for users' IDEs.
.. code-block::

$ pyright

The pyright configuration is in the ``[tool.pyright]`` section of ``pyproject.toml``.

.. tab-item:: ty

To run ty on the PyArrow codebase:

.. code-block::

$ ty check

The ty configuration is in the ``[tool.ty.*]`` sections of ``pyproject.toml``.

.. tab-item:: stubtest

mypy's stubtest compares the stubs with the signatures of the installed
PyArrow. Run it with:

.. code-block::

$ python scripts/run_stubtest.py

Stubs that aren't written yet are skipped. The known mismatches are listed in
``pyarrow-stubs-tests/stubtest_allowlist.txt``; remove the entries of the
mismatches you fix, as stubtest fails on unused entries.

.. admonition:: Note on type checking in the CI
:class: note

The Lint CI job runs the ``python-stubs`` pre-commit hooks above. A separate
CI job runs the type checkers, the type stub tests (see `Testing Type Stubs`_)
and stubtest against the PyArrow it builds, with ``ci/scripts/python_test_type_annotations.sh``.

Type Checking in an IDE
#######################

Editors also run the type checkers from the folder they open. If you open the
whole ``arrow`` repository in VS Code, point the type checker at ``arrow/python``
instead.

Testing Type Stubs
------------------

The tests in ``pyarrow-stubs-tests/`` pin down the annotated types of public
APIs. Each check pairs :func:`typing.assert_type`, which every type checker
verifies statically, with a runtime ``isinstance`` check:

.. code-block:: python

from typing import TYPE_CHECKING

import pyarrow as pa

from _check import assert_type, check

def test_list_type() -> None:
lst = pa.list_(pa.int32())
check(assert_type(lst, pa.ListType), pa.ListType)
check(assert_type(lst.value_type, pa.DataType), pa.DataType)

if TYPE_CHECKING:
pa.list_(1) # type: ignore[call-overload]

The type checkers check these files as part of the regular runs above, and
pytest runs them against the installed PyArrow to make sure the stubs agree
with what PyArrow actually returns:

.. code-block::

$ pytest pyarrow-stubs-tests

Note: ``py.typed`` marker file in the ``pyarrow/`` directory indicates to type
checkers that PyArrow supports type checking according to :pep:`561`.
Code under ``if TYPE_CHECKING:`` is never executed. It holds the usage that
should fail type checking, which is asserted with a
``# type: ignore[<mypy code>]`` comment: unused ignores are reported as errors
in these tests, so the test fails once the usage is no longer rejected. ty
doesn't honor mypy error codes, so it doesn't check invalid usage.

Doctest
=======
Expand Down
4 changes: 2 additions & 2 deletions python/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1079,15 +1079,15 @@ endif()
#
# TODO(GH-32609): Reintroduce type stubs into PyArrow wheels.
# GH-49831: Temporarily do not install pyarrow-stubs into wheels.
# Stubs live in pyarrow-stubs/pyarrow/ during development but are installed
# Stubs live in pyarrow-stubs/ during development but are installed
# alongside the package so type checkers can find them (PEP 561).
# The stubs are currently incomplete, and some type checkers consume .pyi files
# even without the py.typed marker. Re-enable this when the stubs are complete.
if(PYARROW_REQUIRE_STUB_DOCSTRINGS)
message(FATAL_ERROR "PYARROW_REQUIRE_STUB_DOCSTRINGS cannot be used while "
"pyarrow-stubs are omitted from wheels (GH-49831).")
endif()
# set(PYARROW_STUBS_SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/pyarrow-stubs/pyarrow")
# set(PYARROW_STUBS_SOURCE_DIR "${CMAKE_CURRENT_SOURCE_DIR}/pyarrow-stubs")
# if(EXISTS "${PYARROW_STUBS_SOURCE_DIR}")
# install(DIRECTORY "${PYARROW_STUBS_SOURCE_DIR}/"
# DESTINATION "."
Expand Down
Loading
Loading