Repository navigation
Conversation
pyarrow 26 raises an ImportError on import unless NumPy >= 2.0 is installed, but its wheel metadata does not declare a NumPy dependency, so pip does not prevent the combination. TensorFlow 2.15 requires numpy<2, so `pip install raydp[tensorflow]` next to an unpinned pyarrow resolves to pyarrow 26 + NumPy 1.26.4 and `import raydp` fails. Add `pyarrow<26` to the tensorflow and tensorflow-gpu extras so pip resolves a consistent set (pyarrow 25.x with NumPy 1.26.x). The PyPI release workflow installs the wheel without extras and pins numpy<1.24, so pin pyarrow there too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds
pyarrow<26to thetensorflowandtensorflow-gpuextras inpython/setup.py, and pinspyarrow<26in the PyPI release workflow.Why is this change needed?
CI is currently failing on every PR (seen on #492, a Scala-only change) with:
Root cause:
pip install pyarrow, which now resolves to pyarrow 26.0.0. That release raises anImportErrorunless NumPy >= 2.0 is installed, but its wheel metadata declares no NumPy dependency, so pip does not prevent the combination.pip install dist/raydp-*.whl[tensorflow]then pulls intensorflow>=2.15.1,<2.16, which requiresnumpy<2.0.0and resolves to NumPy 1.26.4.import raydpimports pyarrow, which fails.This also affects users:
pip install raydp[tensorflow]alongside pyarrow 26 gives a broken environment. Capping pyarrow in the extras fixes the cause rather than only CI. When the wheel is installed with[tensorflow], pip replaces an already-installed pyarrow 26 with 25.x.pypi_release.ymlinstalls the wheel without extras and pinsnumpy<1.24, so the extras change does not reach it; it gets its ownpyarrow<26pin.Master CI last passed on f186842 (2026-09-11), so the break comes from an unpinned dependency changing, not from a code change.
Related Issue
None filed.
Type of Change
How was this tested?
Reproduced the failure locally (Python 3.11, throwaway venvs):
import pyarrowChecked the fix with
pip install --dry-runof a stand-in package with raydp's dependency shape (numpy,pyarrow>=4.0.1, extratensorflow>=2.15.1,<2.16+pyarrow<26) into a venv that already had pyarrow 26.0.0 and NumPy 2.4.6. pip chose pyarrow 25.0.1, NumPy 1.26.4 and TensorFlow 2.15.1.I have not run the CI matrix or
pytest python/raydp/tests/against the built wheel; this PR's own CI run is the first full check.Note:
pypi_release.ymlchecks out the release tag, so this only helps releases tagged after it lands.Checklist