Skip to content

fix(bfcl): keep the bfcl extra importable by capping pyarrow below 26 - #2091

Merged
Yunnglin merged 1 commit into
modelscope:mainfrom
JerryChaox:fix/bfcl-pyarrow-numpy
Nov 16, 2026
Merged

Yunnglin merged 1 commit into
modelscope:mainfrom
JerryChaox:fix/bfcl-pyarrow-numpy

Conversation

@JerryChaox

Copy link
Copy Markdown
Contributor

Following the documented BFCL setup (pip install evalscope, then pip install bfcl-eval==2025.10.27.1) currently leaves an environment where evalscope cannot be imported:

ImportError: pyarrow requires NumPy 2.0 or newer, found 1.26.4

Cause

  • bfcl-eval pins numpy==1.26.4. Its latest release, 2026.3.23, still does.
  • pyarrow 26.0.0 (2026-10-09, GH-48473: [CI][Python] Require numpy 2.0 apache/arrow#50769) raises at import time when NumPy is older than 2.0, but pyarrow does not declare numpy as a dependency. pip therefore installs pyarrow 26 next to NumPy 1.26.4 without reporting a conflict, and import datasets (and with it import evalscope) fails.
  • This is likely why 59bb367 dropped bfcl-eval from the full CI job.

Fix

  • Cap pyarrow<26 in the bfcl extra, next to the bfcl-eval pin, with a comment explaining the reason. It can be removed once bfcl-eval stops pinning NumPy 1.x.
  • Change the install instructions to pip install 'evalscope[bfcl]', so that installing bfcl-eval always brings the cap with it. Updated: both BFCL guides (en/zh), and the v3/v4 adapter descriptions, with the generated meta and benchmark pages regenerated through benchmark-info --update / --generate-docs. The zh text is a manual one-line edit written through translate_benchmarks, because I don't have a DashScope key. No evaluation semantics change, so evaluation_version is unchanged.

Verification (Python 3.11, pip)

Environment Result
pip install evalscope (1.15.0), then pip install bfcl-eval==2025.10.27.1 (current docs) numpy 1.26.4 + pyarrow 26.0.1, import evalscope.run fails
clean pip install -e '.[bfcl]' on this branch numpy 1.26.4 + pyarrow 25.0.1 + datasets 4.8.4, pip check clean, BFCL-v4 load() works for simple_python and memory_kv
the broken environment above, then pip install -e '.[bfcl]' pyarrow downgraded to 25.0.1, import works
  • BFCL tests (test_bfcl_v3_summary, test_bfcl_v4, test_bfcl_v4_pipeline, test_bfcl_v4_state) and tests/api/test_benchmark_registry_contract.py in the fixed environment: 55 passed, 1 failed. The failure is test_bfcl_v4.py::test_patch_against_real_bfcl_eval_handler, which also fails on unmodified main when it runs after the pipeline/state tests in the same session, and passes on its own. It is order-dependent and unrelated to this change.
  • pytest tests/cli/test_all.py::TestRun::test_ci_lite: pass.
  • pre-commit on the changed files: pass.

Note: the BFCL tests in a clean environment also need soundfile, which bfcl-eval's qwen_agent import chain uses but does not declare. I left that out of this PR to keep the scope tight.

@JerryChaox
JerryChaox marked this pull request as ready for review November 15, 2026 04:25
@JerryChaox

Copy link
Copy Markdown
Contributor Author

@Yunnglin Ready for review: this keeps evalscope[bfcl] importable now that pyarrow 26 requires NumPy 2 while bfcl-eval pins numpy 1.26.4. CI is green.

@Yunnglin Yunnglin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Yunnglin
Yunnglin merged commit 12aa8ac into modelscope:main Nov 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants