can't build `python3-scipy`

[lifeisafractal]

First, thank you for taking up the torch on maintaining python3-scipy support. I'm working to pull parts of your layer into a new project I'm starting so I can build scipy. I'm currently pointing at main of all upstream layers (which I know might be a bad idea...). I plan on pointing to an official Yocto release after I get things stabilized out on the new project.

I brought in your python3-scipy recipe and related requirements (i.e. python3-beniget, etc.). When I go to build python3-scipy I get the following error:

ERROR: python3-scipy-native-1.14.1-r0 do_compile: Execution of '/build/tmp/work/x86_64-linux/python3-scipy-native/1.14.1/temp/run.do_compile.2091' failed with exit code 1
ERROR: Logfile of failure stored in: /build/tmp/work/x86_64-linux/python3-scipy-native/1.14.1/temp/log.do_compile.2091
Log data follows:
| DEBUG: Executing shell function do_compile
| * Getting build dependencies for wheel...
| 
| ERROR Missing dependencies:
| 	pythran<0.17.0,>=0.14.0
| 	beniget~=0.4.0 -> gast~=0.5.0
| WARNING: exit code 1 from a shell command.
ERROR: Task (virtual:native:/build/../work/layers/meta-aether-scipy/recipes-devtools/python/python3-scipy_1.14.1.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 840 tasks of which 839 didn't need to be rerun and 1 failed.
NOTE: No commit since BUILDHISTORY_COMMIT != '1'

Summary: 1 task failed:
  virtual:native:/build/../work/layers/meta-aether-scipy/recipes-devtools/python/python3-scipy_1.14.1.bb:do_compile
    log: /build/tmp/work/x86_64-linux/python3-scipy-native/1.14.1/temp/log.do_compile.2091
Summary: There was 1 ERROR message, returning a non-zero exit code.

When I look in the work dir for python3-scipy-native and look in recipe-sysroot-native, it did indeed populate pythran, beniget and gast. If I run a devshell and run python3 (double checking it's using the recipe-sysroot-native version), I can import all those packages. Any idea what is going on here and why the scipy package build process can not find those dependence? Below is a snipped of the run.do_compile. I suppose it's possible some of those paths are off?

python_pep517_do_compile() {
    nativepython3 -m build --no-isolation --wheel --outdir /build/tmp/work/x86_64-linux/python3-scipy-native/1.14.1/dist /build/tmp/work/x86_64-linux/python3-scipy-native/1.14.1/scipy-1.14.1 -Cbuilddir='/build/tmp/work/x86_64-linux/python3-scipy-native/1.14.1/build' -Csetup-args=--prefix -Csetup-args=/build/tmp/work/x86_64-linux/python3-scipy-native/1.14.1/recipe-sysroot-native/usr -Csetup-args=--bindir -Csetup-args=bin -Csetup-args=--sbindir -Csetup-args=sbin -Csetup-args=--datadir -Csetup-args=share -Csetup-args=--libdir -Csetup-args=lib -Csetup-args=--libexecdir -Csetup-args=libexec -Csetup-args=--includedir -Csetup-args=include -Csetup-args=--mandir -Csetup-args=share/man -Csetup-args=--infodir -Csetup-args=share/info -Csetup-args=--sysconfdir -Csetup-args=/build/tmp/work/x86_64-linux/python3-scipy-native/1.14.1/recipe-sysroot-native/etc -Csetup-args=--localstatedir -Csetup-args=/build/tmp/work/x86_64-linux/python3-scipy-native/1.14.1/recipe-sysroot-native/var -Csetup-args=--sharedstatedir -Csetup-args=/build/tmp/work/x86_64-linux/python3-scipy-native/1.14.1/recipe-sysroot-native/com -Csetup-args=--wrap-mode -Csetup-args=nodownload -Csetup-args=--native-file -Csetup-args=/build/tmp/work/x86_64-linux/python3-scipy-native/1.14.1/meson.native -Csetup-args=-Dblas=openblas -Csetup-args=-Dlapack=openblas -Csetup-args=-Duse-pythran=true
}

Thanks in advance for any help!

[zboszor]

Hi, thanks for the report.

The problem seems to be the indirect versioned dependencies.

scipy 1.14.1 wants beniget~=0.4 which means 0.4 <= version < 0.5, which in turn requires gast~=0.5, i.e. 0.5 <= version < 0.6)

Yocto master has upgraded gast to version 0.6.0, breaking the required versioned dependency chain.

I will have to leave styhead as is and update main to use a newer scipy or newer beniget, if there's one. I haven't looked yet.

[zboszor]

(Looking...) Or just patch beniget to need gast>=0.5 because there's no incompatibility between gast 0.5.x and 0.6.0.

This way styhead and main may stay at the same level.

[zboszor]

Weird.

scipy does not reference beniget directly, only indirectly via pythran. It builds under Yocto 5.1 Styhead with or without the python3-beniget-native build dependency. In both cases, python3-beniget-native is pulled in, either directly or indirectly.

pythran indeed has the versioned dependency beniget~=0.4.0 but beniget has gast>=0.5.0

Either somehow the ~= version check propagates downward in Yocto master (unlikely), or something provides a version of python3-pythran different from 0.16.1 provided by this layer. In the former case, it should be brought up to Yocto.

[lifeisafractal]

Thank you so much for the quick and thorough reply! I was misreading the beniget line and didn't understand it was calling out a mis-match on the transitive dependency of gast. I followed your advice of patching beniget to widen the gast requirement with success (patch below).

I'm leaving this issue open for you to decide if you want to take action in your repo to make it compatible with master of upstream Yocto.

I ran into another issue of the buildpaths QA step failing, but I'll open another issue for that one.

From 200040363de025679ac06ae5dc4b71aad332e0e1 Mon Sep 17 00:00:00 2001
From: Matt Campbell <matt@farixembedded.com>
Date: Fri, 6 Dec 2024 11:50:18 -0500
Subject: [PATCH] Widen version retirement for gast

Beniget (needed by SciPy) has a version conflict with he provided
version of gast in Yocto. This commit widens the requirement to avoid
that conflict as the minor version revisions are compatible with this
version of Beniget.

Upstream-Status: Pending
---
 requirements.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/requirements.txt b/requirements.txt
index 6c43616..3db5af2 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1 +1 @@
-gast ~= 0.5.0
+gast >=0.5.0,<1.0

[zboszor]

Exactly this is my problem. beniget version 0.4.2.post1 has this in requirements.txt:

me@localhost:~/python/beniget-0.4.2.post1$ cat requirements.txt 
gast >= 0.5.0

There should be no need to patch anything, as this version is in this layer in styhead / main.

The next version (see serge-sans-paille/beniget@3f3c950) already requires gast >= 0.5.4