Support a two-stage build for separate libimas-core / imas-core packaging by SimonPinches · Pull Request #62 · iterorganization/IMAS-Core

SimonPinches · 2026-05-28T09:21:17Z

Add an opt-in AL_USE_INSTALLED_CORE option that lets the Python wrapper be built against a pre-installed al-core via find_package, instead of compiling and bundling libal alongside the Cython extensions. This enables splitting the project into two downstream packages (e.g. the libimas-core and imas-core conda packages) without forking the build system, while keeping the existing single-shot wheel build as the default.

New option AL_USE_INSTALLED_CORE (default OFF). When ON, the al library is located with find_package(al-core CONFIG REQUIRED) and aliased as al so the rest of the tree links transparently.
The C/C++ library block is gated with AND NOT AL_USE_INSTALLED_CORE so it is skipped in stage-2 builds.
Install an al-coreTargets export plus a generated al-coreConfig.cmake / al-coreConfigVersion.cmake (additive — also benefits any other downstream C++ consumer).
python/CMakeLists.txt early-returns when AL_USE_INSTALLED_CORE is ON, installing only the Cython extensions and skipping the libal / runtime-deps bundling that is not valid for an alias of an imported target.
Expose AL_USE_INSTALLED_CORE through pyproject.toml so the option flows through scikit-build-core via the matching env var.

Default behaviour is unchanged: with AL_USE_INSTALLED_CORE=OFF the build still compiles libal, builds the Python bindings, and produces the same self-contained wheel as before.

Inspired by conda-forge/staged-recipes#31554, but keeping CMake as the single source of truth rather than introducing a parallel Meson build.

…ging Add an opt-in AL_USE_INSTALLED_CORE option that lets the Python wrapper be built against a pre-installed al-core via find_package, instead of compiling and bundling libal alongside the Cython extensions. This enables splitting the project into two downstream packages (e.g. the libimas-core and imas-core conda packages) without forking the build system, while keeping the existing single-shot wheel build as the default. - New option AL_USE_INSTALLED_CORE (default OFF). When ON, the al library is located with find_package(al-core CONFIG REQUIRED) and aliased as `al` so the rest of the tree links transparently. - The C/C++ library block is gated with `AND NOT AL_USE_INSTALLED_CORE` so it is skipped in stage-2 builds. - Install an al-coreTargets export plus a generated al-coreConfig.cmake / al-coreConfigVersion.cmake (additive — also benefits any other downstream C++ consumer). - python/CMakeLists.txt early-returns when AL_USE_INSTALLED_CORE is ON, installing only the Cython extensions and skipping the libal / runtime-deps bundling that is not valid for an alias of an imported target. - Expose AL_USE_INSTALLED_CORE through pyproject.toml so the option flows through scikit-build-core via the matching env var. Default behaviour is unchanged: with AL_USE_INSTALLED_CORE=OFF the build still compiles libal, builds the Python bindings, and produces the same self-contained wheel as before. Inspired by conda-forge/staged-recipes#31554, but keeping CMake as the single source of truth rather than introducing a parallel Meson build.

SimonPinches · 2026-05-28T11:25:54Z

Windows CI fails due to stale vcpkg cache + flaky gitlab.dkrz.de mirror for libaec. Re-runing to try and succeed.

Cache restored from key: windows-2022-win-deps-a

The cache contains 52 of 58 vcpkg packages but is missing 6, including libaec, hdf5, and a couple of boost components. So every Windows job must re-download those six.

Five of the six come from GitHub and resolve in seconds. Only libaec is hosted on gitlab.dkrz.de, which is unreliable from GitHub-hosted Windows runners: │

Python 3.11 (passing): downloaded k202009-libaec-v1.1.6.tar.gz in ~2.4 s
Python 3.10 (failing): Attempt 1 → 22 s → timeout
Attempt 2 → 22 s → timeout
error: Download timed out (gave up at 45 s)

So gitlab.dkrz.de works for some jobs and not others within the same run — it's a pure upstream-flakiness coin-flip.

Why the cache never heals itself:

Failed to save: Unable to reserve cache with key windows-2022-win-deps-a, another job may be creating this cache.
GHA cache keys are immutable — whoever first saved under windows-2022-win-deps-a (very likely an earlier run that failed after installing 52 packages but before installing libaec) locked it in. Every subsequent save attempt — including from a fully-successful 58-package job — is rejected. So the cache is permanently stuck at 52 packages, forcing every build to roll the libaec dice.

olivhoenen

LGTM

prasad-sawantdesai · 2026-06-01T09:51:25Z

Do I need to pass any parameter to CMake?

$ module list 2>&1 | grep IMAS
 20) iimkl/2023b                              42) IMAS-Core/5.6.0-intel-2023b
$ cmake -B build -D CMAKE_INSTALL_PREFIX="$(pwd)/test-install/" -D AL_USE_INSTALLED_CORE=ON -D CMAKE_C_COMPILER="${CC:-gcc}" -D CMAKE_CXX_COMPILER="${CXX:-g++}"
-- Found Git: /usr/bin/git (found version "2.39.3")
-- The C compiler identification is GNU 13.2.0
-- The CXX compiler identification is GNU 13.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /work/imas/opt/EasyBuild/software/GCCcore/13.2.0/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /work/imas/opt/EasyBuild/software/GCCcore/13.2.0/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Building a development version of the Access Layer core
CMake Error at CMakeLists.txt:141 (find_package):
  Could not find a package configuration file provided by "al-core" with any
  of the following names:

    al-coreConfig.cmake
    al-core-config.cmake

  Add the installation prefix of "al-core" to CMAKE_PREFIX_PATH or set
  "al-core_DIR" to a directory containing one of the above files.  If
  "al-core" provides a separate development package or SDK, be sure it has
  been installed.


-- Configuring incomplete, errors occurred!

SimonPinches · 2026-06-01T10:04:21Z

You don't need to pass any paramters to CMake. The problem is that the IMAS-Core/5.6.0-intel-2023b module was built before this PR, so its install tree only contains lib/libal.so and lib/pkgconfig/al-core.pc — there's no lib/cmake/al-core/al-coreConfig.cmake. The install(EXPORT) rules that emit that file are part of this PR and have never been part of a released module.

AL_USE_INSTALLED_CORE=ON therefore needs a stage-1 install produced from this branch:

# Stage 1: build & install the C/C++ library from this branch
cmake -S . -B build-core \
  -DCMAKE_INSTALL_PREFIX="$(pwd)/libimas-core-install" \
  -DAL_BACKEND_HDF5=ON -DAL_PYTHON_BINDINGS=OFF \
  -DAL_DOWNLOAD_DEPENDENCIES=OFF -DAL_DEVELOPMENT_LAYOUT=OFF \
  -DBoost_NO_BOOST_CMAKE=ON
cmake --build build-core --target install

# Stage 2: Python wrapper picks up al-coreConfig.cmake from the stage-1 prefix.
# Unload the released module so it doesn't shadow the local install.
module unload IMAS-Core
cmake -S . -B build-py \
  -DAL_USE_INSTALLED_CORE=ON -DAL_PYTHON_BINDINGS=ON \
  -DCMAKE_PREFIX_PATH="$(pwd)/libimas-core-install"
cmake --build build-py --target install

So AL_USE_INSTALLED_CORE is for the two-package conda use case where libimas-core has been packaged with the new CMake exports — it's not meant to consume a pre-PR module install. Once this PR is merged and a release is cut, future IMAS-Core modules will ship al-coreConfig.cmake and the workflow will be transparent.

Pushed fb23b74 — when find_package(al-core CONFIG) fails, CMake now emits a tailored message naming the missing file, suggesting CMAKE_PREFIX_PATH / al-core_DIR, and explicitly calling out that releases predating this PR will not satisfy the option.

prasad-sawantdesai · 2026-06-01T12:37:52Z

Thank you.. It provides below message now. Tested with IMAS-Core-5.7.0.
Should the message be changed to --> introduced in IMAS-Core 5.7.1 or later?

$ cmake -B build -D CMAKE_INSTALL_PREFIX="$(pwd)/test-install/" -D AL_USE_INSTALLED_CORE=ON -D CMAKE_C_COMPILER="${CC:-gcc}" -D CMAKE_CXX_COMPILER="${CXX:-g++}"
-- The C compiler identification is GNU 14.3.0
-- The CXX compiler identification is GNU 14.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /work/imas/opt/EasyBuild/software/GCCcore/14.3.0/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /work/imas/opt/EasyBuild/software/GCCcore/14.3.0/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Building a development version of the Access Layer core
-- Could NOT find al-core (missing: al-core_DIR)
CMake Error at CMakeLists.txt:143 (message):
  AL_USE_INSTALLED_CORE=ON requires an installed al-core that ships
  al-coreConfig.cmake (introduced in IMAS-Core 5.7.x).  find_package(al-core
  CONFIG) could not locate it.

  Point CMAKE_PREFIX_PATH (or al-core_DIR) at an install prefix that contains
  lib/cmake/al-core/al-coreConfig.cmake — typically the stage-1 install
  tree of this branch.  Releases built before this change (e.g.
  IMAS-Core/5.6.0) only ship al-core.pc and will NOT satisfy
  AL_USE_INSTALLED_CORE; either install stage 1 from source or wait for a
  release that includes the CMake package config.


-- Configuring incomplete, errors occurred!
$ module list 2>&1 | grep IMAS
 18) XZ/5.8.1-GCCcore-14.3.0                    44) IMAS-Core/5.7.0-intel-2025b
$

find_package(al-core CONFIG REQUIRED) bails with CMake's stock "could not find a package configuration file" message, which doesn't tell the user that AL_USE_INSTALLED_CORE specifically requires a stage-1 install from this branch — released modules (e.g. IMAS-Core/5.6.0) only ship al-core.pc, not al-coreConfig.cmake, so simply `module load`ing an older IMAS-Core does not satisfy the option. Replace REQUIRED with an explicit FATAL_ERROR that names the missing file, points at CMAKE_PREFIX_PATH / al-core_DIR, and explains why the previously-released modules will not work.

SimonPinches · 2026-06-01T13:09:51Z

Thank you.. It provides below message now. Tested with IMAS-Core-5.7.0. Should the message be changed to --> introduced in IMAS-Core 5.7.1 or later?

$ cmake -B build -D CMAKE_INSTALL_PREFIX="$(pwd)/test-install/" -D AL_USE_INSTALLED_CORE=ON -D CMAKE_C_COMPILER="${CC:-gcc}" -D CMAKE_CXX_COMPILER="${CXX:-g++}"
-- The C compiler identification is GNU 14.3.0
-- The CXX compiler identification is GNU 14.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /work/imas/opt/EasyBuild/software/GCCcore/14.3.0/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /work/imas/opt/EasyBuild/software/GCCcore/14.3.0/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Building a development version of the Access Layer core
-- Could NOT find al-core (missing: al-core_DIR)
CMake Error at CMakeLists.txt:143 (message):
  AL_USE_INSTALLED_CORE=ON requires an installed al-core that ships
  al-coreConfig.cmake (introduced in IMAS-Core 5.7.x).  find_package(al-core
  CONFIG) could not locate it.

  Point CMAKE_PREFIX_PATH (or al-core_DIR) at an install prefix that contains
  lib/cmake/al-core/al-coreConfig.cmake — typically the stage-1 install
  tree of this branch.  Releases built before this change (e.g.
  IMAS-Core/5.6.0) only ship al-core.pc and will NOT satisfy
  AL_USE_INSTALLED_CORE; either install stage 1 from source or wait for a
  release that includes the CMake package config.


-- Configuring incomplete, errors occurred!
$ module list 2>&1 | grep IMAS
 18) XZ/5.8.1-GCCcore-14.3.0                    44) IMAS-Core/5.7.0-intel-2025b
$

Done.

prasad-sawantdesai · 2026-06-01T13:15:27Z

In below strategy, al-core is built successfully..

cmake -S . -B build-core \
    -D CMAKE_INSTALL_PREFIX="$PWD/test-install" \
    -D AL_BACKEND_HDF5=ON \
    -D AL_BACKEND_MDSPLUS=ON \
    -D AL_BACKEND_UDA=ON \
    -D AL_BUILD_MDSPLUS_MODELS=ON \
    -D AL_DOWNLOAD_DEPENDENCIES=ON \
    -D AL_EXAMPLES=ON \
    -D AL_HLI_DOCS=ON \
    -D Boost_NO_BOOST_CMAKE=ON \
    -D AL_USE_INSTALLED_CORE=OFF \
    -D AL_PLUGINS=OFF \
    -D CMAKE_C_COMPILER="${CC:-gcc}" \
    -D CMAKE_CXX_COMPILER="${CXX:-g++}"

  cmake --build build-core --target install -j

Next build with Python Binding using al-core previously built

cmake -S . -B build-python-installed-core \
    -D CMAKE_PREFIX_PATH="$PWD/test-install" \
    -D CMAKE_INSTALL_PREFIX="$PWD/test-python-install" \
    -D AL_USE_INSTALLED_CORE=ON \
    -D AL_PYTHON_BINDINGS=ON \
    -D AL_BACKEND_HDF5=OFF \
    -D AL_BACKEND_MDSPLUS=OFF \
    -D AL_BACKEND_UDA=OFF \
    -D AL_BACKEND_UDAFAT=OFF \
    -D Boost_NO_BOOST_CMAKE=ON \
    -D CMAKE_C_COMPILER="${CC:-gcc}" \
    -D CMAKE_CXX_COMPILER="${CXX:-g++}"
Building a development version of the Access Layer core
CMake Error at skbuild.cmake:37 (add_custom_command):
  TARGET 'al' is IMPORTED and does not build here.
Call Stack (most recent call first):
  CMakeLists.txt:308 (include)


-- Configuring incomplete, errors occurred!

Do I need to execute al_env.sh before executing stage2?

When AL_USE_INSTALLED_CORE=ON and CMake is invoked directly (not via scikit-build-core / pip wheel), skbuild.cmake's non-SKBUILD path tried to attach `add_custom_command(TARGET al POST_BUILD ...)` to the `al` target. With AL_USE_INSTALLED_CORE that target is an ALIAS of an imported target, so CMake refused with: TARGET 'al' is IMPORTED and does not build here. The pip-wheel build does not need to be a POST_BUILD hook on `al` — it is fundamentally a wheel build that consumes `al`, not a transform of it. Split the path: when AL_USE_INSTALLED_CORE is ON, drive `pip wheel` from al-python-bindings's own COMMAND (with ALL so default `cmake --build` triggers it); otherwise keep the existing POST_BUILD behaviour that scikit-build-core relies on in the monolithic flow. Verified by reproducing the failing direct-cmake stage 2 reported in PR iterorganization#62 — configure, build, and install now all succeed and the produced wheel imports cleanly against a stage-1 libimas-core install.

SimonPinches · 2026-06-01T13:40:19Z

Good catch - that's a real bug. You're invoking cmake directly (not through pip/scikit-build-core), so skbuild.cmake takes its non-SKBUILD path and tries to attach add_custom_command(TARGET al POST_BUILD ...) to drive pip wheel. With AL_USE_INSTALLED_CORE=ON the al target is an ALIAS of an imported target, and CMake correctly refuses with "TARGET 'al' is IMPORTED and does not build here". I had only validated stage 2 via pip wheel previously, which takes the SKBUILD path and never reaches that line.

You don't need to source al_env.sh — that's a runtime env file, not a build prerequisite.

The fix is to skip the POST_BUILD attachment in the AL_USE_INSTALLED_CORE case and drive pip wheel from al-python-bindings's own command. Pushed 11f19eb. Re-tested locally with exactly your invocation (cmake -S . -B build-python-installed-core -DCMAKE_PREFIX_PATH=… -DAL_USE_INSTALLED_CORE=ON -DAL_PYTHON_BINDINGS=ON …) and configure/build/install all succeed; the produced wheel imports cleanly against the stage-1 libimas-core install.

A small note on flags: when you don't go through pip/scikit-build-core, scikit-build-core's pyproject.toml [tool.scikit-build.cmake.define] block is not consulted, so the AL_BACKEND_* / AL_USE_INSTALLED_CORE values it would inject from env vars don't apply. That's why your second invocation passes them explicitly — that's correct. Also AL_PYTHON_BINDINGS=ON causes skbuild.cmake to spawn pip wheel itself (build-isolated); AL_PYTHON_BINDINGS=no-build-isolation is preferable on systems where Cython/scikit-build-core come from environment modules, otherwise pip rebuilds them inside the isolated env.

prasad-sawantdesai · 2026-06-01T15:02:19Z

LGTM..

SimonPinches requested review from olivhoenen and prasad-sawantdesai as code owners May 28, 2026 09:21

olivhoenen approved these changes May 29, 2026

View reviewed changes

SimonPinches force-pushed the feature/CMake branch from 07b2822 to 0723109 Compare June 1, 2026 13:09

prasad-sawantdesai approved these changes Jun 1, 2026

View reviewed changes

olivhoenen merged commit 36efd07 into iterorganization:develop Jun 1, 2026
20 checks passed

SimonPinches mentioned this pull request Jun 2, 2026

Add Meson build system as alternative to CMake #10

Closed

9 tasks

olivhoenen mentioned this pull request Jun 3, 2026

IMAS-core is no longer an optional dependency iterorganization/IMAS-Python#128

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support a two-stage build for separate libimas-core / imas-core packaging#62

Support a two-stage build for separate libimas-core / imas-core packaging#62
olivhoenen merged 3 commits into
iterorganization:developfrom
SimonPinches:feature/CMake

SimonPinches commented May 28, 2026

Uh oh!

SimonPinches commented May 28, 2026

Uh oh!

olivhoenen left a comment

Uh oh!

prasad-sawantdesai commented Jun 1, 2026

Uh oh!

SimonPinches commented Jun 1, 2026

Uh oh!

prasad-sawantdesai commented Jun 1, 2026

Uh oh!

SimonPinches commented Jun 1, 2026

Uh oh!

prasad-sawantdesai commented Jun 1, 2026

Uh oh!

SimonPinches commented Jun 1, 2026

Uh oh!

prasad-sawantdesai commented Jun 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

SimonPinches commented May 28, 2026

Uh oh!

SimonPinches commented May 28, 2026

Uh oh!

olivhoenen left a comment

Choose a reason for hiding this comment

Uh oh!

prasad-sawantdesai commented Jun 1, 2026

Uh oh!

SimonPinches commented Jun 1, 2026

Uh oh!

prasad-sawantdesai commented Jun 1, 2026

Uh oh!

SimonPinches commented Jun 1, 2026

Uh oh!

prasad-sawantdesai commented Jun 1, 2026

Uh oh!

SimonPinches commented Jun 1, 2026

Uh oh!

prasad-sawantdesai commented Jun 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants