From f0dc9c19864981d3048064ef215a5184281dc61a Mon Sep 17 00:00:00 2001 From: Ivana Date: Wed, 3 Jun 2026 09:42:16 +0000 Subject: [PATCH] docs: document CUDA device-LTO elfLink failure on apt nvidia-cuda-toolkit Add a "Build troubleshooting" section to the advanced install guide covering "nvlink fatal : elfLink linker library load error" during a CUDA-enabled build. On non-Apple builds Cytnx enables CMAKE_INTERPROCEDURAL_OPTIMIZATION, which together with CUDA separable compilation turns on CUDA device LTO (nvcc -dlto). The device link step loads NVVM. The Debian/Ubuntu nvidia-cuda-toolkit apt package installs libnvvm.so under /usr/lib/x86_64-linux-gnu/ but not under the toolkit's lib64 directory, where nvcc points nvlink via -nvvmpath=/usr/lib/nvidia-cuda-toolkit; nvlink then fails to open /usr/lib/nvidia-cuda-toolkit/lib64/libnvvm.so. The empty glibc 2.34+ stub archives (libpthread.a / librt.a / libdl.a) are unrelated and are tolerated by nvlink, so they are explicitly called out as not being the cause. Document the recommended fix (install a complete CUDA toolkit from conda or NVIDIA so libnvvm.so sits in nvvm/lib64) and a symlink workaround for the distribution package. Co-authored-by: Claude --- docs/source/adv_install.rst | 54 +++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/docs/source/adv_install.rst b/docs/source/adv_install.rst index ecfe0ed2a..2d07552c3 100644 --- a/docs/source/adv_install.rst +++ b/docs/source/adv_install.rst @@ -342,6 +342,60 @@ In the case that Cytnx is installed locally from binary build, not from anaconda CYTNX_ROOT is the path where Cytnx is installed from binary build. +Build troubleshooting +************************************* + +CUDA device link fails with ``elfLink linker library load error`` +------------------------------------------------------------------------------------- + +**Symptom.** A CUDA-enabled build (``-DUSE_CUDA=ON``) configures successfully, +but the CUDA *device link* step aborts with:: + + nvlink fatal : elfLink linker library load error + +On non-Apple builds Cytnx turns on interprocedural optimization +(``CMAKE_INTERPROCEDURAL_OPTIMIZATION``) which, together with CUDA separable +compilation, enables CUDA *device* link-time optimization (``nvcc -dlto``). The +device link step then asks ``nvlink`` to load the NVVM library, and the error +above means it could not. + +**Cause.** This is *not* caused by the empty ``libpthread.a`` / ``librt.a`` / +``libdl.a`` stub archives that glibc 2.34+ ships -- those are tolerated by +``nvlink``. It is a layout problem specific to the Debian/Ubuntu +``nvidia-cuda-toolkit`` apt package. That package installs ``libnvvm.so`` into +the multiarch directory ``/usr/lib/x86_64-linux-gnu/`` but does not place it +under the toolkit's ``lib64`` directory, which is where ``nvcc`` tells +``nvlink`` to look. ``nvcc`` passes ``-nvvmpath=/usr/lib/nvidia-cuda-toolkit``, +so ``nvlink`` tries to open ``/usr/lib/nvidia-cuda-toolkit/lib64/libnvvm.so`` +and finds nothing. Regular (non-LTO) device linking does not load NVVM, which is +why the failure appears only once device LTO is enabled. + +**Fix (recommended): use a complete CUDA toolkit.** Install CUDA from conda or +NVIDIA's official installer instead of the distribution's +``nvidia-cuda-toolkit`` package, and make sure its ``nvcc`` is first on +``PATH``: + +.. code-block:: shell + + $conda install -c nvidia cuda + +A toolkit laid out this way keeps ``libnvvm.so`` under ``nvvm/lib64`` where +``nvlink`` expects it, so device LTO works with no further action. This is also +the layout the CUDA build presets assume. + +**Workaround: keep the apt package and add the missing path.** If you must build +against the distribution package, create the directory ``nvlink`` searches and +symlink the packaged ``libnvvm.so`` into it: + +.. code-block:: shell + + $sudo mkdir -p /usr/lib/nvidia-cuda-toolkit/lib64 + $sudo ln -s /usr/lib/x86_64-linux-gnu/libnvvm.so /usr/lib/nvidia-cuda-toolkit/lib64/libnvvm.so + +Then re-run the build. On non-x86_64 hosts the multiarch directory differs; +locate the real library first with ``find /usr -name 'libnvvm.so*'``. + + Check Cytnx version ************************************* The current version of the library can be printed by: