fix(#498): package all shared libs in sm60 OCI image (ldd verify, LD_LIBRARY_PATH)

[ddvnguyen]

Fixes ddvnguyen/hydra_vortex#498 — P100 sm60 OCI image shipped executable without its shared libs.

Problem (original)

sm60 build is BUILD_SHARED_LIBS=ON (libllama.so.0, libggml-*.so.0, libmtmd.so.0, libllama-server-impl.so for llama-server) but the OCI image built via hydra-build.Dockerfile copied only the binary:

FROM scratch
COPY llama-engine /llama-server
ENTRYPOINT ["/llama-server"]

or later COPY bin/ /llama/ without -a and without ldd verify. Deploy shipped new binary against stale .so set in /opt/software/llama-cpp-hydra-sm60/hydra-sm60/lib (empty, libs were in bin/) plus build-host RUNPATH /mnt/WorkDisk/.../build_sm60/bin that doesn't exist on VM. Result: undefined symbol: llama_model_get_quant_label (fork PR #63) and 1120-restart loop, seen in A/B baseline logs (libllama-server-impl.so missing).

Fix (original)

• .github/workflows/hydra-build.Dockerfile: COPY bin/ /llama/ now reliably includes *.so* via staging; add ARG BINARY after FROM, RUN ldd verify (fail on hydra => not found, ignore expected libcuda/libibverbs), ls -lh /llama/*.so*, ENV LD_LIBRARY_PATH=/llama, and symlink fallback for both binaries (ln -sf /llama/${BINARY} /llama/llama-engine) so fixed ENTRYPOINT ["/llama/llama-engine"] works for either image.
• .github/workflows/scripts/build-combo.sh: cp -a (preserve symlinks), ls staging, host ldd check, warn if libllama-server-impl.so missing for llama-server builds.

RUNPATH is already $ORIGIN (-DCMAKE_BUILD_RPATH='$ORIGIN'), so /llama is the search dir; LD_LIBRARY_PATH is defensive fallback.

Follow-up: glibc mismatch (lead zero-trust finding on fix498-v3)

Root cause: Host is Ubuntu 26.04 (glibc 2.43) but runtime base is nvidia/cuda:12.9.2-runtime-ubuntu24.04 (glibc 2.39). Host-built sm60 binaries require GLIBC_2.43 (5× version 'GLIBC_2.43' not found for libllama/libggml-base/libggml-cpu/libggml-cuda/libmtmd in ldd /llama/llama-engine inside the image). The image will crash at engine start on the P100 VM. The Dockerfile's original ldd gate only grepped => not found, so GLIBC version errors were missed.

Fix (smallest correct diff, consistent with hydra-build pipeline):

• Dockerfile: Harden ldd gate to fail on BOTH => not found AND version .*GLIBC.*not found (exclude only host-provided libcuda/libibverbs/libcudart/libcublas). Add libibverbs1 to apt-get.
• build-combo.sh: Build sm60 inside nvidia/cuda:12.9.2-devel-ubuntu24.04 (container's own /usr/local/cuda, cmake, g++-14) to match runtime glibc 2.39. The container installs cmake ninja-build ccache g++-14 libssl-dev libibverbs-dev, creates stub symlink for libcuda.so.1, and runs cmake -B build_hydra_sm60_llama-engine -G Ninja -DCMAKE_CUDA_ARCHITECTURES=60 ... -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc ... and cmake --build. sm86-sm120 stays host-built (same latent glibc issue, hardened gate now protects).

Alternative considered: Multi-stage Dockerfile (devel stage compiles, runtime stage ships) — equivalent correctness but larger churn to the pipeline; containerized build-combo.sh keeps the existing cmake → staging → podman build flow with less diff.

Verify (host build, no P100 runtime change per ggml-org#733 W1)

• Toolchain: Host DCUDAToolkit_ROOT=/opt/software/cuda/12.9 (nvcc 12.9.86) for original, but sm60 now builds inside nvidia/cuda:12.9.2-devel-ubuntu24.04 (glibc 2.39, g++-14, cmake 3.28, /usr/local/cuda/bin/nvcc).
• Build (containerized sm60): podman run -v $PWD:/work -w /work nvidia/cuda:12.9.2-devel-ubuntu24.04 bash -c 'apt-get update && apt-get install -y cmake ninja-build ccache g++-14 libssl-dev libibverbs-dev && ln -sf /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/cuda/lib64/stubs/libcuda.so.1 && cmake -B build_hydra_sm60_llama-engine -G Ninja -DCMAKE_CUDA_ARCHITECTURES=60 -DGGML_CUDA=ON ... -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc -DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/cuda/lib64/stubs" ... && cmake --build build_hydra_sm60_llama-engine --target llama-engine' — success, readelf -d shows RUNPATH [$ORIGIN], strings shows max GLIBC_2.38 (not 2.43), ldd hydra libs resolve via $ORIGIN.
• Image (containerized): podman build --build-arg CUDA_VERSION=12.9.2 --build-arg BINARY=llama-engine -t localhost/llama-sm60-test:fix498-glibc3 -f .github/workflows/hydra-build.Dockerfile staging_sm60_verify — 3.95GB, COPY bin/ /llama/ includes all .so*, RUN ldd now passes for hydra libs and GLIBC.
• ldd in container (runtime, glibc 2.39):

  $ podman run --rm --pull=never --entrypoint /bin/bash localhost/llama-sm60-test:fix498-glibc3 -c 'ldd /llama/llama-engine | head -n 30'
  libmtmd.so.0 => /llama/libmtmd.so.0
  libllama-common.so.0 => /llama/libllama-common.so.0
  libllama.so.0 => /llama/libllama.so.0
  libggml.so.0 => /llama/libggml.so.0
  libggml-rpc.so.0 => /llama/libggml-rpc.so.0
  libggml-base.so.0 => /llama/libggml-base.so.0
  libggml-cpu.so.0 => /llama/libggml-cpu.so.0
  libggml-cuda.so.0 => /llama/libggml-cuda.so.0
  libibverbs.so.1 => /lib/x86_64-linux-gnu/libibverbs.so.1
  libcudart.so.12 => /usr/local/cuda/lib64/libcudart.so.12
  libcublas.so.12 => /usr/local/cuda/lib64/libcublas.so.12
  libcuda.so.1 => not found (host driver, expected)
  $ ldd /llama/llama-engine | grep -E "=> not found|version .*GLIBC.*not found" | grep -vE "libcuda|libcudart|libcublas|libibverbs" | wc -l
  0
  $ strings /llama/llama-engine | grep -E "GLIBC_2\." | sort -u | tail
  GLIBC_2.38  # was 2.43 before, now 2.39-compatible
  

  Zero hydra not found and zero GLIBC version errors (only libcuda expected host mount) — fix verified. ls -lh /llama/*.so* shows all libs.
• Smoke (proves dynamic linker can start):

  $ podman run --rm --pull=never --device nvidia.com/gpu=all localhost/llama-sm60-test:fix498-glibc3 --version
  version: 0 (unknown) [shared]
  built with GNU 13.3.0 for Linux x86_64
  exit:0
  $ podman run --rm --pull=never --device nvidia.com/gpu=all localhost/llama-sm60-test:fix498-glibc3 --help
  -h, --help ...
  exit:0
  

  Both exit 0 — dynamic linker starts the binary (host-built 2.43 image previously failed with 5× GLIBC_2.43 not found).

Note on sm86/sm120: Host-built sm86-sm120 binaries (RTX 5060 Ti + 3060, CUDA 13.2) carry the same latent glibc 2.43 vs 2.39 mismatch; the hardened grep -E "=> not found|version .*GLIBC.*not found" gate now protects them and will fail the image build if they drift.

Risk

Additive Dockerfile + build script only; no P100 VM change, no prod deploy. Image size unchanged (~3.95GB). No force-push.

Refs ddvnguyen/hydra_vortex#498, ggml-org#733 T5b