Skip to content

fix(cargo-oxide): make doctor reachable without a CUDA toolkit or driver - #174

Merged
nihalpasham merged 6 commits into
mainfrom
fix/doctor-without-cuda-toolkit
Jun 12, 2026
Merged

nihalpasham merged 6 commits into
mainfrom
fix/doctor-without-cuda-toolkit

Conversation

@nihalpasham

Copy link
Copy Markdown
Collaborator

Fixes #87. PR #172 covered the cuda-bindings half (a clean build-script error instead of a bindgen panic when cuda.h is missing); this PR covers the reachability half: on a machine with no CUDA toolkit and no driver, cargo-oxide now builds, loads, and cargo oxide doctor prints an actionable diagnosis instead of dying in a dependency's build script.

Before: cargo oxide doctor -> cargo builds cuda-bindings -> hard error on missing cuda.h
After:  cargo oxide doctor -> runs, reports each missing piece and how to fix it

What changed

  • cargo-oxide drops its cuda-host / cuda-core dependencies. Its graph now contains no cuda-bindings, bindgen, or clang-sys, and the binary has no libcuda.so.1 in DT_NEEDED, so it builds and runs on a bare machine. libnvvm-sys / nvjitlink-sys stay: they are pure-Rust runtime-dlopen crates with zero build-time CUDA cost.
  • Run-mode GPU arch auto-detect now shells out to nvidia-smi --query-gpu=compute_cap --format=csv,noheader (the derivation smoketest.sh already uses) instead of the CUDA driver API. Output is trusted only on exit 0 plus a <digits>.<digits> first line, because nvidia-smi prints failure banners to stdout. Precedence is unchanged: --arch and CUDA_OXIDE_TARGET still override, the result stays the advisory CUDA_OXIDE_DEVICE_ARCH hint, and sm_XYa is still emitted for cc >= 9.0.
  • libdevice discovery is consolidated into libnvvm-sys as find_libdevice() (libdevice ships in the toolkit's nvvm/ component next to libnvvm.so), removing a duplicated cuda_roots_from_env. cuda_host::ltoir::find_libdevice remains as a thin wrapper with identical signature and error text.
  • doctor is build-free: a new side-effect-free resolve_doctor_context() (with backend::backend_so_candidate()) replaces resolve_context(), which used to build the backend unconditionally, or even git-clone in standalone mode, before a single check ran.
  • doctor gains two checks:
    • CUDA headers (cuda.h), fatal: mirrors cuda-bindings/build.rs discovery (CUDA_TOOLKIT_PATH > CUDA_HOME > /usr/local/cuda; include/ then targets/<x86_64-linux|sbsa-linux>/include per host arch). On failure it lists every probed path and the env vars to set. The probe carries a lockstep comment: build.rs logic is not importable and cuda-bindings must stay out of the graph, so the two probes are kept in sync by hand. Both helpers are pure and unit-tested.
    • NVIDIA driver / GPU, informational: reports the GPU name and compute capability via nvidia-smi, with a /proc/driver/nvidia/version fallback that distinguishes "driver loaded but nvidia-smi broken" from "no driver at all". Non-fatal because only cargo oxide run needs a driver; GPU-less CI and cross-compiling are supported workflows.
  • The backend .so check downgrades to informational ("not built yet, run cargo oxide setup"): run/build/pipeline build it on demand, so a fresh healthy clone stays green.
  • CI: the cargo-oxide unit-test job drops the toolkit install and stub shadow, and a new step runs the built binary's doctor on that toolkit-less runner and asserts the cuda.h diagnosis is reached (an end-to-end regression gate for the exact issue cargo oxide doctor cannot diagnose missing CUDA headers because cuda-bindings panics first #87 environment). examples-compile drops the libcuda stub shadow that existed only for cargo-oxide, in its own commit so it reverts cleanly.

For users with a toolkit and GPU

  • run/build/pipeline/debug flows are untouched (they always shelled out to cargo; GPU work happens in the example's own process, which still links cuda-core as before).
  • Identical arch values: nvidia-smi's compute_cap is the same cc pair as cuDeviceGetAttribute.
  • doctor: same verdicts as before, plus the headers and driver lines.

Known tradeoffs (documented in code)

  • nvidia-smi lists GPUs in PCI order while CUDA defaults to fastest-first, so on heterogeneous multi-GPU boxes the hint may describe a different GPU than CUDA device 0. Contained: the hint is advisory (the backend ignores incompatible hints) and --arch / CUDA_OXIDE_TARGET are hard overrides.
  • Driver present but nvidia-smi absent or broken: the hint silently becomes None and the backend default applies; doctor's /proc fallback at least diagnoses that state.
  • doctor no longer auto-builds the backend; anyone who relied on that side effect now gets an informational "run cargo oxide setup" line.

Verification

  • cargo tree -p cargo-oxide -e normal,build: no cuda-bindings / cuda-core / cuda-host / bindgen / clang-sys (all five present before).
  • readelf -d cargo-oxide: libcuda.so.1 gone from DT_NEEDED (present before; it is why both CI workflows carried the stub shadow).
  • CUDA_TOOLKIT_PATH=/nonexistent CUDA_HOME=/nonexistent cargo build -p cargo-oxide succeeds: no build script in the graph consults those vars anymore.
  • doctor on a toolkit machine passes every pre-existing check and shows the two new lines; with CUDA_TOOLKIT_PATH=/nonexistent it prints the probed cuda.h paths and exits 1.
  • cargo oxide build vecadd completes clean with a fresh vecadd.ptx and no swallowed-error signatures.
  • fmt / clippy -D warnings / doc -D warnings / tests green on cargo-oxide, libnvvm-sys, cuda-host; the three pre-existing detect_run_target_arch tests pass unmodified.

GPU-box confirmation requested: cargo oxide doctor should print the GPU name and compute cap on the driver line, and cargo oxide run vecadd should report a "Detected GPU arch" matching nvidia-smi --query-gpu=compute_cap and pass.

Suggested labels: QoL-improvement, cargo-oxide, host-apis, build-related.

GPU-box confirmation (done)

Verified on a 2x RTX 5090 box: doctor prints "NVIDIA driver / GPU... ✓ NVIDIA GeForce RTX 5090 (compute capability 12.0)" and exits green; cargo oxide run vecadd reports "Detected GPU arch: sm_120a (via nvidia-smi)" and passes with all 1024 elements correct. readelf -d on the built binary confirms libcuda.so.1 is gone from DT_NEEDED.

libdevice ships in the toolkit's nvvm/ component next to libnvvm.so and
is consumed together with libNVVM in the LTOIR pipeline, so the probe
belongs in libnvvm-sys. cuda_host::ltoir::find_libdevice becomes a thin
delegating wrapper with unchanged error type and message text, and
ltoir.rs drops its now-unused cuda_roots/cuda_roots_from_env copy (the
same helper already lives in libnvvm-sys). This also lets cargo-oxide's
doctor probe libdevice without pulling the cuda-host -> cuda-core ->
cuda-bindings chain.
…without the CUDA toolkit

cargo-oxide had exactly two real CUDA-coupled call sites: the doctor
libdevice probe (cuda_host::ltoir::find_libdevice) and the run-target
auto-detect (cuda_core::CudaContext compute_capability). Both pulled
the cuda-host -> cuda-core -> cuda-bindings chain, whose build script
hard-fails without cuda.h and links libcuda, so on a bare machine
'cargo oxide doctor' died in a build script before main() (issue #87)
and a prebuilt binary could not even load without a libcuda.so.1 stub.

- doctor's libdevice check now calls libnvvm_sys::find_libdevice()
  (same probe, no cuda-bindings in the graph)
- detect_run_target_arch now derives the compute capability from
  'nvidia-smi --query-gpu=compute_cap --format=csv,noheader', the same
  derivation scripts/smoketest.sh already uses; output is trusted only
  on exit 0 AND a <digits>.<digits> shaped first line, since nvidia-smi
  prints failure banners to stdout
- precedence is unchanged: --arch and CUDA_OXIDE_TARGET still override,
  the detected arch stays an advisory CUDA_OXIDE_DEVICE_ARCH hint, and
  format_sm_arch still emits sm_XYa for cc >= 9.0

Known tradeoff, documented on the helper: nvidia-smi lists GPUs in PCI
order while CUDA defaults to fastest-first, so on heterogeneous
multi-GPU boxes the hint may describe a different GPU than CUDA device
0. Safe because the hint is advisory and hard overrides remain.

cargo tree -p cargo-oxide -e normal,build now contains no
cuda-bindings, cuda-core, cuda-host, bindgen, or clang-sys, and the
binary's DT_NEEDED no longer carries libcuda.so.1.
… driver

Fixes #87. With the cuda-bindings edge gone, doctor finally runs on a
bare machine, so make it actually diagnose one:

- new fatal 'CUDA headers (cuda.h)' check that mirrors the discovery in
  cuda-bindings/build.rs (CUDA_TOOLKIT_PATH > CUDA_HOME >
  /usr/local/cuda; include/ then targets/<x86_64-linux|sbsa-linux>/
  include per host arch). On failure it lists every probed path and the
  env vars to set. The probe is kept in lockstep by hand (build.rs
  logic is not a library and cuda-bindings must stay out of
  cargo-oxide's graph); both helpers are pure and unit-tested
- new informational 'NVIDIA driver / GPU' check via nvidia-smi
  (reports name + compute capability), with a /proc/driver/nvidia/
  version fallback that distinguishes 'driver loaded but nvidia-smi
  broken' from 'no driver at all'. Non-fatal: only 'cargo oxide run'
  needs a driver, and GPU-less CI is a supported workflow
- doctor is now build-free: the Doctor arm resolves a side-effect-free
  context (new resolve_doctor_context + backend::backend_so_candidate)
  instead of resolve_context, which unconditionally built the backend
  (and could git-clone in standalone mode) before printing anything
- the backend-.so check downgrades to informational
  '- not built yet (run `cargo oxide setup`)': run/build/pipeline
  build it on demand, so a fresh healthy clone stays green

README and book updated to describe the new checks and that cargo-oxide
plus doctor work without the CUDA toolkit.
cargo-oxide no longer pulls cuda-bindings, so its test job needs neither
the toolkit install (build-time bindgen) nor the libcuda.so.1 stub
shadow. The job now runs on a stock GPU-less, toolkit-less runner, which
makes it the exact environment of issue #87; a follow-up step runs the
built binary's doctor there and asserts the cuda.h diagnosis is reached
(doctor exits 1 by design on that runner, so only the output is
asserted).
The shadow existed solely because the cargo-oxide binary carried a
DT_NEEDED libcuda.so.1 (its run-mode GPU-arch probe used the driver
API). cargo-oxide now probes via nvidia-smi and links no CUDA at all,
so the binary loads on a GPU-less runner unaided. Example binaries are
still linked against the toolkit's lib64/stubs at link time and are
never executed in --compile-only mode, so nothing else on the runner
needs the soname shadow. Kept as its own commit so it reverts cleanly
if some future job step starts exec-ing a cuda-linked binary.
Doctor gained CUDA headers (cuda.h) and driver / GPU checks, the
backend-.so check became informational, and doctor no longer builds the
backend before running. Reflect all of that in the checks table and add
a short note on which checks are informational.
@nihalpasham nihalpasham added bug Something isn't working host-apis Host-side runtime APIs (cuda-host, cuda-core, cuda-async) cargo-oxide The `cargo oxide` subcommand and build automation build-related Build scripts, toolchain discovery, or compilation setup labels Jun 12, 2026
@nihalpasham
nihalpasham merged commit d5ece44 into main Jun 12, 2026
24 checks passed
@nihalpasham
nihalpasham deleted the fix/doctor-without-cuda-toolkit branch June 19, 2026 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working build-related Build scripts, toolchain discovery, or compilation setup cargo-oxide The `cargo oxide` subcommand and build automation host-apis Host-side runtime APIs (cuda-host, cuda-core, cuda-async)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cargo oxide doctor cannot diagnose missing CUDA headers because cuda-bindings panics first

1 participant