Windows: fetch oneMKL through vcpkg manifest mode (fixes setup on VS-bundled vcpkg) - #185
Merged
Merged
Conversation
rileyjmurray
approved these changes
Aug 10, 2026
rileyjmurray
left a comment
rileyjmurray
left a comment
Contributor
There was a problem hiding this comment.
I dislike how much this complicates the (already complicated) Windows installs, but I don't know enough to push back on anything in particular. Approving.
mmelnich
added a commit
to BallisticLA/RandLAPACK
that referenced
this pull request
Aug 10, 2026
…ages Per review feedback on the RandBLAS counterpart (BallisticLA/RandBLAS#185): the manifest-mode machinery fixed the VS-bundled-vcpkg failure but added complexity. vcpkg's only job here was downloading oneMKL, which Intel also publishes as plain zip packages on nuget.org. Downloading those directly (pinned by version and SHA256, arranged into the oneAPI layout) removes vcpkg from the picture entirely: no distribution variance, no discovery, no manifest, no extra CI lane.
mmelnich
added a commit
to BallisticLA/RandLAPACK
that referenced
this pull request
Aug 12, 2026
…ages Per review feedback on the RandBLAS counterpart (BallisticLA/RandBLAS#185): the manifest-mode machinery fixed the VS-bundled-vcpkg failure but added complexity. vcpkg's only job here was downloading oneMKL, which Intel also publishes as plain zip packages on nuget.org. Downloading those directly (pinned by version and SHA256, arranged into the oneAPI layout) removes vcpkg from the picture entirely: no distribution variance, no discovery, no manifest, no extra CI lane.
This was referenced Aug 13, 2026
rileyjmurray
added a commit
that referenced
this pull request
Aug 13, 2026
This PR adds ... 1. new scripts `install/install.sh` and `install/install.ps1`, 2. an `install-script` GitHub Actions CI workflow that exercises the scripts, 3. documentation for the new scripts, and 4. console progress rendering. Both scripts build into a `RandNLA-project` tree laid out exactly as RandLAPACK's installer lays one out, and both honor `RANDNLA_PROJECT_DIR`. Running both projects' installers on one machine therefore **shares one dependency tree** rather than building BLAS++ twice. ### The integer width is read back, not assumed The part most worth reviewing. BLAS++ probes `int32` first and `int64` second, and `blas_int` only filters which *library names* to consider (`cmake/BLASFinder.cmake`). That makes ILP64 genuinely detectable for MKL, where `mkl_intel_ilp64` is a different library from `mkl_intel_lp64` — but not for OpenBLAS, where BLAS++ only ever tries `-lopenblas`. An LP64 OpenBLAS passes the `int32` probe and is accepted, so **a successful `blas_int=int64` configure proves nothing there**. The resolved width therefore comes from BLAS++'s generated `blas/defines.h` after the build. My first version recorded which configure attempt succeeded, which would have stamped an LP64 install as ILP64 and let every later run reuse it believing otherwise. Resulting policy: **ILP64 wherever it is real, LP64 with a named warning elsewhere.** Accelerate is refused outright for ILP64 — Apple has shipped one since macOS 13.3, but BLAS++ implements only the legacy interface ([icl-utk-edu/lapackpp#43](icl-utk-edu/lapackpp#43), tracked our side as #189). An ILP64 OpenBLAS is reachable through `--blas=custom`. ### Everything pinned, reuse gated on provenance BLAS++ (`3057185`), Random123 (`v1.14.0`), GoogleTest (`v1.18.0`) and LAPACK++ (`40b9d0d`) at immutable refs — the same refs RandLAPACK validated. Each install *and each source tree* carries a provenance stamp and is reused only when it matches what we would build now, including backend and integer width. Without that, changing a pin is a silent no-op for anyone who already ran the script. The source stamp exists because a shallow checkout of a tag does not keep the tag ref locally. On the Windows side this replaces `Clone-Head`, which took a branch name and returned early whenever the destination merely existed — so a branch tip could move between runs, and Random123 was fetched at its default branch entirely unpinned. It also moves BLAS++/LAPACK++ off the `RaphaelArkadyMeyerNYU/*` forks; both MSVC fixes merged upstream on 2026-08-06. ### Verification runs, it does not just link The final step compiles, links and *runs* a program against the finished install: it sketches a matrix, multiplies through BLAS++, and checks the result is symmetric positive semidefinite. Through BLAS++ rather than raw `dgemm_`, because that is how RandBLAS actually reaches the BLAS. ### The x64 toolchain guard (Windows) "Developer PowerShell for VS 2022" and "Developer Command Prompt for VS 2022" **both default to an x86 toolchain**, and an x86 linker cannot use the x64 import libraries every BLAS backend ships. Unchecked, the failure surfaces three layers down as BLAS++ reporting `BLAS library not found`. Shell bitness is not a usable signal either — the Developer Command Prompt is a 64-bit process that still selects x86 tools. `toolchain-arch.ps1` reads `VSCMD_ARG_TGT_ARCH`, then the `bin\Host<host>\<target>\` convention, then `cl.exe`'s banner, and refuses x86 and arm64 with different messages because they need different answers. ### Smaller decisions - **GoogleTest is provisioned**, because `BUILD_TESTS` defaults to ON while `find_package(GTest)` is not `REQUIRED` — so a machine without it silently produces a build with zero tests. - **Examples are opt-in.** They need two dependencies RandBLAS does not (LAPACK++, `fast_matrix_market`) and `examples/CMakeLists.txt` requires OpenMP, which stock Apple Clang cannot supply. Declining prints the exact follow-up command; it re-invokes the same script, so there is one code path. - **`randblas_stage_runtime_dlls()` is now exported from the installed package.** Windows searches an executable's own directory first and `PATH` last, so a downstream project linking installed RandBLAS had no way to find the BLAS DLLs — the function lived only in the build tree. Found because the installer's own verification step is such a consumer. - **The clone is never moved.** RandLAPACK's installer relocates its own repository into `lib/`, which breaks git worktrees. A symlink gives the same layout. - **The log is appended, not truncated** — the previous run's output is what you want when this run fails the same way. - `.gitignore` matched `**install/`, intended for build-output trees but also swallowing a source directory named `install/`. Narrowed to `**/*-install/`, the pattern RandLAPACK already uses; the `!**install/.gitkeep` exception it carried was vestigial. - **Progress rendering** has three tiers: Unicode bar on a capable terminal, ASCII bar without colour or UTF-8, and one line per step when not a terminal. Tier 0 is a requirement rather than a fallback — redirected output becomes `install.log` and CI transcripts. The bar is determinate, parsed from Ninja's `[12/34]` and Make's `[ 42%]` on the stream already being captured. ## CI The `install-script` workflow asserts what the core workflows structurally cannot: idempotent re-runs, dependency discovery, the LP64 fallback warning firing for stock OpenBLAS, ILP64 for MKL, Accelerate refusing ILP64, and **piped output containing no terminal escape sequences** — which is what stops the progress bar from quietly filling every CI log. A separate **packager lane deliberately does not run the installer**. It configures with plain CMake against hand-installed dependencies, inside a network namespace with no interfaces, then builds, installs and consumes the result via `find_package`. If that lane stays green a conda-forge recipe and a Spack `package.py` are possible. ## Verification **Linux** (gcc 15.2, oneAPI MKL): | Scenario | Result | |---|---| | Fresh MKL build | 9/9 steps, ILP64 confirmed from `blas/defines.h` | | Re-run in place | all six dependency steps reused, seconds | | Dependency discovery (`*_INSTALL_DIR` pre-seeded) | all three reused, step count drops to 3/3 | | `--examples` | 13/13 steps, all 7 binaries, `fast_matrix_market` at `v1.7.6` | | `--blas=openblas` with no OpenBLAS present | clean failure naming both apt packages | | `--blas=accelerate --blas-int=ilp64` | refused at argument parsing | | Output redirected | no ANSI escapes, no carriage returns | | Progress under a pty | advances 5% → 20% → 49% during the RandBLAS build | **Windows 11, Windows PowerShell 5.1** (not pwsh 7), VS 2022 Build Tools: | Scenario | Result | |---|---| | Plain shell, no `cl.exe`/`cmake` | refused at preflight with the `vcvars64` one-liner | | Real x86 toolchain (`vcvars32.bat`, `VSCMD_ARG_TGT_ARCH=x86`) | refused with the x86-specific explanation | | Full x64 install from scratch | oneMKL via vcpkg, BLAS++, GoogleTest, RandBLAS | | Verification program | compiled, linked, staged its DLLs, ran, reported ILP64 | All `install-script` lanes green, including the offline packager lane. ## Notes for posterity - **Two bugs here were only reachable on real hardware**. `$LASTEXITCODE` is unset until some native command runs and `Set-StrictMode` makes reading an unset variable an error, so the installer failed on exactly the runs where every dependency was cached. And the missing `randblas_stage_runtime_dlls` export only appears when something consumes the *installed* package. - **vcpkg is kept, deliberately.** My plan called for replacing it with pinned NuGet as RandLAPACK did, but #185 had just made vcpkg work in manifest mode with a pinned baseline, and ripping that out immediately would be churn. Worth revisiting: the vcpkg oneMKL path expanded to **6.2 GB** on disk during testing, against roughly 155 MB for RandLAPACK's NuGet fetch. - Windows CI for the installer itself is not added here; the existing `core-windows` lanes cover the provisioner changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On a plain Visual Studio 2022/18 (or Build Tools) machine, the Windows dependency setup fails immediately at the oneMKL step:
setup.ps1located the vcpkg copy that Visual Studio bundles with the C++ workload and ran a classic-modevcpkg install intel-mkl:x64-windows. The bundled copy is manifest-only: it lives read-only under Program Files and has no classic-mode instance, so every classic invocation fails. CI never caught this because GitHub's windows-2022 runners ship a standalone, classic-capable vcpkg atC:\vcpkg(exported asVCPKG_INSTALLATION_ROOT), which the discovery logic finds first. The manual recipe in INSTALL.md Appendix A taught the same classic-mode command. First reported against RandLAPACK's installer, which reuses this pattern.Change
setup.ps1now provisions oneMKL in manifest mode, the one mode every vcpkg distribution supports. It generates a minimalvcpkg.jsonunder the dependency root and redirects vcpkg's downloads/buildtrees/packages scratch trees there as well, since the bundled copy's default scratch locations are not writable. The manifest pinsbuiltin-baselineto vcpkg release 2026.07.29 (intel-mkl 2025.2.0, matching the current CI cache key); the bundled vcpkg requires that field, and the pin makes the installed oneMKL version independent of how old the user's vcpkg copy is. The installed-tree layout (vcpkg-installed\x64-windows) is unchanged, so caches and downstream paths are untouched.VSINSTALLDIR(a developer prompt does not always put it onPATH).windows-vs-bundled-vcpkg(core workflow): hides the runner's standalone vcpkg and runs dependency setup plus the serial core build through the VS-bundled copy, reproducing the user environment that was broken. Thevcpkg-installedtree is deliberately not cached so the manifest-mode fetch is exercised on every run; only the oneMKL installer download is cached.Verification
setup.ps1parses clean under Windows PowerShell and the workflow is valid YAML. The end-to-end proof is the new CI lane in this PR, which fails onmain's classic-mode invocation by construction and must pass here.A matching RandLAPACK PR (BallisticLA/RandLAPACK#156) applies the same fix to its installer; its RandBLAS submodule pin can pick this change up once merged.
🤖 Generated with Claude Code