ci: move ubuntu-latest jobs onto linux-amd64-cpu4 by roivanov · Pull Request #1580 · NVIDIA/cuda-rust

roivanov · 2026-10-20T21:38:01Z

Summary

Fixes #1518. Move the GitHub-hosted ubuntu-latest jobs onto linux-amd64-cpu4 (m7i.xlarge, 4 vCPU / 16 GiB), the same size as ubuntu-latest. Tile test jobs in cutile-rs.yml already run on linux-amd64-cpu16 and stay there. book.yml's deploy job stays on ubuntu-latest, because actions/deploy-pages publishes GitHub Pages from a GitHub-hosted runner.

Supersedes #1377. That draft was closed; this one is opened from roivanov/cuda-rust.

Problem

ubuntu-latest is 4 vCPU / 16 GB. On this public repository those minutes are free, and that size is the whole free pool. A larger GitHub-hosted runner is billed here too. nv-cpu-general is the same class of machine at the EC2 rate, and this repository already has that group for Tile CI.

Rust CodeQL is the long job. On main push 6d47ae4 it took 38m 31s (run); the baseline on ubuntu-latest was 42m 51s (job). On that same push, examples-compile took 9m 39s and clippy 5m 1s. An 8-core run left the Rust extract at about 35 minutes: the extractor loads each Cargo workspace through rust-analyzer with one worker thread. This pull request keeps every moved job on linux-amd64-cpu4.

Security

ubuntu-latest runs a pull_request workflow on its own, including one opened from outside NVIDIA. nv-cpu-general does not. Only an NVIDIA employee can start a job on these runners. An outside pull request runs only after an employee has checked that it is legitimate and approved it through copy-pr-bot.

Each job gets its own VM, destroyed when the job ends. Inbound network access is blocked. The moved workflows trigger on push to main and to pull-request/[0-9]+, which is what cutile-rs.yml already does. copy-pr-bot is already enabled (.github/copy-pr-bot.yaml). Draft pull requests do not auto-sync.

Cost

NVIDIA/cuda-rust is public. Standard ubuntu-latest minutes are free. GitHub larger runners are billed on public repositories too. nv-cpu-general is self-hosted EC2: GitHub does not meter it. The bill is the instance.

Rates are list price. GitHub rounds each job up to the next minute. EC2 is us-east-1 Linux on-demand; the runner group's region and any enterprise discount are not in the runner docs. The 150 GiB volume on these labels is extra and small next to the instance.

6d47ae4 ran every ubuntu-latest job this move takes: 53 jobs, 93.1 minutes of runner time, 125 billed minutes. book.yml deploy is left out of that total.

Runner	Machine	$/min	This push
`ubuntu-latest` (this repo, public)	4 vCPU / 16 GB	$0	$0
GitHub larger Linux 4-core	4 vCPU / 16 GB	$0.012	$1.50
`linux-amd64-cpu4` (this PR)	m7i.xlarge, 4 vCPU / 16 GiB	$0.00336	$0.42

The Rust CodeQL job, measured in rust:1.98.0-bookworm on linux-amd64-cpu8, was 36 billed minutes (run): about $0.24 on cpu8, about $0.12 on cpu4 if that duration holds. Installing rustup on the 8-core VM took 35m 12s (run). The container run took the same 35 minutes.

Changes

runs-on: linux-amd64-cpu4 for the jobs that were on ubuntu-latest: CI's callees (fmt, clippy, unit-tests, cargo-deny), docs, examples-compile, codeql, book build, status-guard, naming-guard, ci-status, and the changes job in cutile-rs.yml.
Drop pull_request. Trigger on push to main and pull-request/[0-9]+.
Jobs that call cargo, rustfmt, or the CodeQL Rust extractor run in rust:1.98.0-bookworm. That matches the root rust-toolchain.toml pin. The Oxide nightly still comes from cuda-oxide/rust-toolchain.toml when the job enters that tree. The nv-cpu-general image has no rustup, and a missing rustup makes the Rust extractor exit 0 with an empty database. The Rust leg fails if rustup show active-toolchain cannot see the root pin.
CUDA jobs (clippy, docs, examples-compile, and the cuda.h unit tests) run in nvidia/cuda:13.1.1-devel-ubuntu24.04. The toolkit installer needs lsb_release and an Ubuntu apt repo, so it fails in rust:bookworm. The devel image already has the toolkit. Container steps use bash; the default shell is dash, which rejects set -o pipefail.
The CUDA jobs call .github/actions/setup-cuda-container with the $/ form, which loads the action from the workflow's own commit before checkout. It installs git first (the CUDA image has none), so checkout gets a real clone, then marks the workspace safe for git and installs a pinned, checksum-verified rustup-init. examples-compile also installs python3 for sync-example-locks.sh.
book.yml deploy stays on ubuntu-latest and runs only for refs/heads/main, so a mirror push cannot publish the site.
Tile test jobs in cutile-rs.yml stay on linux-amd64-cpu16.
Path-filtered workflows (ci, docs, examples-compile, codeql, book, naming-guard) no longer use a workflow-level paths: filter. copy-pr-bot creates pull-request/<n> at an existing SHA, so GitHub often sees no changed files and would skip the run. Each has a changes job (dorny/paths-filter against main, as cutile-rs.yml does); the jobs run when it matches, when no diff is detected, when the event is not a push, or when the gate itself fails.
The intrinsic ABI ledger check compares against the merge base with origin/main on pull-request/<n> pushes, not HEAD^, so an earlier commit of a multi-commit PR cannot hide a breaking change.
CodeQL uploads to refs/pull/<n>/head with the pushed SHA on mirror pushes, so results reach the pull request's code scanning alerts.
cuda-oxide/scripts/check-ci-image-pins.sh (run in the toolchain-pin-parity job and just check-guards) fails if a workflow's rust:<version> image no longer matches the root rust-toolchain.toml.

Result

Container run on linux-amd64-cpu8: Rust analysis 35m 1s. rustc 1.98.0 (88d9e12ae 2026-08-18), active toolchain overridden by the root rust-toolchain.toml. 1,391 files extracted without error, 190 with errors, 1,580 scanned. Analyze (actions) passed in the same image.

Testing

CodeQL Rust on the container run matches the real extract shape (1,391 clean), not an empty ~1m scan: https://github.com/NVIDIA/cuda-rust/actions/runs/38802502685
Baseline: https://github.com/NVIDIA/cuda-rust/actions/runs/37042999630/job/110957435112
A push to pull-request/<n> runs each moved workflow on linux-amd64-cpu4: on head dc0de9b, 62 jobs ran on linux-amd64-cpu4, 6 on linux-amd64-cpu16, and only the skipped deploy on ubuntu-latest
book.yml deploy still runs on ubuntu-latest and only for main (skipped on the mirror push; runs after merge)
No workflow triggers on pull_request any more, so none can target a self-hosted label
A pull request that touches no workflow file: the changes gates start the right workflows and skip the rest (this PR touches .github/workflows/**, so every filter matched)
ci-status runs on the bare runner VM and calls gh; it only runs after a main push, so it is first exercised after merge

Workflow-only change. just check and example smoketests do not apply.

Post-PR TODOs

Reorganize the workflows so we build the container image once, push it to ephemeral storage, and reuse it across jobs.

Checklist

All commits signed off (git commit -s)
SPDX headers on new source files

Questions about the review? Ping us in #contributors on Discord.

roivanov · 2026-10-20T21:56:56Z

@cryos @assignUser please review. I re-created PR from my fork and merged latest main

assignUser · 2026-10-22T22:57:56Z

A local action is loaded from the workspace, so it cannot be the first step.

Actually there is a new syntax where you start the path with $/path/to/action instead of just a file path. This causes the action to be checked out prior to the actual workflow running, as if it was an external action. This allows it to run without a checkout. I have used this here: https://github.com/NVIDIA/cuda-rust/pull/1607/changes/9ab6e688d1e276295732cf728e91b8e04ebb656c..8454bb6cf50956f90df8043e41901b1b948856fd

roivanov · 2026-10-22T23:14:39Z

A local action is loaded from the workspace, so it cannot be the first step.

Actually there is a new syntax where you start the path with $/path/to/action instead of just a file path. This causes the action to be checked out prior to the actual workflow running, as if it was an external action. This allows it to run without a checkout. I have used this here: https://github.com/NVIDIA/cuda-rust/pull/1607/changes/9ab6e688d1e276295732cf728e91b8e04ebb656c..8454bb6cf50956f90df8043e41901b1b948856fd

It worked because we have git checkout before each call. But let me check this new feature!

assignUser · 2026-10-22T23:17:06Z

Yes, but with this you can install git first and get a proper git checkout instead of just the archive from the api fallback!

roivanov · 2026-10-23T19:15:48Z

please take another look @assignUser

Self-hosted runners reject pull_request, so these workflows trigger on the copy-pr-bot mirror. Cargo jobs use rust:1.98.0-bookworm. CUDA jobs use the Ubuntu devel image, which already has the toolkit, and check the repo out before the local setup action. The Pages deploy stays on ubuntu-latest. Signed-off-by: Roman Ivanov <rivanov@nvidia.com>

The CUDA image has no git, so checkout falls back to an API tarball unless git is already installed. The $/ action reference runs that install first. Signed-off-by: Roman Ivanov <rivanov@nvidia.com>

A blank line between workflow steps makes the step list easier to scan. Signed-off-by: Roman Ivanov <rivanov@nvidia.com>

copy-pr-bot creates pull-request/<n> at an existing SHA, and GitHub often treats that push as having no changed files. A workflow-level `paths:` filter then skips the run, so a pull request can pass with CI, docs, examples, CodeQL, book and naming checks never started. cutile-rs.yml already avoids this. Drop `paths:` from ci, docs, examples-compile, codeql, book and naming-guard. Each now has a `changes` job that runs dorny/paths-filter against main with the old path list. Jobs run when the filter matches, when no file diff is detected, or when the event is not a push (schedule, workflow_dispatch). If the gate job itself fails the jobs run, so the gate cannot hide a check. Signed-off-by: Roman Ivanov <rivanov@nvidia.com>

…ror pushes The append-only ABI ledger check picked its base from the pull_request event. On a copy-pr-bot mirror push there is no such event, and on the first push to a new pull-request/<n> branch `before` is all zeros, so the base fell back to HEAD^. Only the last commit was compared, and a breaking ledger change in an earlier commit of a multi-commit pull request passed the final check. For pull-request/<n> branches use the merge base with origin/main, which the fetch-depth 0 checkout already has. Other pushes keep `before`, then HEAD^. Print the chosen base in the log. Signed-off-by: Roman Ivanov <rivanov@nvidia.com>

On a copy-pr-bot mirror push the analyze action uploads under refs/heads/pull-request/<n>, so the scan runs and the job goes green but its results never attach to the pull request's code scanning alerts. Resolve refs/pull/<n>/head and the pushed SHA, which is the pull request's head commit, and pass them to the analyze action's `ref` and `sha` inputs. Pushes to main and scheduled runs leave both empty and keep the default ref. Signed-off-by: Roman Ivanov <rivanov@nvidia.com>

The setup-cuda-container action installed Rust with `curl https://sh.rustup.rs | sh`, with no version and no checksum. The Jimver/cuda-toolkit step it replaced was pinned to a commit. Download rustup-init 1.29.2 from the versioned archive URL and check its published SHA-256 before running it. Fail on a non-x86_64 host, since the pinned checksum is for that target. Signed-off-by: Roman Ivanov <rivanov@nvidia.com>

The book build interpolated ${{ github.ref }} straight into the shell script. A branch name is chosen by whoever pushes it, so pass it through the environment variable GitHub already sets instead of expanding it into the script text. Signed-off-by: Roman Ivanov <rivanov@nvidia.com>

The Rust jobs now run in rust:1.98.0-bookworm because the runner VM image has no rustup. That tag copies the stable pin in the root rust-toolchain.toml, in twelve places, and nothing keeps them together; the existing parity guard only covers the SIMT pin under cuda-oxide/. A stale tag still builds, since rustup installs the pinned toolchain, but it costs a download in every job and puts the CodeQL Rust extractor on a different compiler. Add check-ci-image-pins.sh and run it in the toolchain-pin-parity job and in `just check-guards`. It fails if no rust:<version> reference is found, so it cannot pass by matching nothing. Signed-off-by: Roman Ivanov <rivanov@nvidia.com>

copy-pr-bot · 2026-10-26T18:29:12Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

roivanov · 2026-10-26T18:30:49Z

/ok to test 316a1e0

roivanov marked this pull request as ready for review October 20, 2026 21:56

roivanov requested review from binarybana, cryos, elibol, kkraus14, nihalpasham and rparolin as code owners October 20, 2026 21:56

roivanov force-pushed the ci/codeql-cpu8 branch 3 times, most recently from deb008a to dc0de9b Compare October 23, 2026 17:41

roivanov marked this pull request as draft October 23, 2026 21:22

roivanov added 9 commits October 26, 2026 11:23

ci: install git before checkout from the CUDA setup action

1c3dacf

The CUDA image has no git, so checkout falls back to an API tarball unless git is already installed. The $/ action reference runs that install first. Signed-off-by: Roman Ivanov <rivanov@nvidia.com>

ci: separate the CUDA setup step from checkout

0c4cc4c

A blank line between workflow steps makes the step list easier to scan. Signed-off-by: Roman Ivanov <rivanov@nvidia.com>

roivanov force-pushed the ci/codeql-cpu8 branch from df17c90 to 316a1e0 Compare October 26, 2026 18:29

NVIDIA deleted a comment from copy-pr-bot Bot Oct 26, 2026

roivanov marked this pull request as ready for review October 26, 2026 19:18

Navigation Menu

Repository navigation

ci: move ubuntu-latest jobs onto linux-amd64-cpu4 - #1580

ci: move ubuntu-latest jobs onto linux-amd64-cpu4#1580
roivanov wants to merge 9 commits into
NVIDIA:mainfrom
roivanov:ci/codeql-cpu8

roivanov commented Oct 20, 2026 •

edited

Loading

Uh oh!

roivanov commented Oct 20, 2026

Uh oh!

assignUser commented Oct 22, 2026

Uh oh!

roivanov commented Oct 22, 2026

Uh oh!

assignUser commented Oct 22, 2026 •

edited

Loading

Uh oh!

roivanov commented Oct 23, 2026

Uh oh!

copy-pr-bot Bot commented Oct 26, 2026

Uh oh!

roivanov commented Oct 26, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

roivanov commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Security

Cost

Changes

Result

Testing

Post-PR TODOs

Checklist

Uh oh!

roivanov commented Oct 20, 2026

Uh oh!

assignUser commented Oct 22, 2026

Uh oh!

roivanov commented Oct 22, 2026

Uh oh!

assignUser commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

roivanov commented Oct 23, 2026

Uh oh!

copy-pr-bot Bot commented Oct 26, 2026

Uh oh!

roivanov commented Oct 26, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

roivanov commented Oct 20, 2026 •

edited

Loading

assignUser commented Oct 22, 2026 •

edited

Loading