ci: move ubuntu-latest jobs onto linux-amd64-cpu4 or larger runners #1518

Summary

Move every GitHub-hosted ubuntu-latest job onto NVIDIA nv-cpu-general runners (linux-amd64-cpu4 or larger). linux-amd64-cpu4 is an m7i.xlarge (4 vCPU / 16 GiB), the same size as ubuntu-latest. Tile jobs in cutile-rs.yml already use linux-amd64-cpu16.

Draft #1377 tried this for CodeQL only (linux-amd64-cpu8). That run is the source for the Rust gap below. This issue covers the rest of CI; #1377 does not need to land on its own.

Problem

GitHub-hosted ubuntu-latest is 4 vCPU / 16 GB. On this public repository those minutes are free, and that size is the whole free pool. A larger GitHub-hosted runner is billed here too. nv-cpu-general is the same class of machine at the EC2 rate, and this repository already has that group for Tile CI.

Rust CodeQL is the long job on the free runner: 38m 31s on main push 6d47ae4 (run), and 42m 51s on the #1377 baseline. On that same 6d47ae4 push, examples-compile took 9m 39s and clippy 5m 1s. Both are bound to those 4 cores. The workflows below are the ones still on ubuntu-latest.

#1377 put CodeQL on linux-amd64-cpu8 and it finished in 35m 12s. The Rust extractor never sees --threads. It loads each Cargo workspace through rust-analyzer with one worker thread, so the extra cores show up in queries and almost nowhere else. 8-core run, baseline. Compile jobs can use a larger label. CodeQL belongs on linux-amd64-cpu4.

Security

ubuntu-latest runs a pull_request workflow on its own, including one opened from outside NVIDIA. nv-cpu-general does not. Only an NVIDIA employee can start a job on these runners. An outside pull request runs only after an employee has checked that it is legitimate and benign and approved it through copy-pr-bot.

Each job gets its own VM, and that VM is destroyed when the job ends. Inbound network access to the VM is blocked. That is why the moved workflows trigger on push to main and to pull-request/<n>, which is what cutile-rs.yml already does. copy-pr-bot is already enabled (.github/copy-pr-bot.yaml). Draft pull requests do not auto-sync.

Cost

Rates are the list prices from #1377. EC2 is us-east-1 Linux on-demand. GitHub rounds each job up to the next minute. The runner group's region and any enterprise discount are not in the runner docs. GitHub does not meter nv-cpu-general. The bill is the instance. The 150 GiB volume on these labels is extra and small next to the instance.

6d47ae4 ran every ubuntu-latest job this move would take: 53 jobs, 93.1 minutes of runner time, 125 billed minutes. book.yml deploy is left out of that total. It stays on ubuntu-latest because actions/deploy-pages publishes GitHub Pages from a GitHub-hosted runner.

The cpu8 and cpu16 rows hold today's durations fixed, so they are a ceiling. cargo jobs get shorter on more cores, and the bill falls when the shorter run offsets the higher rate. It does not fall below today's $0.

CodeQL does not get shorter in proportion. The measured 8-core scan is 36 billed minutes, about $0.24. The same scan on cpu4 at today's 39 billed minutes is about $0.13. That is the reason to keep CodeQL on cpu4.

A pull request repeats whichever jobs its paths trigger. 6d47ae4 is one push that ran the full set, not an average day. Against a GitHub larger runner of the same size, m7i.2xlarge is about a third of the 8-core larger-runner rate ($0.00672 versus $0.022).

Jobs still on ubuntu-latest

ci.yml has no runs-on of its own. It calls fmt, clippy, unit-tests, and cargo-deny.

Leave book.yml's deploy job on ubuntu-latest. actions/deploy-pages publishes GitHub Pages and is run on GitHub-hosted runners.

Self-hosted runners do not have Rust

nv-cpu-general images have no rustup, rustc, cargo, or rustfmt. #1377 hit this on CodeQL: the Rust extractor logs a warning and exits 0, so the job is green with an empty database. A real analysis started only after the workflow installed rustup (ca068e0 on that branch). Same result as the ubuntu-latest baseline: 1,391 files extracted, 189 with errors. 8-core run, baseline.

Tile CI already installs Rust in the job, in cutile-rs/.github/actions/setup. SIMT jobs need the same step (shared action or a copy). rust-toolchain.toml still selects the pinned nightly at job time; baking one toolchain into the image does not replace that.

The image does already have what the non-Rust jobs use. nv-gha-runners/vm-images linux/installers/ installs build-essential, curl, ca-certificates, git, gh, python3 / pip / venv (Ubuntu Noble's Python, not 3.13), docker, jq, and yq. There is no installer for Rust, cargo-deny, a CUDA toolkit, or the apt.llvm.org LLVM 21/22 packages. Those last three are installed by the workflow today and can stay that way.

Requesting the missing tools

nv-gha-runners/enterprise-runner-configuration cannot add them. That repo grants repositories access to runner groups. Its agent instructions say a configuration PR must not change runner images. NVIDIA/cuda-rust already has nv-cpu-general (cutile-rs.yml uses linux-amd64-cpu16).

Image contents live in nv-gha-runners/vm-images. Adding a package means a new linux/installers/ script there.

A preinstalled tool is a platform change, not an access change. Support docs say to file a Request for Enhancement in nv-gha-runners/roadmap, or ask in #oss-ci-support first if it is unclear the image should carry it.

Worth requesting rustup on the Linux CPU image so a missing toolchain cannot turn CodeQL into an empty green scan. Not worth blocking this migration on that RFE: install rustup in the workflow, as Tile CI and #1377 already do. CUDA, LLVM, and cargo-deny stay workflow steps.

Other constraints from #1377 and Tile CI

• Self-hosted runners reject pull_request. Workflows that move have to trigger on push to main and to pull-request/[0-9]+, which is what cutile-rs.yml does. copy-pr-bot is already enabled (.github/copy-pr-bot.yaml). Draft PRs do not auto-sync.
• cargo-deny.yml's weekly schedule and ci-status.yml's workflow_run are not pull_request events, so they can move without a mirror branch.
• CodeQL on 8 cores was 35m 12s versus 42m 51s on 4 vCPU. Extract is single-threaded (rust-analyzer with num_worker_threads: 1). Extra cores only shortened queries. Keep CodeQL on linux-amd64-cpu4 unless a later run shows a reason to go bigger. Compile jobs (examples-compile, clippy, tests) are the ones that can use cpu8 or cpu16.

Test plan

• One push to a pull-request/<n> mirror runs each moved workflow on linux-amd64-cpu4 or larger, with rustup installed before any cargo or CodeQL Rust step.
• CodeQL Rust reports a non-empty database (thousands of files extracted, not a ~1m empty extract).
• book.yml deploy still runs on ubuntu-latest.
• A pull_request event no longer targets a self-hosted label.