ci: nbd provisioning groundwork for Blacksmith (KVM jobs blocked on kernel >= 6.7) - #3332
ci: nbd provisioning groundwork for Blacksmith (KVM jobs blocked on kernel >= 6.7)#3332tomassrnka wants to merge 4 commits into
Conversation
Diagnostic-only workflow: dumps kernel config and device availability on Blacksmith x64/arm64 runners and attempts to build the nbd module from matching kernel.org source and insmod it. Establishes whether the orchestrator unit shard and integration tests can be provisioned to run there. Temporary; removed before any of this merges. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
PR SummaryMedium Risk Overview Reviewed by Cursor Bugbot for commit 3b2bb7f. Bugbot is set up for automated code reviews on this repo. Configure here. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
… on arm64 v1 verdicts: x64 kernel has nbd BUILTIN (CONFIG_BLK_DEV_NBD=y, /dev/nbd0 exists, /dev/kvm + tun + uffd all present) so no module is needed - only more device nodes, which nbd's netlink interface creates on demand. arm64 has no nbd at all, so it needs the source-built module; v1's modpost failed on undefined core symbols (no Module.symvers), fixed with KBUILD_MODPOST_WARN=1. Also add swapfile smoke test (init-client.sh swapon) and /proc/cmdline + device-count inventory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Probe results (see blacksmith-kernel-probe.yml runs): Blacksmith x64 kernels ship nbd BUILT IN (16 devices) plus /dev/kvm, tun, uffd, hugepages, swap and cgroup2 - everything the orchestrator needs; the device pool clamps to nbds_max so 16 devices just caps concurrency (unit tests use ~8, integration parallelism is -parallel=4). arm64 kernels ship no nbd at all, but a module built from matching kernel.org source with the running kernel's /proc/config.gz insmods cleanly (36s cold, cached per kernel release thereafter). - new ensure-nbd composite action: driver-present / modprobe / linux-modules-extra / build-from-source, in that order, so the same workflows keep working on GitHub-hosted runners - orchestrator unit shards: infra-tests -> blacksmith-8vcpu, infra-runner-arm -> blacksmith-4vcpu-arm - integration tests: infra-tests -> blacksmith-32vcpu - init-client.sh: skip modprobe when the driver is built in Verification criteria: TestSmokeAllFCVersions must RUN (not skip) on x64, no new environment-gated skips, no NoFreeSlotsError in integration service logs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| echo "strategy=none" >> "$GITHUB_OUTPUT" | ||
| elif sudo modprobe nbd nbds_max='${{ inputs.nbds-max }}' 2>/dev/null; then | ||
| echo "strategy=modprobe" >> "$GITHUB_OUTPUT" | ||
| elif apt-get -s install "linux-modules-extra-$(uname -r)" >/dev/null 2>&1; then |
There was a problem hiding this comment.
Apt strategy probe needs sudo
Medium Severity
The apt provisioning branch is chosen with apt-get -s install, which typically cannot read the dpkg lock without sudo, so the check fails even when linux-modules-extra-$(uname -r) is available and the action falls through to the kernel build path instead of installing the package.
Reviewed by Cursor Bugbot for commit c6e2820. Configure here.
This comment has been minimized.
This comment has been minimized.
Full-suite run on Blacksmith found the real blocker: their 6.6.141 kernel predates UFFD_FEATURE_WP_ASYNC (Linux >= 6.7), which the uffd memory backend and FC dirty-page tracking hard-require. 208 uffd test failures (UFFDIO_API EINVAL) and FC snapshot loads failing with 'Failed to UFFD object' on both arches. Not provisionable from CI - needs a Blacksmith kernel upgrade. Keep the groundwork (no behavior change on GitHub runners): - ensure-nbd action replaces inline modprobe/modules-extra setup in the unit-test workflows; it also handles builtin-nbd and no-module kernels - init-client.sh tolerates built-in nbd and pre-existing active swap - runner labels reverted to infra-tests / infra-runner-arm; comments document the exact kernel blocker and the revisit condition Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Final verdict: hard blocker is the kernel VERSION, not nbd/KVMThe full suite ran on Blacksmith (run 29915860237). Everything we set out to fix was fixed — nbd provisioned on both arches, The uffd memory backend and our Firecracker dirty-page tracking hard-require What this PR now is (mergeable, no behavior change on GitHub runners)
The ask for BlacksmithUpgrade guest kernels to ≥ 6.7 (6.12 LTS ideal). Everything else is already proven working on their runners: KVM boots Firecracker, built-in nbd + netlink device creation, tun/hugepages/cgroup2/swap. With a 6.7+ kernel, flipping the three runner labels back is the entire remaining migration — worth ~80% of our paid larger-runner minutes. 🤖 Generated with Claude Code |
tomassrnka
changed the title
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 3b2bb7f. Configure here.
| mkdir -p /tmp/nbd-ko | ||
| cp drivers/block/nbd.ko /tmp/nbd-ko/ | ||
| fi | ||
| sudo insmod /tmp/nbd-ko/nbd.ko nbds_max='${{ inputs.nbds-max }}' || { |
There was a problem hiding this comment.
Stale nbd.ko skips rebuild
Medium Severity
On the build strategy, a cache miss only rebuilds when /tmp/nbd-ko/nbd.ko is missing. After the runner kernel changes, an older .ko left on persistent /tmp skips the build and insmod uses the wrong module, causing failures or repeated failed jobs until the file is removed manually.
Reviewed by Cursor Bugbot for commit 3b2bb7f. Configure here.
|
Closing: blocked on Blacksmith shipping a kernel >= 6.7 (UFFD_FEATURE_WP_ASYNC). Everything needed to retry lives in this PR's history: the kernel probe workflow, the ensure-nbd action, and the init-client.sh tolerance fixes. |
❌ 1 Tests Failed:
View the top 3 failed test(s) by shortest run time
View the full list of 1 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
1 participant
Goal
Run the remaining GitHub-larger-runner jobs — orchestrator unit shards (x64 + arm64) and the integration suite — on Blacksmith, without losing any test coverage. #3326 found the blocker: Blacksmith's custom kernel ships no nbd module.
What the kernel probes established (runs on this PR)
CONFIG_BLK_DEV_NBD=y), 16 devices at boot/proc/config.gz) insmods cleanly: 36s cold, cacheable per kernel release/dev/kvmnbds_max=256)vsock is confirmed not needed:
PutGuestVsockhas zero callers; guest communication is tap networking + MMDS.Changes
.github/actions/ensure-nbd: one action, four strategies in order — driver already present → plainmodprobe→linux-modules-extravia apt (GitHub arm) → buildnbd.kofrom source + insmod (Blacksmith arm, cached). Workflows stay fully functional on GitHub-hosted runners.blacksmith-8vcpu-ubuntu-2404/blacksmith-4vcpu-ubuntu-2404-armblacksmith-32vcpu-ubuntu-2404init-client.shskips itsmodprobe nbdwhen the driver is built inMerge checklist (coverage, not just green)
TestSmokeAllFCVersionsruns and passes (it skips silently without/dev/kvm— a green check alone is not proof)requires root,/dev/kvm not available,cgroup v2, hugepage-mmap skips…)no free slots(NBD pool clamps to the 16 built-in devices on x64;-parallel=4should stay well under)blacksmith-kernel-probe.ymlbefore mergeKnown limits / fallbacks
NoFreeSlotsErrorever shows up, options: netlink pre-creation + a small pool change to size from the actual device inventory, or move integration back.uname -r) rolls over automatically and one job pays the 36s rebuild.🤖 Generated with Claude Code