Skip to content

ci: Windows CPU integration smoke job + OS label routing fix (RoFormer Windows plan, part 1) #294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
beveradb merged 4 commits into from
Jul 20, 2026
Merged

ci: Windows CPU integration smoke job + OS label routing fix (RoFormer Windows plan, part 1) #294

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4492be4
ci: add Windows CPU integration smoke job + OS label routing fix
beveradb Jul 20, 2026
47457cb
ci/docs: address review feedback (gate messaging, doc precision, plan…
beveradb Jul 20, 2026
3e83eb4
fix(tests): resolve audio-separator CLI next to sys.executable for Wi…
beveradb Jul 20, 2026
eb08cd6
fix(ci): create model dir before Windows integration tests
beveradb Jul 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 68 additions & 8 deletions .github/workflows/run-integration-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ jobs:
ensemble-presets:
needs: changes
if: needs.changes.outputs.should_run == 'true'
runs-on: [self-hosted, gpu]
runs-on: [self-hosted, linux, gpu]
timeout-minutes: 15
env:
AUDIO_SEPARATOR_MODEL_DIR: /opt/audio-separator-models
Expand Down Expand Up @@ -79,7 +79,7 @@ jobs:
core-models:
needs: changes
if: needs.changes.outputs.should_run == 'true'
runs-on: [self-hosted, gpu]
runs-on: [self-hosted, linux, gpu]
timeout-minutes: 15
env:
AUDIO_SEPARATOR_MODEL_DIR: /opt/audio-separator-models
Expand Down Expand Up @@ -128,7 +128,7 @@ jobs:
stems-and-quality:
needs: changes
if: needs.changes.outputs.should_run == 'true'
runs-on: [self-hosted, gpu]
runs-on: [self-hosted, linux, gpu]
timeout-minutes: 15
env:
AUDIO_SEPARATOR_MODEL_DIR: /opt/audio-separator-models
Expand Down Expand Up @@ -167,10 +167,61 @@ jobs:
*.flac
tests/*.flac

# ── Windows CPU integration smoke (GitHub-hosted, free) ────────────
#
# End-to-end separation on Windows with the CPU provider — one model per
# architecture that DirectML supports (RoFormer, MDX, VR). Catches
# Windows-specific breakage (paths, soundfile/ffmpeg, model download)
# independent of GPU/DirectML. Python 3.12 to match the DirectML jobs
# (torch-directml has no 3.13 wheels).
#
# Models are downloaded on first run and cached with actions/cache
# (hosted runners have no pre-baked model dir).

windows-cpu-integration:
needs: changes
if: needs.changes.outputs.should_run == 'true'
runs-on: windows-latest
timeout-minutes: 45
env:
AUDIO_SEPARATOR_MODEL_DIR: C:\audio-separator-models
steps:
- uses: actions/checkout@v4
- name: Install poetry
run: pipx install poetry
- name: Set up Python with cache
uses: actions/setup-python@v5
with:
python-version: '3.12'
cache: poetry
- name: Install ffmpeg
run: choco install ffmpeg -y --no-progress
- name: Install Poetry dependencies (CPU)
run: poetry install -E cpu
- name: Cache separation models
uses: actions/cache@v4
with:
path: C:\audio-separator-models
key: windows-cpu-smoke-models-v1
- name: Ensure model directory exists
# Separator raises FileNotFoundError if AUDIO_SEPARATOR_MODEL_DIR
# doesn't exist (it never creates it); a cache miss leaves it absent.
run: New-Item -ItemType Directory -Force -Path C:\audio-separator-models | Out-Null
- name: "Run: one model per DML-relevant architecture (RoFormer, MDX, VR)"
run: poetry run pytest -sv tests/integration/test_cli_integration.py -k "mel_band_roformer_karaoke or Inst_HQ_4 or 2_HP-UVR"
- name: Upload test artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: windows-cpu-integration-results
path: |
*.flac
tests/*.flac

# ── Gate job for branch protection ────────────────────────────────

integration-test:
needs: [changes, ensemble-presets, core-models, stems-and-quality]
needs: [changes, ensemble-presets, core-models, stems-and-quality, windows-cpu-integration]
if: always()
runs-on: ubuntu-latest
steps:
Expand All @@ -181,14 +232,23 @@ jobs:
exit 0
fi

echo "ensemble-presets: ${{ needs.ensemble-presets.result }}"
echo "core-models: ${{ needs.core-models.result }}"
echo "stems-and-quality: ${{ needs.stems-and-quality.result }}"
echo "ensemble-presets: ${{ needs.ensemble-presets.result }}"
echo "core-models: ${{ needs.core-models.result }}"
echo "stems-and-quality: ${{ needs.stems-and-quality.result }}"
echo "windows-cpu-integration: ${{ needs.windows-cpu-integration.result }}"

if [[ "${{ needs.ensemble-presets.result }}" == "failure" ]] || \
[[ "${{ needs.core-models.result }}" == "failure" ]] || \
[[ "${{ needs.stems-and-quality.result }}" == "failure" ]]; then
echo "Integration tests failed"
exit 1
fi
echo "All integration tests passed"

# windows-cpu-integration is reported above but does not fail the
# gate yet — it gets promoted to blocking (here + ruleset) after a
# stabilization period. See docs/CI-GPU-RUNNERS.md.
if [[ "${{ needs.windows-cpu-integration.result }}" == "success" ]]; then
echo "All integration tests passed"
else
echo "GPU integration tests passed; windows-cpu-integration did not succeed (non-blocking)"
fi
184 changes: 116 additions & 68 deletions docs/CI-GPU-RUNNERS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,58 +4,106 @@ This document explains how the GPU-based integration test infrastructure works f

## Overview

Integration tests require GPU hardware to run ML model inference. GPU VMs are expensive (~$1.62/hr for 3x T4), so they auto-scale to zero when idle. The system automatically starts runners when CI jobs need them and stops them after 15 minutes of inactivity.
Integration tests require GPU hardware to run ML model inference. GPU VMs are
expensive, so the fleet is **fully ephemeral**: every queued CI job gets a
fresh single-use GCE VM created on demand from a pre-baked image, and the VM
deletes itself when the job finishes. Nothing runs (or costs money) while CI
is idle.

> **History**: until 2026-05-18 this was a fixed pool of long-lived VMs
> (`github-gpu-runner-{1,2,3}`) that were started/stopped on demand. Phase 4
> of the ephemeral-runners rollout (karaoke-gen PR #780) deleted that pool.
> If you see references to starting/stopping runner VMs, they're stale.

## Architecture

```
GitHub webhook (workflow_job.queued)
GitHub org webhook (workflow_job.queued)
│
▼
Cloud Function (github-runner-manager)
│
├── Job has "gpu" label? → Start GPU runners (3x n1-standard-4 + T4)
├── Job has "self-hosted" label? → Start CPU runners
└── Neither? → Ignore
├── resolve image family from job labels:
│ gpu → gha-runner-gpu (Linux, n1-standard-4 + T4)
│ docker-build → gha-runner-build (Linux, e2-standard-8)
│ otherwise → gha-runner-general (Linux, e2-standard-4)
│ windows+gpu → gha-runner-gpu-windows (Windows Server + T4; in progress)
│
├── mint a JIT (just-in-time) ephemeral runner config via GitHub API
└── create a single-use GCE VM from the family image
│
▼
VM boots, runs ONE job (`run.sh --jitconfig`), de-registers,
shuts itself down (boot disk auto-deletes on VM delete)

Cloud Scheduler (every 15 min)
│
▼
Cloud Function (?action=check_idle)
Cloud Function (?action=cleanup_orphans)
│
└── No pending jobs + runner idle > 15 min? → Stop runner
├── delete VMs whose runner is gone (age > 30 min) or hung (age > 120 min)
└── de-register zombie runner registrations with no live VM
```

### Components
### Components (all in the karaoke-gen repo)

| Component | Location | Purpose |
|-----------|----------|---------|
| Cloud Function | `karaoke-gen/infrastructure/functions/runner_manager/main.py` | Starts/stops runner VMs based on demand |
| Pulumi module | `karaoke-gen/infrastructure/modules/runner_manager.py` | Deploys the function, scheduler, and IAM |
| GPU VM definitions | `karaoke-gen/infrastructure/compute/github_runners.py` | 3x n1-standard-4 with T4 GPU |
| GPU startup script | `karaoke-gen/infrastructure/compute/startup_scripts/github_runner_gpu.sh` | Installs NVIDIA drivers, Python, registers runner |
| Config | `karaoke-gen/infrastructure/config.py` | Runner count, labels, idle timeout |
| GitHub webhook | Org-level (`nomadkaraoke`) | Sends `workflow_job` events to Cloud Function |
| Dispatcher + cleanup | `karaoke-gen/infrastructure/functions/runner_manager/ephemeral.py` | Family resolution, JIT config, VM create, orphan cleanup |
| Webhook entry point | `karaoke-gen/infrastructure/functions/runner_manager/main.py` | Signature verification, event routing |
| Image bake workflow | `karaoke-gen/.github/workflows/build-runner-images.yml` | Builds `gha-runner-<variant>` image families (monthly cron + manual dispatch) |
| Image provisioning | `karaoke-gen/infrastructure/scripts/runner-image-provision.sh` | Installs NVIDIA driver, Python, Poetry, runner binary, ~14GB model cache |
| GitHub webhook | Org-level (`nomadkaraoke`) | Sends `workflow_job` events to the Cloud Function |

### GPU runner VMs (ephemeral)

- **Name pattern**: `gha-gpu-<hex>` (also `gha-general-<hex>`, `gha-build-<hex>`)
- **Machine type**: n1-standard-4 (4 vCPU, 15GB RAM) + 1× NVIDIA T4
- **Zones**: us-central1-a primary, us-east4-c fallback on stockout
- **Image**: `gha-runner-gpu` family — NVIDIA driver and ~14GB of models at
`/opt/audio-separator-models` are baked in, so job start is fast
- **Lifetime**: one CI job (~7 min typical), then self-destructs

### Runner labels and `runs-on`

### GPU Runner VMs
Linux GPU runners advertise: `self-hosted, linux, x64, gcp, gpu`.
Windows GPU runners (in progress) advertise: `self-hosted, windows, x64, gcp, gpu`.

- **Count**: 3 (configurable via `NUM_GPU_RUNNERS` in config.py)
- **Machine type**: n1-standard-4 (4 vCPU, 15GB RAM) + 1x NVIDIA T4
- **Zone**: us-central1-a
- **Labels**: `self-hosted, linux, x64, gcp, gpu`
- **Startup time**: ~15-20 min (NVIDIA driver install, Python build, model download)
- **Model cache**: ~14GB of ML models pre-downloaded to `/opt/audio-separator-models/`
**Always include the OS label in `runs-on`** (e.g.
`[self-hosted, linux, gpu]`). GitHub schedules a job onto any runner whose
labels are a superset of the job's — a bare `[self-hosted, gpu]` job could be
picked up by a Windows GPU runner.

### Required GitHub Branch Protection Checks
## Windows coverage

Two tiers (added 2026-07 for RoFormer/DirectML support, issue #292):

1. **`windows-cpu-integration`** — GitHub-hosted `windows-latest` (free),
runs one model per DirectML-relevant architecture (RoFormer, MDX, VR)
end-to-end on CPU. Models are cached via `actions/cache`. Python 3.12
(torch-directml has no 3.13 wheels, and the DML jobs must match).
2. **`windows-directml`** (planned) — self-hosted ephemeral Windows Server +
T4 VM (`gha-runner-gpu-windows` family), runs separation with
`--use_directml` and compares output quality against CPU results. The
image uses the NVIDIA **GRID** driver (WDDM mode) — the datacenter driver
puts the T4 in TCC mode, which has no DirectX support and breaks DirectML.

## Required GitHub branch protection checks

The `Protect main` ruleset (ID: 529535) requires these checks to pass before merge:

- `unit-tests` — from `run-unit-tests.yaml` (runs on GitHub-hosted runners)
- `ensemble-presets` — from `run-integration-tests.yaml` (runs on GPU runners)
- `core-models` — from `run-integration-tests.yaml` (runs on GPU runners)
- `stems-and-quality` — from `run-integration-tests.yaml` (runs on GPU runners)
- `unit-tests` — from `run-unit-tests.yaml` (GitHub-hosted runners)
- `ensemble-presets` — from `run-integration-tests.yaml` (GPU runners)
- `core-models` — from `run-integration-tests.yaml` (GPU runners)
- `stems-and-quality` — from `run-integration-tests.yaml` (GPU runners)

**IMPORTANT**: If integration test job names change (e.g., splitting or renaming jobs), you MUST update the ruleset to match. The ruleset is configured at:
`windows-cpu-integration` and `windows-directml` are intentionally **not**
required yet; they get added to the ruleset (and to the gate job's failure
conditions) after a stabilization period.

**IMPORTANT**: If integration test job names change (e.g., splitting or
renaming jobs), you MUST update the ruleset to match. The ruleset is
configured at:
https://github.com/nomadkaraoke/python-audio-separator/settings/rules/529535

To update via API:
Expand Down Expand Up @@ -92,74 +140,74 @@ EOF

**Diagnosis steps**:

1. Check if GPU runners are online:
1. Check the Cloud Function's response to the webhook — this is the most
common failure point. A 503 with `{"error": "HTTP Error 403"}` usually
means the `github-runner-pat` secret is expired/unauthorized (it's used to
mint JIT configs) — inspect the full error in the function logs to rule
out org-policy or rate-limit causes. Check recent deliveries on the org
webhook:
```bash
gh api orgs/nomadkaraoke/actions/runners \
--jq '.runners[] | select(.labels[].name == "gpu") | {name, status, busy}'
gh api orgs/nomadkaraoke/hooks # find the workflow_job webhook id
hook_id=123456 # substitute the id from above
gh api "orgs/nomadkaraoke/hooks/${hook_id}/deliveries" --paginate | head -50
```
Fix: rotate the PAT and add a new secret version:
```bash
echo -n "<new-pat>" | gcloud secrets versions add github-runner-pat \
--project=nomadkaraoke --data-file=-
```

2. Check if GPU VMs exist:
2. Check whether ephemeral VMs were actually created:
```bash
gcloud compute instances list --project=nomadkaraoke --filter="name~gpu"
gcloud compute instances list --project=nomadkaraoke \
--filter='labels.purpose="gha-ephemeral-runner"'
```

3. Check Cloud Function logs for webhook delivery:
3. Check Cloud Function logs for dispatch errors:
```bash
gcloud logging read 'resource.labels.service_name="github-runner-manager"' \
--project=nomadkaraoke --limit=20 \
--format="value(timestamp,textPayload,jsonPayload.message)"
```

4. Check GPU runner startup logs (if VMs are RUNNING but GitHub shows offline):
4. Check runner registrations GitHub-side:
```bash
gcloud compute ssh github-gpu-runner-1 --zone=us-central1-a --project=nomadkaraoke \
--command="tail -50 /var/log/github-runner-startup.log"
gh api orgs/nomadkaraoke/actions/runners \
--jq '.runners[] | {name, status, busy, labels: [.labels[].name]}'
```

### GPU VMs don't exist

If `gcloud compute instances list` shows no GPU runners but Pulumi state thinks they exist:

```bash
# 1. Remove stale state (from karaoke-gen/infrastructure/ dir)
pulumi state delete "urn:pulumi:prod::karaoke-gen-infrastructure::gcp:compute/instance:Instance::github-gpu-runner-1" --target-dependents --yes
pulumi state delete "urn:pulumi:prod::karaoke-gen-infrastructure::gcp:compute/instance:Instance::github-gpu-runner-2" --target-dependents --yes
pulumi state delete "urn:pulumi:prod::karaoke-gen-infrastructure::gcp:compute/instance:Instance::github-gpu-runner-3" --target-dependents --yes

# 2. Recreate
pulumi up --yes

# 3. Re-import dependent resources that got removed (runner-manager function, IAM, scheduler)
# Check `pulumi preview` for what needs importing
```

### GPU runner startup fails (NVIDIA driver issues)
### VM boots but the job never starts

The startup script handles kernel header mismatches by upgrading the kernel and rebooting once. If the runner still fails:
The orphan-cleanup pass logs the VM's serial console before deleting a VM
that never registered — check the Cloud Function logs (above). To look at a
live VM yourself:

```bash
# SSH in and check
gcloud compute ssh github-gpu-runner-1 --zone=us-central1-a --project=nomadkaraoke \
--command="nvidia-smi; dkms status; uname -r"
# Use the zone the VM was actually created in (us-central1-a primary,
# us-east4-c on stockout fallback — check `gcloud compute instances list`).
gcloud compute instances get-serial-port-output <vm-name> \
--zone=us-central1-a --project=nomadkaraoke --port=1 | tail -100
```

See `karaoke-gen` memory file `project_gpu_runner_drivers.md` for known issues.
### Image problems (driver failures, missing models)

### Webhook not firing
Images are rebuilt monthly (and on demand) by the `Build GHA Runner Images`
workflow in karaoke-gen. To rebuild just the GPU image:

Check the org-level webhook configuration:
```bash
gh api orgs/nomadkaraoke/hooks \
--jq '.[] | select(.events[] == "workflow_job") | {id, active, config: {url: .config.url}}'
gh workflow run build-runner-images.yml --repo nomadkaraoke/karaoke-gen \
-f variants=gpu
```

The webhook URL should point to: `https://us-central1-nomadkaraoke.cloudfunctions.net/github-runner-manager`
Known NVIDIA driver issues (kernel-header mismatches, DKMS failures) are
handled by the provisioning script in
`karaoke-gen/infrastructure/scripts/runner-image-provision.sh`.

## Cost

| Scenario | Cost |
|----------|------|
| Per GPU runner hour | ~$0.54/hr (n1-standard-4 + T4) |
| 3 runners × 15 min CI run | ~$0.41 |
| Idle (scale to zero) | $0 |
| Typical daily cost (5 PRs) | ~$2 |
| Per Linux GPU VM-hour | ~$0.54/hr (n1-standard-4 + T4, on-demand) |
| 3 parallel jobs × ~10 min | ~$0.27 per CI run |
| Per Windows GPU VM-hour (planned) | ~$0.72/hr (adds Windows Server license) |
| Idle | $0 (no VMs exist between jobs) |
Loading
Loading