Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
d67d52f
Add manual ARM64 and Windows platform smoke tests
danielmawhirter Oct 21, 2026
7ec9a9b
Enable manual PR76 platform qualification via trusted copy
danielmawhirter Oct 21, 2026
da6fab2
Run native Windows CPU smoke on a hosted runner
danielmawhirter Oct 21, 2026
8be0496
Resolve PyTorch dependencies from their appropriate indexes
danielmawhirter Oct 21, 2026
2f9f415
Expose synthetic ingest failures and cache native Rust dependencies
danielmawhirter Oct 21, 2026
b88aff6
Use label routing and existing T4 access for WSL smoke
danielmawhirter Oct 21, 2026
f16cd1e
Record successful WSL CUDA smoke qualification
danielmawhirter Oct 21, 2026
b5315f2
Record passing native Windows CPU smoke
danielmawhirter Oct 21, 2026
716918f
Record repeat WSL pass on qualified Windows revision
danielmawhirter Oct 21, 2026
38bf605
Run platform qualification from the upstream development branch
danielmawhirter Oct 21, 2026
72dd970
Exercise Windows path validation and document Faiss platforms
danielmawhirter Oct 22, 2026
ac75d57
Qualify installed platform wheels with CPU training
danielmawhirter Oct 30, 2026
ada0791
Document the existing native wheel CPU baseline
danielmawhirter Oct 30, 2026
bb4a7d2
Document the Windows Visual C++ runtime prerequisite
danielmawhirter Oct 30, 2026
0d43e01
Limit Windows release support to WSL
danielmawhirter Oct 30, 2026
62beb3b
Gate ARM64 release artifacts on CPU qualification
danielmawhirter Oct 30, 2026
8f54717
Check saved model reload in platform wheel smoke
danielmawhirter Nov 2, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 105 additions & 0 deletions .github/workflows/kumo-engine-platform-smoke.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.
# All rights reserved.
# SPDX-License-Identifier: Apache-2.0

name: Kumo Engine - Platform wheel CPU qualification

on: # yamllint disable-line rule:truthy
workflow_call:

permissions:
contents: read

jobs:
smoke:
name: CPU wheel (${{ matrix.target.platform }}, Python ${{ matrix.python }})
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
python: ['3.11', '3.12', '3.13', '3.14']
target:
- platform: windows-native
runner: windows-2022
artifact: windows-x86_64
- platform: windows-wsl
runner: wsl-amd64-gpu-t4-latest-1
artifact: x86_64
- platform: linux-arm64
runner: ubuntu-24.04-arm
artifact: aarch64
runs-on: ${{ matrix.target.runner }}
defaults:
run:
shell: bash
env:
PYTHONUNBUFFERED: '1'
PYTHONPATH: ''
PYTEST_DISABLE_PLUGIN_AUTOLOAD: '1'
CUDA_VISIBLE_DEVICES: ''
UV_LINK_MODE: copy
UV_NO_CONFIG: 'true'
SMOKE_PLATFORM: ${{ matrix.target.platform }}
SMOKE_PYTHON: ${{ matrix.python }}
CANDIDATE_SHA: ${{ github.sha }}
steps:
- name: Checkout the candidate test harness
uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6
with:
ref: ${{ github.sha }}
persist-credentials: false

- name: Download the candidate wheel from this run
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: kumo-engine-wheel-${{ matrix.target.artifact }}-${{ github.sha }}
path: ${{ runner.temp }}/platform-wheel/artifacts

- name: Set up uv
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
with:
enable-cache: false

- name: Clean install and qualify the finished wheel
run: |
set -euo pipefail
unset PYTHONPATH
stage="$RUNNER_TEMP/platform-wheel"
if [ "$RUNNER_OS" = Windows ]; then
stage="$(cygpath -u "$stage")"
fi
mkdir -p "$stage/results" "$stage/test"
cp packages/kumo-engine/test/smoke_platform.py "$stage/test/"
cp packages/kumo-engine/scripts/verify_installed_wheel.py "$stage/test/"
cd "$stage"
wheels=(artifacts/kumo_engine-*.whl)
if [ "${#wheels[@]}" -ne 1 ] || [ ! -f "${wheels[0]}" ]; then
echo 'Expected exactly one candidate engine wheel.' >&2
exit 1
fi
uv venv --python "$SMOKE_PYTHON" --seed venv
if [ "$RUNNER_OS" = Windows ]; then
python="$stage/venv/Scripts/python.exe"
else
python="$stage/venv/bin/python"
fi
uv pip install --python "$python" --torch-backend=cpu \
"${wheels[0]}" pytest 2>&1 | tee results/install.log
uv pip check --python "$python" 2>&1 | tee results/dependencies.log
"$python" test/verify_installed_wheel.py \
--wheel "${wheels[0]}" --platform "$SMOKE_PLATFORM" \
--python-version "$SMOKE_PYTHON" --commit "$CANDIDATE_SHA" \
--output results/environment.json
cd test
"$python" -m pytest -q -s --confcutdir=. \
--junitxml=../results/junit.xml smoke_platform.py \
2>&1 | tee ../results/smoke.log

- name: Save platform qualification evidence
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: engine-wheel-cpu-${{ matrix.target.platform }}-py${{ matrix.python }}-${{ github.sha }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/platform-wheel/results
if-no-files-found: warn
retention-days: 14
14 changes: 8 additions & 6 deletions .github/workflows/kumo-engine-wheel.yml
Original file line number Diff line number Diff line change
Expand Up @@ -239,17 +239,20 @@ jobs:
with:
wheel_artifact: kumo-engine-wheel-x86_64-${{ github.sha }}

platform-qualification:
needs: [build]
uses: ./.github/workflows/kumo-engine-platform-smoke.yml

attach-to-release:
needs: [build, check-requirements, wheel-smoke]
needs: [build, check-requirements, wheel-smoke, platform-qualification]
if: ${{ github.event_name == 'release' }}
strategy:
fail-fast: false
matrix:
include:
- arch: x86_64
- arch: arm64
# Enabled by dependent qualification PR #92 with its runtime gate.
# - arch: aarch64
- arch: aarch64
# Native Windows remains experimental; Windows releases use WSL.
# - arch: windows-x86_64
runs-on: ubuntu-24.04
Expand Down Expand Up @@ -279,9 +282,8 @@ jobs:
# Native Windows remains experimental; Windows releases use WSL.
# - arch: windows-x86_64
# platform: win_amd64
# Enabled by dependent qualification PR #92 with its runtime gate.
# - arch: aarch64
# platform: manylinux_2_34_aarch64
- arch: aarch64
platform: manylinux_2_34_aarch64
- arch: x86_64
platform: manylinux_2_34_x86_64
- arch: arm64
Expand Down
61 changes: 42 additions & 19 deletions docs/development/releasing-kumo-engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@ Faiss is selected at install time: `faiss-gpu==1.14.3` on Linux x86-64 and
`faiss-cpu==1.14.3` on macOS. These markers live on the engine's direct
dependencies so merging bundled packages preserves them. This dependency
selection does not establish wheel or runtime support. Faiss is optional on
Windows and Linux ARM64; the `cpu` extra opts into CPU Faiss there.
Linux ARM64 and experimental native Windows builds; the `cpu` extra opts into
CPU Faiss there. Windows support for this release is WSL2 only.

Releases are cut from `NVIDIA/kumo-engine`: its GitHub releases trigger the
workflows below, which attach the wheels to the release. Their `kitmaker` job
Expand All @@ -48,10 +49,10 @@ The build matrix targets `cp311-abi3-manylinux_2_34_x86_64` and
WSL uses the Linux x86-64 wheel. All require CPython >= 3.11
(DiskGraph's stable-ABI floor), with classifiers for 3.11–3.14. Adding build
targets does not establish published availability or installed-wheel runtime
qualification. Release attachment and publication currently allow only Linux
x86-64 and macOS ARM64. Linux ARM64 remains a CI artifact until installed-wheel
qualification is integrated; native Windows remains experimental and CI-only.
Windows release support uses WSL with the Linux x86-64 wheel. The merger reads
qualification. Native Windows builds and tests are experimental: their wheels
remain CI artifacts and are excluded from public release attachments and
Kitmaker publication. The supported Windows route for this release is WSL2.
The merger reads
every bundled wheel's `Requires-Python` and writes their intersection into the
final wheel.
Stricter minimums, upper bounds, and exclusions are retained; incompatible
Expand Down Expand Up @@ -124,17 +125,18 @@ platform's `release-version`.
Labels and ranks can change only when their scores justify the change
within that bound. The macOS wheel has no GPU smoke
check. See the [test criteria](testing.md#finished-kumo-engine-wheel).
- attaches only the Linux x86-64 and macOS ARM64 wheels to the GitHub
release; ARM64 Linux and experimental native Windows builds stay in CI.
- attaches the Linux x86_64/ARM64 and macOS ARM64 wheels to the GitHub
release; the Linux x86_64 wheel also serves WSL2. Native Windows upload
routes remain disabled despite experimental build/test coverage.
5. **Publish through Kitmaker.** The `kitmaker` job runs in the
`kumo-engine-release` environment and waits for approval from
`kumo-engineers`. The person who starts the run cannot approve it.
It checks that the Portal project
has the expected name and owner and that the attached wheel is the one this
run built, has Kitmaker dry-run it, and only then publishes it to PyPI and
pypi.nvidia.com, where it confirms both serve exactly those bytes. All enabled
platform wheels are published. A publication cannot be recalled, and PyPI never
accepts the same file name twice.
pypi.nvidia.com, where it confirms both serve exactly those bytes. All
enabled platform wheels are published. A publication cannot be recalled,
and PyPI never accepts the same file name twice.
6. **Smoke-test the published wheel** in a clean virtualenv:

```bash
Expand Down Expand Up @@ -169,6 +171,14 @@ as a workflow artifact for 30 days. It never
attaches or publishes anything: attaching runs only for release events, and PyPI
rejects local versions anyway.

## Native CPU baseline

DiskGraph builds retain the fixed CPU baseline in `.cargo/config.toml` and
the native build backend: AES plus SSE2 on x86-64; AES plus NEON on ARM64.
There is no `target-cpu=native` setting or AVX2 requirement. Qualification
runners and deployment CPUs must expose that baseline, including through
virtualization.

## Build locally

```bash
Expand All @@ -189,11 +199,23 @@ and takes the final platform tag from the built DiskGraph wheel. Rust, Xcode
Command Line Tools, and CMake are required. The release workflow builds this
wheel on a macOS ARM64 runner with the same deployment target.

For experimental native Windows builds, run the builder in Git Bash with
native Windows Python, the Rust MSVC toolchain, CMake, and an activated
Visual Studio x64 C++/Windows SDK environment. Set `PYTHON=python` if `python3`
is not available. Zig and ELF/glibc checks are disabled; the builder requires
DiskGraph's `win_amd64`
### Experimental native Windows builds

These instructions are for development and CI qualification, not a supported
installation route for this release. Use WSL2 and the Linux x86-64 wheel on
Windows. Native Windows release uploads are disabled.

Experimental native Windows x64 installations require a Microsoft Visual C++ v14 runtime
at least as recent as the build tools used for the wheel. If the runtime is
missing or too old, install the
[latest supported x64 Redistributable](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist).
This runtime prerequisite is separate from the MSVC build tools required to
build from source; the engine wheel does not bundle the redistributable.

On native Windows, run the builder in Git Bash with native Windows Python,
the Rust MSVC toolchain, CMake, and an activated Visual Studio x64 C++/Windows
SDK environment. Set `PYTHON=python` if `python3` is not available. Zig and
ELF/glibc checks are disabled; the builder requires DiskGraph's `win_amd64`
tag. `--verify-image` is not supported for Windows: qualify the installed
wheel natively. The workflow activates the image's existing MSVC tools;
it does not install system toolchains.
Expand All @@ -203,12 +225,13 @@ hashes verify exact source bytes. For a fresh local clone, use
only to the checkout action's Git processes, without changing repository or
user Git configuration files or weakening the integrity checks.

### Linux ARM64 builds

Linux ARM64 uses the same Zig build and glibc checks as Linux x86-64,
with `--platform-tag manylinux_2_34_aarch64`. The workflow validates the
assembled native payload before retaining it as a CI artifact. Clean
installed-wheel CPU ingestion, training, and prediction are a separate
qualification requirement; GPU execution is not required for these platform
wheels.
assembled native payload before uploading it. Clean installed-wheel CPU
ingestion, training, and prediction are a separate qualification requirement;
GPU execution is not required for these platform wheels.

## PyPI projects

Expand Down
15 changes: 12 additions & 3 deletions packages/kumo-engine/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,21 @@ Kumo Engine (`kumo.engine`) trains graph neural networks on relational data, in-

## Requirements

- Linux x86_64 with glibc 2.34 or newer, or macOS 14 or newer on Apple silicon
- Linux x86_64 or ARM64 with glibc 2.34 or newer, or macOS 14 or newer on Apple silicon
- On Windows, use WSL2 x86_64 with a Linux distribution meeting that glibc requirement
- CPython 3.11 or newer
- For production workloads, an NVIDIA GPU with a compatible driver; see [prerequisites](https://docs.nvidia.com/sdgm/kumo-engine/get-started-with-kumo-engine/prerequisites) and [GPU sizing](https://docs.nvidia.com/sdgm/kumo-engine/get-started-with-kumo-engine/gpu-sizing)

Kumo Engine uses a GPU when one is available: an NVIDIA GPU through CUDA, or an Apple silicon GPU through PyTorch's MPS backend. Without one, training and prediction run on CPU, which is enough for small jobs such as the quickstart below. To choose the device, pass `accelerator=` (`"auto"`, `"cpu"`, `"gpu"`, or `"mps"`) to `Trainer`.

The bundled native engine requires CPU AES instructions: AES and SSE2 on
x86-64, or AES and NEON on ARM64. Check that your host or virtual machine
exposes these features before installing a wheel.

Windows support for this release is limited to WSL2 using the Linux x86_64
wheel. Run installation and Python commands inside WSL. Native Windows is not
supported, and native Windows wheels are not published in this release.

## Installation

```bash
Expand All @@ -25,8 +34,8 @@ pip install kumo-engine

Linux x86_64 requires GPU Faiss, which also handles CPU retrieval. macOS keeps
CPU Faiss and its existing Metal retrieval support. Faiss is optional on ARM64
Linux and native Windows: without it, affected link-prediction retrieval raises
an installation error while other tasks remain available. On those platforms,
Linux: without it, affected link-prediction retrieval raises an installation
error while other tasks remain available. On ARM64 Linux,
use `pip install "kumo-engine[cpu]"` for CPU retrieval with a compatible backend
wheel for your platform and Python version. For CUDA retrieval, install a
compatible GPU Faiss build directly, building from source on ARM64 Linux if
Expand Down
60 changes: 60 additions & 0 deletions packages/kumo-engine/platform-smoke.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Installed-wheel platform qualification

The manual [wheel build workflow](../../.github/workflows/kumo-engine-wheel.yml)
builds the candidate once and passes those artifacts to the reusable
[platform smoke workflow](../../.github/workflows/kumo-engine-platform-smoke.yml).
Select **Kumo Engine - Build and publish wheel**, **Run workflow**, and the
candidate branch. Manual runs upload artifacts; they do not publish a release.

Qualification covers Python 3.11, 3.12, 3.13 and 3.14 on native Windows x86-64,
Windows through WSL x86-64, and Linux ARM64: twelve independent CPU jobs.
Native Windows jobs provide experimental build and runtime evidence only.
For this release, Windows support is limited to WSL2; native Windows wheels
are retained as CI artifacts but excluded from public release attachments and
publication. A passing native job does not establish supported native Windows
distribution.
WSL installs the same Linux x86-64 wheel as ordinary Linux; it is not a
separate wheel format. GPU availability is not a qualification requirement.
The producer also retains the macOS ARM64 artifact and its existing checks;
this smoke does not add macOS runtime qualification.

Each consumer creates a fresh virtual environment, installs the downloaded
wheel and public CPU Torch dependencies, and runs dependency consistency
checks. It copies only the smoke test and standalone environment auditor
outside the checkout. The auditor rejects checkout imports, records origins
for engine/core/API/PQL/DiskGraph, verifies the packaged ingestion executable,
and records the wheel filename, SHA-256, candidate revision, Python, platform,
and Torch version. Native Windows and Linux ARM64 must run without Faiss;
WSL retains the wheel's existing Linux x86-64 Faiss dependency.

[smoke_platform.py](test/smoke_platform.py) creates a synthetic Parquet table,
performs real snapshot ingestion, and selects CPU through the public
`Trainer(..., accelerator='cpu')` API. Acceptance requires four optimizer
steps with finite CPU gradients, finite training metrics, and all 100 expected
finite predictions. It reloads the same fit's saved bundle with the public
`Trainer.load(..., accelerator='cpu')` API and predicts the same population and
snapshot. Reloaded entity IDs and counts must match, and prediction values must
match within `atol=1e-4` and NumPy's default relative tolerance. The optimizer
hook stays active throughout, requiring no additional training steps during
either prediction. Synchronous DataLoader execution limits Windows coverage
to zero worker processes; Unix worker pools and native Windows GPU execution
are outside this qualification. Subprocess diagnostics are printed only for
this generated fixture, preserving production error privacy.

The WSL job checks the Linux guest kernel. A Windows host shell or ordinary
Linux x86-64 runner cannot substitute for WSL. Consumer jobs install no Rust
compiler and rebuild no native artifacts: missing executables or libraries
must fail the downloaded-wheel check. Logs, environment evidence and pytest
results distinguish packaging failures from training failures.

The [installed-wheel qualification run](https://github.com/NVIDIA/kumo-engine/actions/runs/41101558418)
at `8f6c496f` passed all twelve jobs, each with four CPU optimizer steps and
100 finite predictions. Windows native and Linux ARM64 passed without Faiss.
This establishes runtime evidence for those artifacts, including experimental
native Windows coverage; it does not change the WSL-only Windows release
boundary or publish a package. Later runtime or packaging changes require
qualification of their candidate artifacts.

PR checks remain a later decision after manual qualification is reliable.
This change does not require an ARM GPU runner grant or change publication
approval policy.
Loading
Loading