Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/kitmaker-portal.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,10 @@ name: Kitmaker Portal via Charon
on: # yamllint disable-line rule:truthy
workflow_call:
secrets:
# Environment secrets are resolved by the protected portal job below;
# callers cannot read them before that job enters its environment.
KITMAKER_API_TOKEN:
required: true
required: false
inputs:
project:
type: string
Expand Down
10 changes: 4 additions & 6 deletions .github/workflows/kumo-engine-wheel.yml
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ jobs:
grep -Eq '^test result: ok\. [1-9][0-9]* passed;' snapshot-mirror-tests.log

# GitHub-hosted artifact storage, not S3: no AWS role to provision.
# Native Windows wheels are experimental internal CI artifacts only.
# Release jobs publish only after this candidate passes qualification.
- name: Upload wheel
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7
with:
Expand Down Expand Up @@ -296,8 +296,7 @@ jobs:
- arch: x86_64
- arch: arm64
- arch: aarch64
# Native Windows remains experimental; Windows releases use WSL.
# - arch: windows-x86_64
- arch: windows-x86_64
runs-on: ubuntu-24.04
permissions:
contents: write
Expand Down Expand Up @@ -339,9 +338,8 @@ jobs:
fail-fast: false
matrix:
include:
# Native Windows remains experimental; Windows releases use WSL.
# - arch: windows-x86_64
# platform: win_amd64
- arch: windows-x86_64
platform: win_amd64
- arch: aarch64
platform: manylinux_2_34_aarch64
- arch: x86_64
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
- **Training data from one query:** A predictive query generates labeled training examples at points in each entity's history. Each example sees only records timestamped at or before its point in time, with no hand-written label logic or point-in-time joins.
- **Trains directly on relational data:** A built-in graph neural network (GNN) learns from the related records without requiring hand-built features. It handles classification, regression, forecasting, and recommendation tasks.
- **End-to-end workflow:** Hyperparameter search, explanations, embedding export, and batch prediction are built in. Kumo Engine also works with MLflow, Databricks, SageMaker, and Snowflake to log each run, version models so scheduled prediction jobs use the one you promote, and monitor prediction drift.
- **Runs locally or in your data platform:** Run Kumo Engine in Python on Linux, macOS, or Windows (WSL2), or in a Snowflake or Databricks notebook. Read data from, and write predictions to, local files, cloud storage, and data warehouses.
- **Runs locally or in your data platform:** Run Kumo Engine in Python on Linux, macOS, or Windows (native CPU or WSL2 for NVIDIA GPUs), or in a Snowflake or Databricks notebook. Read data from, and write predictions to, local files, cloud storage, and data warehouses.
- **Works with coding agents:** The bundled agent skill teaches your coding agent the Kumo Engine API, so it can build, train, and run models on your data.

## Part of NVIDIA Kumo
Expand All @@ -55,7 +55,7 @@ pip install kumo-engine
| Requirement | Details |
| --- | --- |
| Python | CPython 3.11 or later |
| Operating system | Linux x86-64 or ARM64 with glibc 2.34 or later, macOS 14 or later on Apple silicon, or Windows through WSL2 (x86-64) |
| Operating system | Linux x86-64 or ARM64 with glibc 2.34 or later, macOS 14 or later on Apple silicon, or Windows x86-64 on CPU. Windows through WSL2 also supports NVIDIA GPUs. |
| CPU | AES and SSE2 instructions on x86-64, or AES and NEON on ARM64 |
| GPU | An NVIDIA GPU for production workloads. The quickstart and other small datasets also run on a CPU or an Apple silicon GPU. The Apple silicon GPU needs macOS 15 or later. On macOS 14, pass `accelerator="cpu"` to `Trainer`, `Trainer.load` and `Trainer.load_from_connector`. |
| Storage | A local NVMe SSD for production workloads. Network-attached volumes, such as Amazon EBS, make sampling much slower. |
Expand All @@ -78,7 +78,7 @@ Choose your hardware and runtime environment:

- **Hardware:** [Prerequisites](https://docs.nvidia.com/sdgm/kumo-engine/get-started-with-kumo-engine/prerequisites) · [GPU sizing](https://docs.nvidia.com/sdgm/kumo-engine/get-started-with-kumo-engine/gpu-sizing)
- **Setup guides:** [Linux](https://docs.nvidia.com/sdgm/kumo-engine/get-started-with-kumo-engine/installation/linux-machine-deployment) · [macOS](https://docs.nvidia.com/sdgm/kumo-engine/get-started-with-kumo-engine/installation/macos) · [Windows (WSL2)](https://docs.nvidia.com/sdgm/kumo-engine/get-started-with-kumo-engine/installation/windows-wsl2) · [Snowflake notebook](https://docs.nvidia.com/sdgm/kumo-engine/get-started-with-kumo-engine/installation/snowflake-notebook) · [Databricks notebook](https://docs.nvidia.com/sdgm/kumo-engine/get-started-with-kumo-engine/installation/databricks-notebook)
- **Platform details:** the [package README](packages/kumo-engine/README.md#installation) covers ARM64 Linux setup, including Faiss for recommendation tasks.
- **Platform details:** the [package README](packages/kumo-engine/README.md#installation) covers native Windows CPU installation with `kumo-engine[cpu]` and ARM64 Linux setup, including Faiss for recommendation tasks. Native Windows CUDA is deferred.

## Quickstart

Expand Down
34 changes: 19 additions & 15 deletions docs/development/releasing-kumo-engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,11 @@ Faiss is selected at install time: `faiss-gpu==1.14.3` on Linux x86-64 and
`faiss-cpu==1.14.3` on macOS. These markers live on the engine's direct
dependencies so merging bundled packages preserves them. This dependency
selection does not establish wheel or runtime support. Faiss is optional on
Linux ARM64 and experimental native Windows builds; the `cpu` extra opts into
CPU Faiss there. Windows support for this release is WSL2 only.
Linux ARM64 and native Windows builds; the `cpu` extra opts into CPU Faiss
there. Native Windows x86-64 supports CPU execution, including recommendation
retrieval with `pip install "kumo-engine[cpu]"`. Select `accelerator="cpu"`
when training and loading models. Native Windows CUDA is deferred; WSL2
retains the Linux x86-64 GPU route.

Releases are cut from `NVIDIA/kumo-engine`: its GitHub releases trigger the
workflows below, which attach the wheels to the release. Their `kitmaker` job
Expand All @@ -49,13 +52,12 @@ The build matrix targets `cp311-abi3-manylinux_2_34_x86_64` and
WSL uses the Linux x86-64 wheel. All require CPython >= 3.11
(DiskGraph's stable-ABI floor), with classifiers for 3.11–3.14. Adding build
targets does not establish published availability or installed-wheel runtime
qualification. Native Windows builds and tests are experimental: their wheels
remain CI artifacts and are excluded from public release attachments and
Kitmaker publication. The supported Windows route for this release is WSL2.
qualification. Native Windows wheels are attached and published alongside the
Linux and macOS wheels only after the candidate passes the required checks.
Every wheel build also qualifies the native Windows CPU wheel on Python
3.11–3.14 in fresh producer and consumer processes, with the `cpu` extra for
Faiss retrieval. This check is a prerequisite for release attachment; it does
not enable Windows publication. It exports and restores both models and
Faiss retrieval. This check is a prerequisite for release attachment and
Kitmaker publication. It exports and restores both models and
snapshots after deleting the producer's temporary storage, and checks ordinary
loader defaults without CI or synchronous-loader overrides. Results are saved
in `engine-wheel-reload-windows-native-cpu-*` artifacts. Native Windows CUDA
Expand Down Expand Up @@ -134,9 +136,8 @@ platform's `release-version`.
Labels and ranks can change only when their scores justify the change
within that bound. The macOS wheel has no GPU smoke
check. See the [test criteria](testing.md#finished-kumo-engine-wheel).
- attaches the Linux x86_64/ARM64 and macOS ARM64 wheels to the GitHub
release; the Linux x86_64 wheel also serves WSL2. Native Windows upload
routes remain disabled despite experimental build/test coverage.
- attaches the Linux x86_64/ARM64, macOS ARM64 and native Windows x86_64
wheels to the GitHub release; the Linux x86_64 wheel also serves WSL2.

Before you approve the `kitmaker` job, start
[`Kumo Engine - Desktop reload qualification`][desktop-reload] by hand with
Expand Down Expand Up @@ -168,6 +169,8 @@ platform's `release-version`.
5. **Publish through Kitmaker.** The `kitmaker` job runs in the
`kumo-engine-release` environment and waits for approval from
`kumo-engineers`. The person who starts the run cannot approve it.
The Portal job reads `KITMAKER_API_TOKEN` from that environment after
approval; reusable-workflow callers do not need access to the token.
It checks that the Portal project
has the expected name and owner and that the attached wheel is the one this
run built, has Kitmaker dry-run it, and only then publishes it to PyPI and
Expand Down Expand Up @@ -237,13 +240,14 @@ and takes the final platform tag from the built DiskGraph wheel. Rust, Xcode
Command Line Tools, and CMake are required. The release workflow builds this
wheel on a macOS ARM64 runner with the same deployment target.

### Experimental native Windows builds
### Native Windows builds

These instructions are for development and CI qualification, not a supported
installation route for this release. Use WSL2 and the Linux x86-64 wheel on
Windows. Native Windows release uploads are disabled.
Native Windows x86-64 supports CPU execution. Install the published wheel with
`pip install "kumo-engine[cpu]"` for CPU Faiss retrieval, and select
`accelerator="cpu"` when training and loading models. Use WSL2 and the Linux
x86-64 wheel for NVIDIA GPU workloads; native Windows CUDA is deferred.

Experimental native Windows x64 installations require a Microsoft Visual C++ v14 runtime
Native Windows x64 installations require a Microsoft Visual C++ v14 runtime
at least as recent as the build tools used for the wheel. If the runtime is
missing or too old, install the
[latest supported x64 Redistributable](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist).
Expand Down
7 changes: 5 additions & 2 deletions packages/kumo-engine/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ pip install kumo-engine
| Requirement | Details |
| --- | --- |
| Python | CPython 3.11 or later |
| Operating system | Linux x86-64 or ARM64 with glibc 2.34 or later, macOS 14 or later on Apple silicon, or Windows through WSL2 (x86-64) |
| Operating system | Linux x86-64 or ARM64 with glibc 2.34 or later, macOS 14 or later on Apple silicon, or Windows x86-64 on CPU. Windows through WSL2 also supports NVIDIA GPUs. |
| CPU | AES and SSE2 instructions on x86-64, or AES and NEON on ARM64 |
| GPU | An NVIDIA GPU for production workloads. The quickstart and other small datasets also run on a CPU or an Apple silicon GPU. The Apple silicon GPU needs macOS 15 or later. On macOS 14, pass `accelerator="cpu"` to `Trainer`, `Trainer.load` and `Trainer.load_from_connector`. |
| Storage | A local NVMe SSD for production workloads. Network-attached volumes, such as Amazon EBS, make sampling much slower. |
Expand All @@ -27,6 +27,7 @@ The base package reads and writes local CSV and Parquet files. Optional extras a

| Extra | Adds |
| --- | --- |
| `cpu` | CPU Faiss for recommendation retrieval on native Windows and Linux ARM64 |
| `s3` | Amazon S3 |
| `gcs` | Google Cloud Storage |
| `adls` | Azure Data Lake Storage Gen2 |
Expand All @@ -39,7 +40,9 @@ The base package reads and writes local CSV and Parquet files. Optional extras a

Kumo Engine trains on an NVIDIA GPU or an Apple silicon GPU when one is available, and on the CPU otherwise. To choose the device, pass `accelerator="gpu"`, `"mps"`, or `"cpu"` to `Trainer`.

Recommendation tasks use Faiss for retrieval. On Linux x86-64 and macOS, `pip install kumo-engine` installs Faiss as a dependency. On Linux ARM64, Faiss is optional, and other tasks run without it. To run recommendation tasks there, install CPU Faiss with `pip install "kumo-engine[cpu]"` and pass `accelerator="cpu"` to `Trainer`, or follow the [DGX Spark guide](https://github.com/NVIDIA/kumo-engine/blob/main/packages/kumo-engine/dgx-spark.md) to build GPU Faiss. The guide also covers CUDA training on ARM64.
On native Windows, install `pip install "kumo-engine[cpu]"` and pass `accelerator="cpu"` to `Trainer`, `Trainer.load` and `Trainer.load_from_connector`. Native Windows CUDA support is deferred; use WSL2 for NVIDIA GPU workloads.

Recommendation tasks use Faiss for retrieval. On Linux x86-64 and macOS, `pip install kumo-engine` installs Faiss as a dependency. On native Windows and Linux ARM64, Faiss is optional, and other tasks run without it. To run recommendation tasks there, install CPU Faiss with `pip install "kumo-engine[cpu]"` and pass `accelerator="cpu"` to `Trainer`. For GPU Faiss on Linux ARM64, follow the [DGX Spark guide](https://github.com/NVIDIA/kumo-engine/blob/main/packages/kumo-engine/dgx-spark.md), which also covers CUDA training on ARM64.

On NVIDIA GPUs, the optional `torch-scatter` package speeds up some training workloads. Install the build for your PyTorch and CUDA versions as described in [PyG's installation guide](https://pytorch-geometric.readthedocs.io/en/latest/install/installation.html).

Expand Down
39 changes: 24 additions & 15 deletions packages/kumo-engine/platform-smoke.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,11 @@ Select **Kumo Engine - Build and publish wheel**, **Run workflow**, and the
candidate branch. Manual runs upload artifacts; they do not publish a release.

Qualification covers Python 3.11, 3.12, 3.13 and 3.14 on native Windows x86-64,
Windows through WSL x86-64, and Linux ARM64: twelve independent CPU jobs.
Native Windows jobs provide experimental build and runtime evidence only.
For this release, Windows support is limited to WSL2; native Windows wheels
are retained as CI artifacts but excluded from public release attachments and
publication. A passing native job does not establish supported native Windows
distribution.
Windows through WSL x86-64, and Linux ARM64: twelve synthetic CPU smoke jobs
plus four additional native Windows CPU reload jobs.
Native Windows x86-64 CPU wheels participate in release attachment and
publication after the required candidate checks pass. Native Windows CUDA
support is deferred; WSL2 retains the Linux x86-64 GPU route.
WSL installs the same Linux x86-64 wheel as ordinary Linux; it is not a
separate wheel format. GPU availability is not a qualification requirement.
The producer also retains the macOS ARM64 artifact and its existing checks;
Expand All @@ -24,8 +23,8 @@ checks. It copies only the smoke test and standalone environment auditor
outside the checkout. The auditor rejects checkout imports, records origins
for engine/core/API/PQL/DiskGraph, verifies the packaged ingestion executable,
and records the wheel filename, SHA-256, candidate revision, Python, platform,
and Torch version. Native Windows and Linux ARM64 must run without Faiss;
WSL retains the wheel's existing Linux x86-64 Faiss dependency.
and Torch version. Native Windows and Linux ARM64 synthetic smoke jobs run
without Faiss; WSL retains the wheel's existing Linux x86-64 Faiss dependency.

[smoke_platform.py](test/smoke_platform.py) creates a synthetic Parquet table,
performs real snapshot ingestion, and selects CPU through the public
Expand All @@ -36,11 +35,21 @@ finite predictions. It reloads the same fit's saved bundle with the public
snapshot. Reloaded entity IDs and counts must match, and prediction values must
match within `atol=1e-4` and NumPy's default relative tolerance. The optimizer
hook stays active throughout, requiring no additional training steps during
either prediction. Synchronous DataLoader execution limits Windows coverage
to zero worker processes; Unix worker pools and native Windows GPU execution
are outside this qualification. Subprocess diagnostics are printed only for
either prediction. This synthetic smoke uses synchronous DataLoader execution.
Subprocess diagnostics are printed only for
this generated fixture, preserving production error privacy.

The four additional native Windows jobs install the candidate with `[cpu]` and
run
[desktop_reload_qualification.py](test/desktop_reload_qualification.py) on CPU.
Its binary classification and static-link tasks train, export models and
snapshots, terminate the producer, delete its temporary storage, and load and
predict in fresh consumer processes. Static-link retrieval must use CPU Faiss.
The gate checks ordinary loader defaults with CI and synchronous-loader
overrides removed. Evidence includes the candidate revision and wheel digest
in `engine-wheel-reload-windows-native-cpu-*` artifacts. Native CUDA and
restricted-account execution remain outside this qualification.

The WSL job checks the Linux guest kernel. A Windows host shell or ordinary
Linux x86-64 runner cannot substitute for WSL. Consumer jobs install no Rust
compiler and rebuild no native artifacts: missing executables or libraries
Expand All @@ -50,10 +59,10 @@ results distinguish packaging failures from training failures.
The [installed-wheel qualification run](https://github.com/NVIDIA/kumo-engine/actions/runs/41101558418)
at `8f6c496f` passed all twelve jobs, each with four CPU optimizer steps and
100 finite predictions. Windows native and Linux ARM64 passed without Faiss.
This establishes runtime evidence for those artifacts, including experimental
native Windows coverage; it does not change the WSL-only Windows release
boundary or publish a package. Later runtime or packaging changes require
qualification of their candidate artifacts.
This is historical runtime evidence for those artifacts, which did not
include native Windows Faiss retrieval or snapshot export and restore.
It does not qualify later candidate artifacts or establish their published
availability. The current native Windows gate covers those paths.

PR checks remain a later decision after manual qualification is reliable.
This change does not require an ARM GPU runner grant or change publication
Expand Down
1 change: 1 addition & 0 deletions packages/kumo-engine/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ classifiers=[
"Intended Audience :: Science/Research",
"Operating System :: MacOS",
"Operating System :: POSIX :: Linux",
"Operating System :: Microsoft :: Windows",
"Programming Language :: Python",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
Expand Down
11 changes: 6 additions & 5 deletions test/scripts/test_engine_wheel_workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -181,10 +181,10 @@ def test_all_platforms_require_validation_and_release_environment() -> None:
row["arch"]
for row in JOBS["attach-to-release"]["strategy"]["matrix"]["include"]
}
assert {"x86_64", "arm64"} <= release_arches
assert release_arches <= {"x86_64", "arm64", "aarch64"}
if "aarch64" in release_arches:
assert "platform-qualification" in JOBS["attach-to-release"]["needs"]
assert release_arches == arches
assert "platform-qualification" in JOBS["attach-to-release"]["needs"]
assert next(row for row in JOBS["kitmaker"]["strategy"]["matrix"]["include"]
if row["arch"] == "windows-x86_64")["platform"] == "win_amd64"
for job in ["attach-to-release", "kitmaker"]:
published = {row["arch"] for row in JOBS[job]["strategy"]["matrix"]["include"]}
assert published == release_arches
Expand Down Expand Up @@ -221,8 +221,9 @@ def test_all_platforms_require_validation_and_release_environment() -> None:
PORTAL_WORKFLOW[True]["workflow_call"]["secrets"]["KITMAKER_API_TOKEN"][
"required"
]
is True
is False
)
assert portal["env"]["KITMAKER_API_TOKEN"] == "${{ secrets.KITMAKER_API_TOKEN }}"
glove = yaml.safe_load(
(WORKFLOW.parent / "kumo-glove-6b-50d-wheel.yml").read_text()
)
Expand Down
Loading