Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
63f37f3
Add PyTorch nightly AML coverage and benchmark campaigns
nittin-murthi Nov 16, 2026
eedd113
Exercise nightly failure detection without decimal transport fix
nittin-murthi Nov 16, 2026
76a27da
Validate memory peaks below the cgroup namespace root
nittin-murthi Nov 16, 2026
c812c30
Support CUDA UUID objects and retain independent benchmark evidence
nittin-murthi Nov 16, 2026
0db8f90
Keep native coverage metadata in benchmark reports
nittin-murthi Nov 16, 2026
542f212
Run existing CUDA package checks on the nightly GPU runner
nittin-murthi Nov 16, 2026
f2d3f7d
Allow independent nightly candidate revisions to validate
nittin-murthi Nov 16, 2026
f27cce6
Correct CUDA metadata check and record approved AML read grant
nittin-murthi Nov 16, 2026
e8ee909
Record collective nightly fault-detection results
nittin-murthi Nov 16, 2026
57201f9
Lengthen nightly GPU sampling windows with more real batches
nittin-murthi Nov 16, 2026
57eca2b
Support explicitly labelled sampled host peaks on nightly GPU runners
nittin-murthi Nov 16, 2026
0e6bcc4
Record live GPU sampling and sampled host-memory validation
nittin-murthi Nov 16, 2026
f90e6ce
Add test-only Polars loader timing benchmarks
nittin-murthi Nov 16, 2026
0132f18
Add GPU reference to loader timing comparisons
nittin-murthi Nov 16, 2026
7357926
Preserve CUDA kernel names and validate GPU references early
nittin-murthi Nov 16, 2026
0683ead
Record Polars and live GPU reference validation status
nittin-murthi Nov 16, 2026
0e28fd0
Restore worker identity environment after cgroup provisioning
nittin-murthi Nov 16, 2026
bebf2a8
Clarify Kumo PyTorch package regression check name
nittin-murthi Nov 16, 2026
75adf93
Fail kumo-pytorch T2a and T2b checks when an import is missing
anle2024 Nov 16, 2026
de5c2b6
Check special-token IDs after a DiskGraph reload in a new process
anle2024 Nov 16, 2026
2bd2e82
Test fitting at the 4,096-anchor chunk edge
anle2024 Nov 16, 2026
e6ae2a7
Shorten the fitting chunk-edge test
anle2024 Nov 16, 2026
d7235aa
Use the lifecycle harness for the explicit-ID reload check
anle2024 Nov 16, 2026
1c617db
Add required multi-source history acceptance checks
nittin-murthi Nov 16, 2026
bdb21fc
Add full-population AML acceptance checks
nittin-murthi Nov 16, 2026
e558a41
Add fresh-process finetuning and exact mid-epoch resume checks
nittin-murthi Nov 16, 2026
c9bf93e
test: add nightly training acceptance and fitting memory scaling
nittin-murthi Nov 17, 2026
2305e3b
Merge remote-tracking branch 'origin/main' into codex/pytorch-nightly…
nittin-murthi Nov 17, 2026
9334978
Remove the docstring from the explicit-ID contract builder
anle2024 Nov 17, 2026
d753e7e
Remove the docstring and comments from the chunk-edge test
anle2024 Nov 17, 2026
d5deeb1
Merge PR #361 lifecycle import guards into nightly suite
nittin-murthi Nov 17, 2026
203912d
Merge PR #362 fresh-process special-ID tests into nightly suite
nittin-murthi Nov 17, 2026
10f7a1e
Merge PR #363 fitting chunk-boundary tests into nightly suite
nittin-murthi Nov 17, 2026
08a0e9c
Document testing PR consolidation and current acceptance gaps
nittin-murthi Nov 17, 2026
b0670a2
Test unshuffled anchor order across storage shards
nittin-murthi Nov 17, 2026
6568f52
Merge latest main into PyTorch nightly suite
nittin-murthi Nov 17, 2026
23e49c3
Complete benchmark measurements and enforce AML test coverage
nittin-murthi Nov 17, 2026
1cb4e3c
Keep new PyTorch coverage DiskGraph-only and add golden boundaries
nittin-murthi Nov 17, 2026
ba203e0
Require training acceptance for all five AML model bindings
nittin-murthi Nov 17, 2026
f2d11d1
Parallelize nightly AML campaigns and fitting repetitions
nittin-murthi Nov 17, 2026
55e3d0d
Clarify PyTorch workflow and job display names
nittin-murthi Nov 17, 2026
1b3e7ac
Migrate nightly production loaders to native AnchorTable iteration
nittin-murthi Nov 17, 2026
000f6db
Integrate PyTorch merge gate and complete nightly accounting
nittin-murthi Nov 17, 2026
41659d6
Migrate PyTorch coverage to native fixtures and AnchorTables
nittin-murthi Nov 17, 2026
0c6b312
Accept explicit preflight capability refusals in PyTorch tests
nittin-murthi Nov 18, 2026
d55cab6
Fix nightly harness and isolate length-planning coverage
nittin-murthi Nov 18, 2026
f6ae2ae
Unify dataset campaigns and add LI Combined feature coverage
nittin-murthi Nov 18, 2026
d996a7b
Extend LI Medium campaigns and refine nightly conformance
nittin-murthi Nov 18, 2026
559ce91
Unify PyTorch PR and nightly execution and reporting
nittin-murthi Nov 18, 2026
304a587
Add Synthetic Medium customer-history binding coverage
nittin-murthi Nov 18, 2026
27e4c5a
Add Synthetic Medium PRAGMA binding with BPE feature coverage
nittin-murthi Nov 18, 2026
81ed751
Add Synthetic Medium Combined binding with aggregate feature checks
nittin-murthi Nov 18, 2026
98583e4
Add MBD GraphSAGE binding with generated count checks
nittin-murthi Nov 18, 2026
f286997
Organize PyTorch campaigns by test purpose
nittin-murthi Nov 19, 2026
a83d27a
Add MBD sequence coverage and readable suite diagnostics
nittin-murthi Nov 19, 2026
46211d9
Add Streaming 25B GraphSAGE scaling binding
nittin-murthi Nov 19, 2026
e2812d5
Add difficult behavior coverage and readable failure summaries
nittin-murthi Nov 19, 2026
5b1c9f2
Include difficult behavior in the required full suite
nittin-murthi Nov 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
35 changes: 35 additions & 0 deletions .github/actions/kumo-pytorch-results/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.
# All rights reserved.
# SPDX-License-Identifier: Apache-2.0

name: PyTorch readable results
description: Retain full command output and publish a concise job summary.
inputs:
title:
required: true
description: Human-readable job title.
artifact:
required: true
description: Unique diagnostics artifact name for this matrix job.
status:
required: true
description: Job status before reporting, including failures in external actions.
runs:
using: composite
steps:
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
id: logs
if: ${{ always() }}
with:
name: ${{ inputs.artifact }}
path: ${{ runner.temp }}/pytorch-ci-logs
if-no-files-found: warn
retention-days: 14
- name: Show results and artifact link
if: ${{ always() }}
shell: bash
env:
KUMO_CI_TITLE: ${{ inputs.title }}
KUMO_CI_ARTIFACT_URL: ${{ steps.logs.outputs.artifact-url }}
KUMO_CI_JOB_STATUS: ${{ steps.logs.outcome == 'failure' && 'failure (artifact upload)' || inputs.status }}
run: python3 "$GITHUB_WORKSPACE/test/kumo_pytorch/ci_reporting.py" summary
19 changes: 16 additions & 3 deletions .github/actions/kumo-pytorch-setup/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,16 @@

name: kumo-pytorch setup
description: >-
kumo-engine's locked CPU workspace environment (CPU torch, kumo-core and its
kumo-engine's locked workspace environment (CPU or CUDA torch, kumo-core and its
torch-geometric) plus kumo-pytorch and its dev extra (with notebooks: 'true',
also its examples extra and a pinned Jupyter kernel and client); with
diskgraph-wheel-artifact, also DiskGraph from that wheel of this commit's
packages/diskgraph.
inputs:
torch-extra:
description: Locked workspace PyTorch extra (cpu or cu130).
required: false
default: cpu
python-version:
required: false
default: '3.12'
Expand Down Expand Up @@ -41,21 +45,30 @@ runs:
- name: Install DiskGraph from the wheel
if: inputs.diskgraph-wheel-artifact != ''
shell: bash
run: ./scripts/diskgraph install --python "$(command -v python)" -f --wheel /tmp/diskgraph-wheel
run: |
python3 "$GITHUB_WORKSPACE/test/kumo_pytorch/ci_reporting.py" run \
--title "Setup / Install DiskGraph from the wheel" -- bash -e -o pipefail <<'KUMO_CI_COMMAND'
./scripts/diskgraph install --python "$(command -v python)" -f --wheel /tmp/diskgraph-wheel
KUMO_CI_COMMAND
- name: Install the locked workspace environment, then kumo-pytorch into it
shell: bash
env:
NOTEBOOKS: ${{ inputs.notebooks }}
TORCH_EXTRA: ${{ inputs.torch-extra }}
run: |
python3 "$GITHUB_WORKSPACE/test/kumo_pytorch/ci_reporting.py" run \
--title "Setup / Install the locked workspace environment, then kumo-pytorch into it" -- bash -e -o pipefail <<'KUMO_CI_COMMAND'
set -euo pipefail
case "$TORCH_EXTRA" in cpu|cu130) ;; *) exit 1 ;; esac
examples=()
if [ "$NOTEBOOKS" = true ]; then examples=(--extra examples); fi
# --inexact keeps the DiskGraph wheel installed above and the root environment.
uv sync --locked --inexact --no-install-package diskgraph --extra cpu
uv sync --locked --inexact --no-install-package diskgraph --extra "$TORCH_EXTRA"
uv sync --locked --inexact --no-install-package diskgraph --no-install-package torch \
--package kumo-pytorch --extra dev "${examples[@]}"
# The notebook runner's tools are not package dependencies.
if [ "$NOTEBOOKS" = true ]; then uv pip install nbclient==0.11.0 ipykernel==7.4.0; fi
KUMO_CI_COMMAND
- name: Runner
shell: bash
run: |
Expand Down
20 changes: 19 additions & 1 deletion .github/workflows/_diskgraph-wheel.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,11 @@ name: DiskGraph wheel
# DiskGraph from source.
on: # yamllint disable-line rule:truthy
workflow_call:
inputs:
readable-output:
description: Retain PyTorch build logs and publish its concise diagnostics summary.
type: boolean
default: false

permissions:
contents: read
Expand All @@ -25,6 +30,7 @@ jobs:
GITHUB_REPOSITORY: ${{ github.repository }}
GITHUB_REPOSITORY_OWNER: ${{ github.repository_owner }}
GITHUB_RUN_ID: ${{ github.run_id }}
KUMO_CI_READABLE: ${{ inputs.readable-output }}
# Oldest runner image: the wheel must load on every consumer's glibc.
runs-on: ubuntu-22.04
timeout-minutes: 60
Expand Down Expand Up @@ -64,7 +70,12 @@ jobs:
key: python-wheel-py${{ steps.python.outputs.version }}
- name: Build wheel
if: steps.cache.outputs.cache-hit != 'true'
run: uv build --wheel --out-dir /tmp/diskgraph-wheel packages/diskgraph
run: |
output=()
if [[ "$KUMO_CI_READABLE" == true ]]; then
output=(python3 "$GITHUB_WORKSPACE/test/kumo_pytorch/ci_reporting.py" run --title 'DiskGraph wheel build' --)
fi
"${output[@]}" uv build --wheel --out-dir /tmp/diskgraph-wheel packages/diskgraph
- if: steps.cache.outputs.cache-hit != 'true'
uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5
with:
Expand All @@ -77,3 +88,10 @@ jobs:
if-no-files-found: error
compression-level: 0
retention-days: 7
- name: Publish readable results and full logs
if: ${{ always() && inputs.readable-output && steps.cache.outputs.cache-hit != 'true' }}
uses: ./.github/actions/kumo-pytorch-results
with:
title: DiskGraph wheel build
artifact: kumo-pytorch-diagnostics-diskgraph-build
status: ${{ job.status }}
80 changes: 80 additions & 0 deletions .github/workflows/_kumo-pytorch-aml-checks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.
# All rights reserved.
# SPDX-License-Identifier: Apache-2.0

name: PyTorch — Dataset & Model Checks

on: # yamllint disable-line rule:truthy
workflow_call:
inputs:
diskgraph-wheel-artifact:
description: DiskGraph wheel built from this run's candidate revision.
type: string
required: true

permissions:
contents: read

jobs:
checks:
name: Correctness / Shared Dataset & Model Checks
runs-on: ubuntu-24.04
timeout-minutes: 45
defaults:
run:
shell: bash
working-directory: packages/experimental/kumo-pytorch
env:
KUMO_CI_LABEL: Shared dataset and model fixtures
KUMO_CI_REPORTS: |
$RUNNER_TEMP/*.xml
$RUNNER_TEMP/*acquisition.json
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
with:
persist-credentials: false
- uses: ./.github/actions/kumo-pytorch-setup
with:
diskgraph-wheel-artifact: ${{ inputs.diskgraph-wheel-artifact }}
- name: Install test-only Polars for required CPU coverage
run: |
python3 "$GITHUB_WORKSPACE/test/kumo_pytorch/ci_reporting.py" run \
--title "$KUMO_CI_LABEL / Install test-only Polars for required CPU coverage" -- bash -e -o pipefail <<'KUMO_CI_COMMAND'
uv pip install polars==2.1.0
KUMO_CI_COMMAND
- name: Check shared dataset/model consumers on tiny Kumo batches
working-directory: .
run: |
python3 "$GITHUB_WORKSPACE/test/kumo_pytorch/ci_reporting.py" run \
--title "$KUMO_CI_LABEL / Check shared dataset/model consumers on tiny Kumo batches" -- bash -e -o pipefail <<'KUMO_CI_COMMAND'
python -m pytest -c test/kumo_pytorch/pytest.ini \
--confcutdir=test/kumo_pytorch --basetemp="$RUNNER_TEMP/aml-checks" \
test/kumo_pytorch/model_checks --required-coverage=cpu -q --junitxml="$RUNNER_TEMP/aml-checks.xml"
KUMO_CI_COMMAND
- name: Retain controlled fixture learning reports
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
if: ${{ always() }}
with:
name: kumo-pytorch-aml-learning
path: |
${{ runner.temp }}/aml-checks.xml
${{ runner.temp }}/aml-checks/**/learning.json
${{ runner.temp }}/aml-checks/**/training-acceptance/**/report.json
${{ runner.temp }}/aml-checks/**/training-acceptance/**/loss-curve.csv
${{ runner.temp }}/aml-checks/**/results.json
${{ runner.temp }}/aml-checks/**/finetuning/report.json
${{ runner.temp }}/aml-checks/**/finetuning/finetune.log
${{ runner.temp }}/aml-checks/**/finetuning/child.log
${{ runner.temp }}/aml-checks/**/mid-epoch-resume/report.json
${{ runner.temp }}/aml-checks/**/mid-epoch-resume/child.log
${{ runner.temp }}/aml-checks/**/mid-epoch-resume/restore-fingerprints.json
${{ runner.temp }}/aml-checks/**/coverage.json
${{ runner.temp }}/aml-checks/**/report.md
retention-days: 14
- name: Publish readable results and full logs
if: ${{ always() }}
uses: ./.github/actions/kumo-pytorch-results
with:
title: Shared dataset and model fixtures
artifact: kumo-pytorch-diagnostics-fixtures
status: ${{ job.status }}
Loading
Loading