Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .agents/skills/build-ui/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Planning remains read-only until the user approves implementation.
- `--skip-browser`: Skip browser-based visual verification
- `--no-review`: Skip self-review
- `--validate-only`: Complete implementation and verification but stop before push/PR creation
- `--base-branch` (default `master`)
- `--base-branch` (default `main` on `upstream`)
- `--resume <stage>`
- `--mode light|standard|full` (default `standard` for interactive/local use; the GitHub headless workflow defaults to `light` unless the trigger passes `--mode standard` or `--mode full`)

Expand Down Expand Up @@ -549,9 +549,9 @@ Checkpoint 1
Only after Checkpoint 1 approval:

```bash
git fetch origin
git branch --list "ui/{issue}-*"
git checkout -b ui/{issue}-{slug} origin/{base-branch}
git fetch upstream
git branch --list "dm-ui-{issue}-*"
git checkout --no-track -b dm-ui-{issue}-{slug} upstream/{base-branch}
```

Rules:
Expand Down Expand Up @@ -882,7 +882,7 @@ If approved:

1. fetch latest base branch
2. optionally rebase if the user wants that behavior
3. push the feature branch
3. push the feature branch to the same-name upstream branch with `git push --set-upstream upstream {work-branch}`
4. determine the release-notes label
5. detect breaking changes
6. create the PR from `.github/PULL_REQUEST_TEMPLATE.md`
Expand Down Expand Up @@ -938,7 +938,7 @@ Use the exact phase label from this table in `lastCompletedPhase`. Do not collap
Rules:

- reuse the existing work branch
- in `issue` mode, that is usually `ui/{issue}-*`
- in `issue` mode, that is usually `dm-ui-{issue}-*`
- in `pr` mode with continuation, that is usually `{headRefName}`
- verify current branch state before skipping earlier stages
- resume only when the branch state clearly corresponds to the requested stage
Expand Down
2 changes: 1 addition & 1 deletion .agents/skills/build-ui/references/decision-tables.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Use this file for stop/skip decisions, dependency handling, catalog gaps, browse
| 0 | Working git repo, readable worktree | Stop | Cannot plan safely |
| 0 | `gh` available/authenticated for issue or PR work | Stop or continue local-only planning | No GitHub-backed issue/PR work |
| 0b | User approved implementation | Stop | No mutations before approval |
| 0b | Base branch exists on origin | Stop | Branch setup blocked |
| 0b | Base branch exists on upstream (`main` by default) | Stop | Branch setup blocked |
| P | PR exists and is readable | Stop | Cannot inspect existing work |
| 1 | Issue exists and is readable | Stop | Cannot derive requirements |
| 1 | Enough issue detail to synthesize AC | Ask user to clarify | No implementation without usable AC |
Expand Down
11 changes: 11 additions & 0 deletions .github/actions/engine-gpu-smoke/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,17 @@ runs:
PYTHONDONTWRITEBYTECODE: '1'
run: |
set -euo pipefail
edge_backend="${DG_EDGE_BACKEND-mmap}"
case "$edge_backend" in
mmap|buffered|direct) ;;
*) echo "Unsupported DG_EDGE_BACKEND: $edge_backend" >&2; exit 2 ;;
esac
export DG_EDGE_BACKEND="$edge_backend"
if [[ "$edge_backend" == direct ]]; then
export DG_DIRECT_CACHE_GB=1
fi
printf 'Native edge backend: %s (direct cache GB: %s)\n' \
"$DG_EDGE_BACKEND" "${DG_DIRECT_CACHE_GB:-default}"
for name in ${!PIP_@}; do unset "$name"; done
export PIP_CONFIG_FILE=/dev/null
mkdir -p "$RUNNER_TEMP/engine-gpu-smoke"
Expand Down
8 changes: 8 additions & 0 deletions .github/workflows/kumo-engine-pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,12 @@ on: # yamllint disable-line rule:truthy
description: Pull request whose current head is tested
required: true
type: string
edge_backend:
description: Native edge I/O backend for all eight GPU tasks
required: false
type: choice
default: mmap
options: [mmap, buffered, direct]
workflow_call:
inputs:
build_only:
Expand Down Expand Up @@ -263,6 +269,8 @@ jobs:
name: engine-pr-wheel-${{ github.sha }}
path: ${{ runner.temp }}/pr-wheel
- uses: ./.github/actions/engine-gpu-smoke
env:
DG_EDGE_BACKEND: ${{ inputs.edge_backend || 'mmap' }}
with:
wheel-path: ${{ runner.temp }}/pr-wheel/*.whl
reload-atol: '0.005'
Expand Down
125 changes: 125 additions & 0 deletions .github/workflows/testing.yml
Original file line number Diff line number Diff line change
Expand Up @@ -396,6 +396,39 @@ jobs:
workspaces: packages/diskgraph -> target
key: rust-tests-py${{ steps.python.outputs.version }}
- run: cargo test --locked --workspace
- name: Check native I/O and fork lifecycle
run: bash scripts/check_native_io.sh

diskgraph-macos:
permissions:
contents: read
runs-on: macos-15
timeout-minutes: 60
env:
MACOSX_DEPLOYMENT_TARGET: '14.0'
defaults:
run:
working-directory: packages/diskgraph
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
with:
persist-credentials: false
- uses: ./.github/actions/rust-toolchain
- uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with:
python-version: '3.12'
activate-environment: true
enable-cache: false
- name: Build and install the candidate macOS wheel
run: |
uv build --wheel --out-dir dist
uv pip install --no-deps dist/*.whl
uv pip install pytest pyarrow==24.0.0
- name: Import the installed wheel and exercise default mmap ingestion and sampling
run: |
python -m pytest -q -c /dev/null \
--confcutdir=diskgraph-pybind/python/tests \
diskgraph-pybind/python/tests/test_backend_portability.py

diskgraph-clippy:
permissions:
Expand Down Expand Up @@ -511,6 +544,98 @@ jobs:
subprocess.run([str(ingest), '--help'], check=True)
PY

diskgraph-cpu-tasks:
# Exercise the candidate wheel through real source Engine training without
# hosted-service credentials or a GPU dependency profile.
needs: [diskgraph-wheel]
if: ${{ !cancelled() && needs.diskgraph-wheel.result == 'success' }}
runs-on: ubuntu-22.04
timeout-minutes: 30
permissions:
contents: read
env:
GITHUB_TOKEN: ''
AWS_EC2_METADATA_DISABLED: 'true'
SETUPTOOLS_SCM_PRETEND_VERSION: '0.0.0'
DG_TEST_NATIVE_TASKS: '1'
DG_TEST_NATIVE_IO: '1'
DG_TEST_CORE_IO: '1'
DG_DIRECT_CACHE_GB: '1'
DG_IO_RINGS: '2'
DG_IO_QD: '8'
RAYON_NUM_THREADS: '2'
OMP_NUM_THREADS: '1'
OPENBLAS_NUM_THREADS: '1'
MKL_NUM_THREADS: '1'
MPLCONFIGDIR: /tmp/diskgraph-cpu-matplotlib
PYTHONPATH: ${{ github.workspace }}/packages/kumo-engine:${{ github.workspace }}/packages/kumo-core/src:${{ github.workspace }}/packages/kumo-api:${{ github.workspace }}/packages/kumo-pql:${{ github.workspace }}/packages/kumo-engine/test:${{ github.workspace }}
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6
with:
persist-credentials: false
- uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with:
python-version-file: .python-version
activate-environment: true
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8
with:
name: diskgraph-wheel
path: dist
- name: Install candidate native wheel and workspace CPU dependencies
run: |
sha256sum dist/diskgraph-*.whl
uv pip install --no-deps dist/diskgraph-*.whl
# Keep the prebuilt candidate; the root CPU profile installs the
# workspace Core/API/PQL and CPU Faiss, without Engine's GPU extra.
uv sync --active --frozen --inexact --extra cpu --extra test \
--no-install-package diskgraph
uv pip install --no-deps -e packages/kumo-settings
python - <<'PY'
import hashlib
import sys
import zipfile
from importlib.metadata import distributions
from pathlib import Path

import diskgraph._c as native
import kumo.settings
import torch
from io_utils import fsutil

faiss = {d.metadata['Name'] for d in distributions()} & {'faiss-cpu', 'faiss-gpu'}
assert faiss == {'faiss-cpu'}, faiss
assert torch.version.cuda is None, torch.version.cuda
settings_source = Path('packages/kumo-settings/src/kumo/settings.py').resolve()
assert Path(kumo.settings.__file__).resolve() == settings_source
assert Path(native.__file__).resolve().is_relative_to(Path(sys.prefix).resolve())
[wheel] = fsutil.listdir('dist', suffix='.whl')
with fsutil.get_binary_handle(wheel, 'rb') as stream, zipfile.ZipFile(stream) as archive:
[extension] = [name for name in archive.namelist() if name.endswith('/_c.abi3.so')]
expected = hashlib.sha256(archive.read(extension)).hexdigest()
actual = hashlib.sha256(fsutil.read_binary(native.__file__)).hexdigest()
assert actual == expected, 'workspace sync replaced the candidate extension'
print(sys.version, native.__file__, actual)
PY
- name: Train, predict and reload all eight tasks on three backends
run: |
python -m pytest -q -s -c /dev/null \
--confcutdir=packages/kumo-engine/test \
packages/kumo-engine/test/test_native_io_tasks.py
- name: Verify native Python ownership, parity and seeded sampling
run: |
python -m pytest -q -c /dev/null \
--confcutdir=packages/diskgraph/diskgraph-pybind/python/tests \
--junitxml=/tmp/diskgraph-native-python.xml \
packages/diskgraph/diskgraph-pybind/python/tests/test_native_io_readiness.py
python - <<'PY'
import xml.etree.ElementTree as ET

from io_utils import fsutil

suites = ET.fromstring(fsutil.read_text('/tmp/diskgraph-native-python.xml'))
assert sum(int(suite.get('skipped', '0')) for suite in suites) == 0
PY

pyarrow-range:
# DiskGraph accepts pyarrow>=16,<25 and every other job runs the locked
# version. Run the DiskGraph, kumo-core and kumo-adls suites at both ends.
Expand Down
40 changes: 39 additions & 1 deletion 3rdparty/notices/inventory.json

Large diffs are not rendered by default.

Loading
Loading