Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .github/workflows/kumo-pytorch-diskgraph.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,13 @@ name: kumo-pytorch DiskGraph
on: # yamllint disable-line rule:truthy
pull_request:
paths:
- 'pyproject.toml'
- 'packages/experimental/kumo-pytorch/**'
- 'packages/kumo-engine/**'
- 'packages/kumo-core/**'
- 'packages/kumo-api/**'
- 'packages/kumo-pql/**'
- 'packages/kumo-settings/**'
- 'packages/diskgraph/**'
- 'scripts/diskgraph'
- '.github/workflows/_diskgraph-wheel.yml'
Expand All @@ -22,7 +28,13 @@ on: # yamllint disable-line rule:truthy
push:
branches: [main]
paths:
- 'pyproject.toml'
- 'packages/experimental/kumo-pytorch/**'
- 'packages/kumo-engine/**'
- 'packages/kumo-core/**'
- 'packages/kumo-api/**'
- 'packages/kumo-pql/**'
- 'packages/kumo-settings/**'
- 'packages/diskgraph/**'
- 'scripts/diskgraph'
- '.github/workflows/_diskgraph-wheel.yml'
Expand Down Expand Up @@ -150,3 +162,26 @@ jobs:
run: |
../fresh/bin/python -m pytest -p no:cacheprovider -m notebook --require-imports -q \
--durations=0 tests/test_notebooks.py
- name: Build the engine and its workspace dependencies as wheels
working-directory: .
run: |
set -euo pipefail
for package in kumo-api kumo-pql kumo-settings kumo-core kumo-engine; do
uv build --wheel --package "$package" --out-dir "$RUNNER_TEMP/engine-wheels"
done
cp packages/kumo-engine/test/test_pytorch_snapshot.py "$RUNNER_TEMP/stage/tests/"
- name: Install the engine wheels alongside PyTorch and DiskGraph
working-directory: ${{ runner.temp }}
run: |
uv pip install --python fresh/bin/python --torch-backend cpu engine-wheels/*.whl
uv pip check --python fresh/bin/python
- name: Connector snapshot to columnar batches, from installed wheels
working-directory: ${{ runner.temp }}/stage
run: |
../fresh/bin/python -I -c '
import pathlib, sys
import diskgraph, kumo.engine, kumo.pytorch, kumo_core, kumoapi, kumopql, kumo.settings
for module in (diskgraph, kumo.engine, kumo.pytorch, kumo_core, kumoapi, kumopql, kumo.settings):
assert pathlib.Path(module.__file__).is_relative_to(sys.prefix), module.__file__
'
../fresh/bin/python -m pytest -p no:cacheprovider -q --require-imports tests/test_pytorch_snapshot.py
78 changes: 76 additions & 2 deletions packages/experimental/kumo-pytorch/docs/04-engines-and-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,8 +179,14 @@ read.

The index holds rows, key maps and relations. It holds no anchors, labels or
fitted encoders, so several contracts over the same tables can share one index.
Fitting keeps the column statistics of encoders over stored columns in
`col_stats/`, so it needs the index directory to be writable. Each encoder
By default, fitting keeps column statistics of encoders over stored columns in
the index's `col_stats/`, so that directory must be writable. For the ordinary
`DiskGraphEngine(index=...)` constructor, omitting `work_dir` or explicitly setting
it to the index path uses this same layout and reuses the same caches. Paths that
resolve to the index, including symlinks, are equivalent. Other paths inside the
index are rejected. External work directories and their writable roots must be
disjoint from the base, including after resolving symlinks. Supply an external
`work_dir` to put newly computed statistics and task state there instead. Each encoder
cutoff has one directory there, with a `<table>.col_stats.json` file per table.
A later fit at the same cutoff computes only the columns and statistics those
files lack, and adds them. Ingest computes the statistics for the cutoffs in
Expand All @@ -195,6 +201,74 @@ it was built. With
`ingest="never"`, `prepare` requires an index that already exists, which is
what you usually want in a scoring job.

### Open an existing snapshot or index

`DiskGraphEngine.from_snapshot(snapshot, work_dir=...)` consumes an existing
`kumo.engine.GraphSnapshot`. Use the engine's existing snapshot loader to
restore an exported snapshot first. `DiskGraphEngine.from_index(index_path,
work_dir=...)` opens the native index directory directly, including a
PyTorch-produced index. Neither factory exports sources or ingests the base.

<!-- test: skip (snapshot, sample_spec, batch_spec and anchors belong to the caller) -->
```python
engine = api.DiskGraphEngine.from_snapshot(snapshot, work_dir="/scratch/loader")
contract = api.DataLoaderSpec(
schema=engine.schema,
sample=sample_spec,
batch=batch_spec,
)
with api.prepare(contract=contract, engine=engine) as plan:
with plan.materialize(anchors) as task:
for batch in task.batches(output_format="columnar"):
graph = batch.inputs["graph"]
```

`SampleSpec` and `BatchSpec` retain their usual meaning. `engine.schema` derives
stored types, keys, clocks and relationships from the existing native config
and metadata. PyTorch-produced indexes retain their logical relation aliases;
engine/native indexes initially use native FK names. A caller can provide a
compatible schema projection and relation aliases: binding checks endpoints,
FK columns and direction against the actual index. `prepare` rejects a Dataset
alongside an existing-index engine. Anchors still select entity/time requests;
neighborhoods are sampled during iteration.

The base may be read-only. Task extensions, scalar-materialization scratch,
new statistics, tokenizers and token-length caches go under the external work
directory; existing immutable base caches can be read in place. Closing a task
cleans up its temporary files. Derived caches remain for reuse. No new base
metadata file is written. Fitted-state statistics provenance records absolute paths
in external mode (including reused base files); these are diagnostic references,
not inputs used when restoring fitted encoders. They can refer to the old location
after relocation. External statistics share one directory per artifact and cutoff;
the native writer locks, merges existing keys, and atomically replaces each file.
Concurrent requests may duplicate computation but do not discard published keys.
External storage requires a DiskGraph build supporting `cutoff_col_stats(output_dir=...)`;
preparation checks that capability because older and newer development builds can
share version 0.2.0. Install DiskGraph from the matching repository revision.
Existing snapshot identity or PyTorch producer fingerprints survive relocation. A bare completed native index without producer
identity is hashed in full on each opening, which can be expensive for large artifacts.

Opening requires the resolved `diskgraph_config.json` and the producer's
completion metadata. Engine snapshots use the native inventory/integrity check;
PyTorch indexes use their existing manifest, published after ingestion. Older
artifacts missing resolved config are refused. Native edge-table layouts,
end-time validity and unsupported stored types (including lists) are refused
explicitly. Snapshot adoption does not recover relations omitted during ingestion
or enable additional sampling strategies.

For **source ingestion/reuse** after connector restaging, pass
`DiskGraphEngine(index=..., work_dir=..., source_versions=...)`. Map every table
to `{"source": "connector/source-coordinate", "version": "connector-version"}`.
Equal trusted identities/versions and schema permit reuse across export paths;
a changed identity/version or schema refuses reuse. Without trustworthy versions,
keep the existing file-validation mode. This opt-in trusts the caller to change
the version whenever source data changes; reuse does not check Parquet contents,
file sizes or mtimes. Reusing a stale version token therefore reuses the old index.
Physical file checks still detect source changes during initial ingestion.
Opening an already materialized index
never checks whether its sources now have a newer version; refreshing sources
remains an explicit producer operation.

`output_format="model"` lets the DiskGraph engine build the batch's tensors
column by column, without building a Python record per event. It refuses raw types
that a model tensor can't hold (strings, dates, decimals) before fetching any
Expand Down
2 changes: 1 addition & 1 deletion packages/experimental/kumo-pytorch/docs/api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ The [guide](../README.md) explains the concepts in order. The behavior spec, wit
| [`InputExposureWarning`](targets.md#inputexposurewarning) | `kumo.pytorch.api` | A masked value is also selected by an unmasked named input. |
| [`DataLoaderSpec`](lifecycle.md#dataloaderspec) | `kumo.pytorch.api` | Schema, sampled values, and explicitly encoded model outputs. |
| [`check`](lifecycle.md#check) | `kumo.pytorch.api` | Return everything wrong with this combination, without raising. |
| [`prepare`](lifecycle.md#prepare) | `kumo.pytorch.api` | Validate, then bind the dataset through the engine. |
| [`prepare`](lifecycle.md#prepare) | `kumo.pytorch.api` | Validate, then bind sources or an existing index through the engine. |
| [`preview`](lifecycle.md#preview) | `kumo.pytorch.api` | Build up to eight raw samples and one model-facing batch. |
| [`Plan`](lifecycle.md#plan) | `kumo.pytorch.api` | A contract bound to a dataset through an engine, before any anchors. |
| [`Task`](lifecycle.md#task) | `kumo.pytorch.api` | Anchors bound to a plan: what is iterated, served from, and written out. |
Expand Down
50 changes: 50 additions & 0 deletions packages/experimental/kumo-pytorch/docs/api/engines.md
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,8 @@ class DiskGraphEngine(
num_shards: int = 1,
ingest: Literal['if_missing', 'never'] = 'if_missing',
encoder_cutoffs: Sequence[str | int | datetime | date | None] = (),
work_dir: str | Path | None = None,
source_versions: Mapping[str, Mapping[str, str]] | None = None,
)
```

Expand Down Expand Up @@ -330,6 +332,54 @@ step, before any data is read.
- `num_shards`: index shards.
- `ingest`: `"if_missing"` builds the index when absent; `"never"` requires an existing one. A mismatching manifest is an error, never permission to reuse.
- `encoder_cutoffs`: `encoder_cutoff` values whose column statistics an ingest computes.
- `work_dir`: task/cache directory. Omit it or pass the index path (including an alias resolving to it) for the original index-local layout and cache reuse. Other paths must be outside the index; existing-index factories require an external directory.
- `source_versions`: optional table-to-identity mapping for source ingestion/reuse. Each entry has nonempty `source` (connector plus source identity) and `version` strings. These trusted connector versions replace file path/size/mtime identity; missing versions are never treated as equal. Omit for legacy file validation. The caller must change the version whenever source data changes: reuse does not compare Parquet contents, file sizes or mtimes in this opt-in mode.

#### `DiskGraphEngine.from_index`

*classmethod*

```python
DiskGraphEngine.from_index(
index: str | Path,
*,
work_dir: str | Path,
num_threads: int | None = None,
) -> DiskGraphEngine
```

Open a complete native index without its original sources or any base writes.

Read schema/relations from existing metadata. Older indexes without the
resolved native config are refused. Native edge tables and non-scalar
stored columns are not supported. Bare indexes without producer identity
are hashed in full on each open; engine provenance and PyTorch manifests avoid that scan.
`work_dir` holds task extensions and derived caches outside the base.

#### `DiskGraphEngine.from_snapshot`

*classmethod*

```python
DiskGraphEngine.from_snapshot(
snapshot: Any,
*,
work_dir: str | Path,
num_threads: int | None = None,
) -> DiskGraphEngine
```

Bind a kumo.engine GraphSnapshot using its existing index and identity.

Uses public `index_path` and `id` fields; importing kumo.engine is
unnecessary. Load exported snapshots through the existing engine loader.
No source export, base ingestion or additional metadata file is needed.

#### `DiskGraphEngine.schema`

*property*

Resolved stored schema for a from_index/from_snapshot engine.

#### `DiskGraphEngine.check_contract`

Expand Down
16 changes: 10 additions & 6 deletions packages/experimental/kumo-pytorch/docs/api/lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,8 +144,8 @@ A readable multi-line summary of the contract.
```python
def check(
contract: DataLoaderSpec,
dataset: Dataset,
engine: Engine,
dataset: Dataset | None = None,
engine: Engine | None = None,
*,
anchors: AnchorTable | None = None,
inference: bool = False,
Expand All @@ -169,14 +169,18 @@ The whole contract must still be well formed.
```python
def prepare(
contract: DataLoaderSpec,
dataset: Dataset,
engine: Engine,
dataset: Dataset | None = None,
engine: Engine | None = None,
*,
inference: bool = False,
) -> Plan
```

Validate, then bind the dataset through the engine.
Validate, then bind sources or an existing index through the engine.

Omit `dataset` only for a DiskGraphEngine created with `from_index` or
`from_snapshot`. Its stored schema supplies omitted types; it never reads
the original sources or rebuilds the base. `engine` is always required.

Every check that does not need data happens first, so an unexecutable
combination is refused before a row is read. Anchors, and everything fitted
Expand Down Expand Up @@ -221,7 +225,7 @@ Called as `preview(task, requests)` or
```python
class Plan(
contract: DataLoaderSpec,
dataset: Dataset,
dataset: Dataset | None,
engine: Engine,
source: SampleSource,
*,
Expand Down
65 changes: 65 additions & 0 deletions packages/experimental/kumo-pytorch/src/kumo/pytorch/_storage.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.
# All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Storage policy for index-local and external consumer state."""

from __future__ import annotations

import hashlib
from pathlib import Path


def normalize_work_dir(
index: str | Path, work_dir: str | Path | None, *, external_only: bool = False
) -> Path | None:
"""Use the legacy layout for an omitted or explicitly equal index path."""
base = Path(index).resolve()
work = None if work_dir is None else Path(work_dir).resolve()
if not external_only and (work is None or work == base):
return None
if work is None:
raise ValueError("work_dir must be outside the base index")
# Validate every writable root before creating any of them, including symlinks.
for destination in (work, work / "tasks", work / "train_tables", work / "cache"):
resolved = destination.resolve()
if resolved.is_relative_to(base) or base.is_relative_to(resolved):
raise ValueError(
"work_dir and its writable roots must be outside and disjoint from the base index"
)
return work


class ColumnCache:
"""Read immutable base caches and publish missing entries to the writable root."""

def __init__(self, index: str | Path, root: str | Path | None = None):
self.index = Path(index)
self.root = self.index if root is None else Path(root)
self.external = self.root.resolve() != self.index.resolve()
if not self.external:
self.root = self.index

@classmethod
def for_work_dir(cls, index: str | Path, work_dir: Path | None, identity: str):
root = (
None
if work_dir is None
else work_dir / "cache" / hashlib.sha256(identity.encode()).hexdigest()
)
return cls(index, root)

def destination(self, name: str) -> Path:
return self.root / "col_stats" / name

def candidates(self, name: str, *, base_name: str | None = None) -> tuple[Path, ...]:
base = self.index / "col_stats" / (name if base_name is None else base_name)
return tuple(dict.fromkeys((base, self.destination(name))))

def resolve(self, name: str) -> Path:
return next(
(path for path in self.candidates(name) if path.is_file()), self.destination(name)
)

def reference(self, path: Path) -> str:
return str(path if self.external else path.relative_to(self.index / "col_stats"))
Loading
Loading