Skip to content
89 changes: 89 additions & 0 deletions packages/experimental/kumo-pytorch/docs/05-training-batches.md
Original file line number Diff line number Diff line change
Expand Up @@ -380,3 +380,92 @@ engine](04-engines-and-data.md#the-diskgraph-engine) for how the index is built
and reused.

Next: [inference and serving](06-inference-and-serving.md).

## Arrow graphs for foundation-model payloads

For consumers that build their own model representation, a materialized
DiskGraph task can expose sampled graph columns without first encoding them
into tensor/mask dictionaries:

Reuse the schema, Parquet dataset and anchor table above with a graph-only
contract. Select the training split because the anchor table contains both
training and validation rows.

<!-- test: diskgraph -->
```python
graph_contract = api.DataLoaderSpec(
schema=schema,
sample=api.SampleSpec(
entity=api.EntitySpec(table="users"),
neighborhoods={
"history": api.NeighborhoodSpec(
layers=(
api.GraphLayer(
samples=(api.GraphSample("user_purchases", 4, strategy="recent"),)
),
)
)
},
),
batch=api.BatchSpec(
inputs={
"graph": api.Input(
source=api.NeighborhoodSource("history"),
representation=api.Graph(
node_fields={
"users": api.Fields((api.Encode("segment", api.Raw()),)),
"purchases": api.Fields((api.Encode("amount", api.Raw()),)),
}
),
layout=None,
)
},
batching=api.Batching(max_samples=1024),
),
)
with (
api.prepare(graph_contract, parquet, engine) as graph_plan,
graph_plan.materialize(anchors=anchor_table, fit_split="train") as graph_task,
):
for batch in graph_task.batches(split="train", output_format="columnar"):
graph = batch.inputs["graph"] # api.ColumnarGraph
roots = graph.nodes[graph.root_type]
root_ids = roots.ids.slice(0, batch.batch_size)
for table_name, nodes in graph.nodes.items():
features = nodes.features # pyarrow.Table; NULLs stay in validity bitmaps
example_positions = nodes.owners # indexes into batch.metadata["sample_ids"]
for relation_name, edges in graph.edges.items():
selected_pairs = edges.index # Arrow columns: source, destination
print("training roots:", root_ids.to_pylist())
```

<!-- test: output -->
```text
training roots: [1, 1, 2, 2, 3, 3, 4, 4, 5, 5]
```

Node rows are occurrences within an example: repeated anchors and different
cutoffs do not share graph occurrences. The root table starts with one root
per batch example, in batch order. Edge endpoints index the node tables in
**declared left-to-right orientation**, including reverse traversal and
self-relations. They contain only sampled links. The same selected pair can
appear in multiple chunks. Consumers can normalize these pairs while building
their payload, or pass `deduplicate_edges=True` to `batches` for unique pairs
per relation in first-occurrence order. This opt-in step materializes edge
arrays and adds deduplication work; the default `False` retains native chunks.
Deduplication preserves separate examples, including repeated entity IDs.
Chunks do not identify traversal direction or discovery radius.

Feature columns contain exactly the requested Raw fields, in declaration order
and declared types. Primary-key identity is available separately in `ids`.
Native buffers are shared where compatible; restoring declared types may
allocate. All buffers remain valid after iteration and task closure, and
`Batch.to(...)` keeps them as CPU Arrow values.

This format currently supports one graph-only input, explicit Raw encodings,
fixed sample counts, and a selection contained in one split. It refuses
request-backed tasks, non-DiskGraph engines, other sample values, targets,
entity filters, and availability checks before iteration. Select a single
split explicitly when anchors span several splits. Shuffle, rank sharding,
padding, drop-last, and iterator resume retain their existing behavior.
Existing sampling limits, including timeless destinations, still apply.
3 changes: 3 additions & 0 deletions packages/experimental/kumo-pytorch/docs/api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,9 @@ The [guide](../README.md) explains the concepts in order. The behavior spec, wit
| [`Fields`](batches.md#fields) | `kumo.pytorch.api` | Select ordered fields with explicit encoders, grouping, and field IDs. |
| [`Encode`](batches.md#encode) | `kumo.pytorch.api` | Encode one sampled field, optionally translating IDs into a final space. |
| [`Graph`](batches.md#graph) | `kumo.pytorch.api` | Node attributes use the same explicit field encodings as ordinary columns. |
| [`ColumnarGraph`](batches.md#columnargraph) | `kumo.pytorch.api` | CPU Arrow graph buffers emitted by `Task.batches(output_format="columnar")`. |
| [`ColumnarNodes`](batches.md#columnarnodes) | `kumo.pytorch.api` | Aligned columns for node occurrences, isolated per example. |
| [`ColumnarEdges`](batches.md#columnaredges) | `kumo.pytorch.api` | Selected edges in declared left-to-right relation orientation. |
| [`Columns`](batches.md#columns) | `kumo.pytorch.api` | Keep selected encoded fields as separate typed columns. |
| [`Tokens`](batches.md#tokens) | `kumo.pytorch.api` | Concatenate one row's encoded IDs in the declared field order. |
| [`Composite`](batches.md#composite) | `kumo.pytorch.api` | One category ID for the ordered tuple of scalar component results. |
Expand Down
71 changes: 69 additions & 2 deletions packages/experimental/kumo-pytorch/docs/api/batches.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,71 @@ Node attributes use the same explicit field encodings as ordinary columns.

- `node_fields`: table name to the `Fields` (scalar column encodings of physical columns) emitted for its nodes. Field-name IDs are not supported on graph nodes.

### `ColumnarGraph`

*class* · `kumo.pytorch.api.ColumnarGraph`

```python
class ColumnarGraph(
root_type: str,
nodes: Mapping[str, ColumnarNodes],
edges: Mapping[str, ColumnarEdges],
)
```

CPU Arrow graph buffers emitted by `Task.batches(output_format="columnar")`.

`nodes` uses table names and `edges` uses declared relation names.
The first `Batch.batch_size` rows of `nodes[root_type]` are the roots,
in batch order, including repeated entities at distinct prediction times.
Isolated roots and empty declared node/edge types are retained.

Buffers remain valid after advancing or closing the iterator/task. Treat
them as read-only. Compatible native Arrow buffers are shared; declared
dtype restoration may allocate. `Batch.to` leaves these CPU buffers in
Arrow form. This output avoids tensor encoding and is intended for callers
that construct their own model representation, such as foundation models.

### `ColumnarNodes`

*class* · `kumo.pytorch.api.ColumnarNodes`

```python
class ColumnarNodes(ids: Array | ChunkedArray, owners: Array, features: Table)
```

Aligned columns for node occurrences, isolated per example.

`ids` contains primary-key values (index-local row IDs when no primary
key is declared). `owners` is a non-null uint32 or uint64 Arrow array of positions
into `Batch.metadata["sample_ids"]`. The same entity may occur under
several owners. `features` is an Arrow table containing exactly the
declared Raw fields, in declaration order and declared dtypes. Arrow
validity bitmaps retain NULLs; no separate masks or filled values are used.
With no declared fields, the table has zero columns; use `len(ids)` for
the node count. All three outputs otherwise have the same row order.

### `ColumnarEdges`

*class* · `kumo.pytorch.api.ColumnarEdges`

```python
class ColumnarEdges(source_type: str, destination_type: str, index: Table)
```

Selected edges in declared left-to-right relation orientation.

`index` is an Arrow table with non-null uint32 or uint64 `source` and
`destination` columns, indexing rows in the corresponding node tables.
Endpoints always belong to the same example. Only edges selected by the
sampling layers are included, never all links among returned nodes.
Reverse traversals are reoriented, including self-relations. Chunks may
repeat a pair selected in both directions. Consumers must normalize these
pairs or request `Task.batches(..., deduplicate_edges=True)` to retain
the first occurrence per relation, at the cost of edge-array allocation.
Deduplication uses node positions, preserving separate examples even when
entity IDs repeat. Chunk order is not traversal provenance.

### `Columns`

*class* · `kumo.pytorch.api.Columns`
Expand Down Expand Up @@ -482,8 +547,10 @@ never mixed. Separate columns sit under their field names with `values`
and `valid`; token outputs use `value_ids`, `position_ids`, validity
and source-event indices, with optional `field_ids` and `field_valid`.
Raw types without a tensor representation stay Arrow arrays in inspectable
output. Metadata carries `sample_ids`, `entity_ids`, `as_of_time`,
`splits`, `is_padding`, request metadata, fingerprints and candidate
output. Columnar output holds `ColumnarGraph` objects with Arrow feature
tables and selected edge indexes. Metadata carries `sample_ids`,
`entity_ids`, `as_of_time`, `splits`, `is_padding`, request metadata,
fingerprints and candidate
identities; it is not a route for undeclared features.

#### `Batch.to`
Expand Down
14 changes: 13 additions & 1 deletion packages/experimental/kumo-pytorch/docs/api/lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -388,7 +388,8 @@ Task.batches(
world_size: int = 1,
drop_last: bool | None = None,
chunk_batches: int = 8,
output_format: Literal['inspectable', 'model'] = 'inspectable',
output_format: Literal['inspectable', 'model', 'columnar'] = 'inspectable',
deduplicate_edges: bool = False,
) -> TaskBatches
```

Expand All @@ -413,6 +414,17 @@ and `world_size`; ranks are disjoint before padding. A token budget
representation before fetching a batch; `"inspectable"` keeps them
as Arrow arrays. Both emit the same `Batch` structure.

`"columnar"` emits `ColumnarGraph` inputs with Arrow feature tables
and selected edge indexes, without tensor encoding. It requires
materialized DiskGraph anchors, one graph input with explicit Raw
fields, fixed sample counts, and a selection contained in one split.
Other sample values, targets, entity filters and availability checks
are refused before iteration. Shuffle, sharding, padding, drop-last
and resume retain their normal semantics. By default, selected edge pairs
may repeat across native chunks. `deduplicate_edges=True` is available
only for columnar output: it allocates edge arrays to retain the first
occurrence of each pair per relation, preserving example isolation.

#### `Task.preview`

```python
Expand Down
20 changes: 19 additions & 1 deletion packages/experimental/kumo-pytorch/src/kumo/pytorch/assembly.py
Original file line number Diff line number Diff line change
Expand Up @@ -2186,7 +2186,17 @@ def _validate_model_contract(self) -> None:
"inspectable output or an explicit encoder"
)

def _collate(self, values: Any, *, epoch: int, seed: int, model=False, requests=()) -> Batch:
def _collate(
self,
values: Any,
*,
epoch: int,
seed: int,
model=False,
requests=(),
columnar=False,
deduplicate_edges=False,
) -> Batch:
declarations = {
name: value.declarations
for name, value in self._prepared_inputs.items()
Expand All @@ -2206,6 +2216,14 @@ def _collate(self, values: Any, *, epoch: int, seed: int, model=False, requests=
source=prepared.spec.source.name,
node_fields=prepared.spec.representation.node_fields,
)
if columnar:
result[name] = self._source.columnar_graph_batch(
self.sample,
graph,
values.runtime_batch,
deduplicate_edges=deduplicate_edges,
)
continue
if model and getattr(values, "runtime_batch", None) is not None:
result[name] = self._source.sample_graph_batch(
self.sample, graph, prepared.state, values.runtime_batch
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

"""Batch declarations: sources, explicit encodings, arrangements, and layouts."""

from kumo.pytorch.contract.batch.columnar import ColumnarEdges, ColumnarGraph, ColumnarNodes
from kumo.pytorch.contract.batch.encodings import (
Buckets,
Raw,
Expand Down Expand Up @@ -55,6 +56,9 @@
"Buckets",
"CandidateSource",
"CandidateTarget",
"ColumnarEdges",
"ColumnarGraph",
"ColumnarNodes",
"Columns",
"Composite",
"Encode",
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.
# All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Arrow graph outputs for consumers that assemble their own model payloads."""

from __future__ import annotations

from collections.abc import Mapping
from dataclasses import dataclass

import pyarrow as pa


@dataclass(frozen=True)
class ColumnarNodes:
"""Aligned columns for node occurrences, isolated per example.

``ids`` contains primary-key values (index-local row IDs when no primary
key is declared). ``owners`` is a non-null uint32 or uint64 Arrow array of positions
into ``Batch.metadata["sample_ids"]``. The same entity may occur under
several owners. ``features`` is an Arrow table containing exactly the
declared Raw fields, in declaration order and declared dtypes. Arrow
validity bitmaps retain NULLs; no separate masks or filled values are used.
With no declared fields, the table has zero columns; use ``len(ids)`` for
the node count. All three outputs otherwise have the same row order.
"""

ids: pa.Array | pa.ChunkedArray
owners: pa.Array
features: pa.Table


@dataclass(frozen=True)
class ColumnarEdges:
"""Selected edges in declared left-to-right relation orientation.

``index`` is an Arrow table with non-null uint32 or uint64 ``source`` and
``destination`` columns, indexing rows in the corresponding node tables.
Endpoints always belong to the same example. Only edges selected by the
sampling layers are included, never all links among returned nodes.
Reverse traversals are reoriented, including self-relations. Chunks may
repeat a pair selected in both directions. Consumers must normalize these
pairs or request ``Task.batches(..., deduplicate_edges=True)`` to retain
the first occurrence per relation, at the cost of edge-array allocation.
Deduplication uses node positions, preserving separate examples even when
entity IDs repeat. Chunk order is not traversal provenance.
"""

source_type: str
destination_type: str
index: pa.Table


@dataclass(frozen=True)
class ColumnarGraph:
"""CPU Arrow graph buffers emitted by ``Task.batches(output_format="columnar")``.

``nodes`` uses table names and ``edges`` uses declared relation names.
The first ``Batch.batch_size`` rows of ``nodes[root_type]`` are the roots,
in batch order, including repeated entities at distinct prediction times.
Isolated roots and empty declared node/edge types are retained.

Buffers remain valid after advancing or closing the iterator/task. Treat
them as read-only. Compatible native Arrow buffers are shared; declared
dtype restoration may allocate. ``Batch.to`` leaves these CPU buffers in
Arrow form. This output avoids tensor encoding and is intended for callers
that construct their own model representation, such as foundation models.
"""

root_type: str
nodes: Mapping[str, ColumnarNodes]
edges: Mapping[str, ColumnarEdges]
Original file line number Diff line number Diff line change
Expand Up @@ -346,8 +346,10 @@ class Batch:
and ``valid``; token outputs use ``value_ids``, ``position_ids``, validity
and source-event indices, with optional ``field_ids`` and ``field_valid``.
Raw types without a tensor representation stay Arrow arrays in inspectable
output. Metadata carries ``sample_ids``, ``entity_ids``, ``as_of_time``,
``splits``, ``is_padding``, request metadata, fingerprints and candidate
output. Columnar output holds `ColumnarGraph` objects with Arrow feature
tables and selected edge indexes. Metadata carries ``sample_ids``,
``entity_ids``, ``as_of_time``, ``splits``, ``is_padding``, request metadata,
fingerprints and candidate
identities; it is not a route for undeclared features.
"""

Expand Down
Loading
Loading