Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,13 @@ history capping/decoding and cross-component event merging are separate work.
A selection spanning multiple split stages uses native stage batches. Each stage
is shuffled independently; batches never mix split stages. The output covers the
same anchors, while its order differs from the previous global Python shuffle.
Native graph output formats partition ranks, pad and drop partial
tails per stage. Refit does not combine partial tails from different splits.
With `drop_last=True`, either topology may drop a split completely when it is
smaller than one full batch. Padding uses a rank-owned anchor from another
selected stage if its current shard is empty; only a rank with no owned anchors
in the whole selection borrows marked padding from another rank. Padding-only
batches retain their source anchor's split and metadata.
Encoder fitting remains a separate stage and is not made streaming here.

The native model path avoids rebuilding a `SampleRequest` for each anchor.
Expand Down
70 changes: 65 additions & 5 deletions packages/experimental/kumo-pytorch/docs/05-training-batches.md
Original file line number Diff line number Diff line change
Expand Up @@ -462,10 +462,70 @@ Native buffers are shared where compatible; restoring declared types may
allocate. All buffers remain valid after iteration and task closure, and
`Batch.to(...)` keeps them as CPU Arrow values.

This format currently supports one graph-only input, explicit Raw encodings,
fixed sample counts, and a selection contained in one split. It refuses
The default relations format supports one graph-only input, explicit Raw encodings,
fixed sample counts, and selections across split stages. It refuses
request-backed tasks, non-DiskGraph engines, other sample values, targets,
entity filters, and availability checks before iteration. Select a single
split explicitly when anchors span several splits. Shuffle, rank sharding,
padding, drop-last, and iterator resume retain their existing behavior.
entity filters, and availability checks before iteration. Shuffle, rank sharding,
padding, drop-last, and iterator resume use the native stage-local behavior below.
Existing sampling limits, including timeless destinations, still apply.

## Sampled graphs and logical relationships

`Graph(node_fields=..., topology="relations")` retains the existing output:
selected relationships in each declared relation's left-to-right orientation.
Model and inspectable output deduplicate pairs; columnar output keeps repeats
unless `deduplicate_edges=True`. Sampling direction and hop provenance are not
part of that graph.

Use `Graph(node_fields=..., topology="sampled")` for a GNN consuming sampled
message-passing topology. The current implementation supports DiskGraph
AnchorTables with fixed-size batching. Topology describes the output layout;
native iteration is the currently supported delivery path, rather than an
intrinsic requirement of sampled graphs. During native iteration, refit may
select several splits; each stage shuffles and batches
independently, without joining partial tails across splits. Drop-last applies
per stage in either topology on this native path, including dropping a whole split smaller than
one batch. Use `drop_last=False` to retain partial tails.
Ranks pad from their own selected anchors when possible; a rank owning no
anchors in any selected stage may borrow only explicitly marked padding.
Model and inspectable output validate explicit `EntitySpec.available_from`
using the same strict-before-cutoff check as the request path. Columnar output
continues to refuse availability declarations before iteration.
Each input keeps its neighborhood's native node order, directed edge
occurrences and edge order. It does not deduplicate, symmetrize, reorder by
shortest-path distance or rebuild a graph in Python. Multiple neighborhoods
share the native anchor scheduler but retain their own topology and counts.

Model/inspectable graph dictionaries retain `nodes` and `edges`. Sampled
`edges` keeps string keys: relations topology uses logical relation names,
while sampled topology uses unique native physical relation names. Sampled edge
values also expose `relation` (the declared logical name), `direction`
(`forward` or `reverse`), `source_type` and `destination_type`. Custom models
can use logical names and direction without parsing internal keys. A Kumo GNN
adapter constructs `(edge["source_type"], key, edge["destination_type"])`
per edge type, passing edge tensors through unchanged. The graph also exposes
`num_sampled_nodes` (per table, including seed count then successive hops) and
`num_sampled_edges` (using the same physical string keys, per sampling hop). These counts
refer to the returned native ordering and are lists, compatible with the
count-padding operations in Kumo's HeteroGNN. Nodes retain IDs, example ownership and
the declared field encodings. Each node occurrence belongs to one anchor;
shared global nodes are not merged across anchors. Edges are in native traversal orientation; a
consumer using PyG's incoming-message convention may transpose endpoints and
rename the corresponding reverse relation without recomputing topology.

Columnar sampled graphs expose the same counts as tuples on `ColumnarGraph`,
preserving Arrow edge buffers. `deduplicate_edges=True` is refused in sampled topology:
coalescing would invalidate its hop accounting. Requests built in Python,
length-aware graph batching and the in-memory engine do not implement this
topology; they refuse it instead of silently reconstructing it. Existing
relation graphs and saved default contracts keep their behavior and identity.
The row-based `ServingAssembler.assemble` also does not implement sampled
topology. Relations-inspectable contracts that still use Python fallback
(for example, those with targets) retain their existing batching behavior;
the stage-local scheduling guarantees above apply to native iteration.

Sampling hops are not shortest-path distances in the final relationship graph.
For example, traversing `A -> B -> C -> D -> A` discovers D on hop three.
Symmetrizing that cycle and recomputing BFS would place D one hop from A,
changing which messages a layer receives. Sampled output retains the original
discovery counts; it does not promise equivalence to that reconstructed graph.
2 changes: 1 addition & 1 deletion packages/experimental/kumo-pytorch/docs/api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ The [guide](../README.md) explains the concepts in order. The behavior spec, wit
| [`Graph`](batches.md#graph) | `kumo.pytorch.api` | Node attributes use the same explicit field encodings as ordinary columns. |
| [`ColumnarGraph`](batches.md#columnargraph) | `kumo.pytorch.api` | CPU Arrow graph buffers emitted by `Task.batches(output_format="columnar")`. |
| [`ColumnarNodes`](batches.md#columnarnodes) | `kumo.pytorch.api` | Aligned columns for node occurrences, isolated per example. |
| [`ColumnarEdges`](batches.md#columnaredges) | `kumo.pytorch.api` | Selected edges in declared left-to-right relation orientation. |
| [`ColumnarEdges`](batches.md#columnaredges) | `kumo.pytorch.api` | Selected edges in the graph topology's declared orientation. |
| [`Columns`](batches.md#columns) | `kumo.pytorch.api` | Keep selected encoded fields as separate typed columns. |
| [`Tokens`](batches.md#tokens) | `kumo.pytorch.api` | Concatenate one row's encoded IDs in the declared field order. |
| [`Composite`](batches.md#composite) | `kumo.pytorch.api` | One category ID for the ordered tuple of scalar component results. |
Expand Down
37 changes: 31 additions & 6 deletions packages/experimental/kumo-pytorch/docs/api/batches.md
Original file line number Diff line number Diff line change
Expand Up @@ -258,14 +258,18 @@ Encode one sampled field, optionally translating IDs into a final space.
*class* · `kumo.pytorch.api.Graph`

```python
class Graph(node_fields: Mapping[str, Fields] = {})
class Graph(
node_fields: Mapping[str, Fields] = {},
topology: Literal['relations', 'sampled'] = 'relations',
)
```

Node attributes use the same explicit field encodings as ordinary columns.

**Parameters**

- `node_fields`: table name to the `Fields` (scalar column encodings of physical columns) emitted for its nodes. Field-name IDs are not supported on graph nodes.
- `topology`: `"relations"` keeps each declared relation's orientation; model and inspectable output deduplicate pairs, while columnar output keeps repeats unless `deduplicate_edges=True`. `"sampled"` preserves DiskGraph's directed edge occurrences, ordering and per-hop counts. It is currently implemented for fixed-size native AnchorTable batches. Selected split stages shuffle and batch independently; it does not reconstruct topology.

### `ColumnarGraph`

Expand All @@ -276,12 +280,21 @@ class ColumnarGraph(
root_type: str,
nodes: Mapping[str, ColumnarNodes],
edges: Mapping[str, ColumnarEdges],
num_sampled_nodes: Mapping[str, tuple[int, ...]] = {},
num_sampled_edges: Mapping[str, tuple[int, ...]] = {},
)
```

CPU Arrow graph buffers emitted by `Task.batches(output_format="columnar")`.

`nodes` uses table names and `edges` uses declared relation names.
`nodes` uses table names. With relations topology, `edges` uses declared
relation names. Sampled topology uses native physical relation-name strings,
retaining edge order and per-hop counts without reorienting or deduplicating
edges. Its edge values expose the logical `relation` and traversal
`direction`, alongside endpoint types. A model adapter can construct
`(edge.source_type, key, edge.destination_type)` without changing arrays.
Counts use the same string keys as edges and align with
the native node and edge ordering, not a newly computed shortest-path order.
The first `Batch.batch_size` rows of `nodes[root_type]` are the roots,
in batch order, including repeated entities at distinct prediction times.
Isolated roots and empty declared node/edge types are retained.
Expand Down Expand Up @@ -316,21 +329,33 @@ the node count. All three outputs otherwise have the same row order.
*class* · `kumo.pytorch.api.ColumnarEdges`

```python
class ColumnarEdges(source_type: str, destination_type: str, index: Table)
class ColumnarEdges(
source_type: str,
destination_type: str,
index: Table,
relation: str | None = None,
direction: Literal['forward', 'reverse'] | None = None,
)
```

Selected edges in declared left-to-right relation orientation.
Selected edges in the graph topology's declared orientation.

`index` is an Arrow table with non-null uint32 or uint64 `source` and
`destination` columns, indexing rows in the corresponding node tables.
Endpoints always belong to the same example. Only edges selected by the
sampling layers are included, never all links among returned nodes.
Reverse traversals are reoriented, including self-relations. Chunks may
In relations topology, reverse traversals are reoriented, including
self-relations. Chunks may
repeat a pair selected in both directions. Consumers must normalize these
pairs or request `Task.batches(..., deduplicate_edges=True)` to retain
the first occurrence per relation, at the cost of edge-array allocation.
Deduplication uses node positions, preserving separate examples even when
entity IDs repeat. Chunk order is not traversal provenance.
entity IDs repeat. Chunk order is not traversal provenance. Sampled
topology instead preserves native direction/order and exposes hop counts
on ColumnarGraph; deduplication is refused for that topology.
Sampled edges also carry the declared logical `relation` and traversal
`direction`. Those optional fields remain unset for relations output,
whose logical identity is already the mapping key and orientation is normalized.

### `Columns`

Expand Down
13 changes: 12 additions & 1 deletion packages/experimental/kumo-pytorch/docs/api/lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -407,6 +407,17 @@ repeating one of its own anchors, and `Batch.metadata["is_padding"]`
marks those rows so a loss can exclude them. `drop_last` applies to
the final batch of each rank's pass, not to every chunk.

Native DiskGraph graph batches keep complete
split-stage batches for refit: shuffle, rank partitioning, padding and
`drop_last` apply separately to each selected split, without mixing
partial tails. A split smaller than one batch may be dropped completely
with `drop_last=True`, independently of graph topology.
An empty stage shard pads from
an anchor the rank owns in another selected split; only a rank owning
no selected anchors borrows marked padding from another rank.
Relations-inspectable contracts using Python fallback (for example,
those with targets) retain their existing scheduling instead.

Order is a pure function of `seed`, `epoch`, `shuffle`, `rank`
and `world_size`; ranks are disjoint before padding. A token budget
(`Batching.max_tokens`) with `world_size > 1` is refused.
Expand All @@ -417,7 +428,7 @@ as Arrow arrays. Both emit the same `Batch` structure.
`"columnar"` emits `ColumnarGraph` inputs with Arrow feature tables
and selected edge indexes, without tensor encoding. It requires
materialized DiskGraph anchors, one graph input with explicit Raw
fields, fixed sample counts, and a selection contained in one split.
fields and fixed sample counts. Both topologies support refit stages.
Other sample values, targets, entity filters and availability checks
are refused before iteration. Shuffle, sharding, padding, drop-last
and resume retain their normal semantics. By default, selected edge pairs
Expand Down
13 changes: 9 additions & 4 deletions packages/experimental/kumo-pytorch/docs/api/samples.md
Original file line number Diff line number Diff line change
Expand Up @@ -489,8 +489,12 @@ class NeighborhoodSpec(layers: tuple[GraphLayer, ...])

Rows belonging to one sampled neighborhood around the entity.

Layers expand in order from the entity. A row or edge reached more than once
appears once per sample. Node attributes are chosen in `BatchSpec` through
Layers expand in order from the entity. Relations topology emits one node
per sample/row and deduplicates edge pairs in model and inspectable output.
Columnar relations output can retain repeated pairs unless
`deduplicate_edges=True`. `Graph(topology="sampled")` retains native
directed edge occurrences, discovery order and hop counts.
Node attributes are chosen in `BatchSpec` through
a `Graph` representation, not here.

### `GraphLayer`
Expand Down Expand Up @@ -520,8 +524,9 @@ class GraphSample(

One bounded relationship traversal used to select neighboring rows.

Emitted edges keep the relation's left-to-right orientation whatever the
traversal direction, self-relations included.
Relations-topology edges keep the declared left-to-right orientation whatever
the traversal direction, self-relations included. Sampled topology retains
the native traversal direction instead.

**Parameters**

Expand Down
11 changes: 8 additions & 3 deletions packages/experimental/kumo-pytorch/src/kumo/pytorch/assembly.py
Original file line number Diff line number Diff line change
Expand Up @@ -2195,7 +2195,11 @@ def _collate(
requests=(),
columnar=False,
deduplicate_edges=False,
encoding_model=None,
) -> Batch:
# Native buffers and the requested encoding surface are independent:
# inspectable output still permits Raw values without tensor encodings.
encoding_model = model if encoding_model is None else encoding_model
declarations = {
name: value.declarations
for name, value in self._prepared_inputs.items()
Expand All @@ -2214,6 +2218,7 @@ def _collate(
graph = SimpleNamespace(
source=prepared.spec.source.name,
node_fields=prepared.spec.representation.node_fields,
topology=prepared.spec.representation.topology,
)
if columnar:
result[name] = self._source.columnar_graph_batch(
Expand Down Expand Up @@ -2252,7 +2257,7 @@ def _collate(
)
elif isinstance(prepared.spec.representation.row, Columns):
result[name] = _column_output(
prepared, rows[name], model=model, state=self._input_state
prepared, rows[name], model=encoding_model, state=self._input_state
)
else:
result[name], produced = _token_output(
Expand All @@ -2262,7 +2267,7 @@ def _collate(
self._input_state,
self._emitted,
entity_ids,
model=model,
model=encoding_model,
packer=getattr(self._source, "pack_token_stream", None) if model else None,
seed=seed,
epoch=epoch,
Expand All @@ -2276,7 +2281,7 @@ def _collate(
self._prepared_inputs,
rows,
values,
model=model,
model=encoding_model,
)
)
if model:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
from __future__ import annotations

from collections.abc import Mapping
from dataclasses import dataclass
from dataclasses import dataclass, field
from typing import Literal

import pyarrow as pa

Expand All @@ -33,30 +34,45 @@ class ColumnarNodes:

@dataclass(frozen=True)
class ColumnarEdges:
"""Selected edges in declared left-to-right relation orientation.
"""Selected edges in the graph topology's declared orientation.

``index`` is an Arrow table with non-null uint32 or uint64 ``source`` and
``destination`` columns, indexing rows in the corresponding node tables.
Endpoints always belong to the same example. Only edges selected by the
sampling layers are included, never all links among returned nodes.
Reverse traversals are reoriented, including self-relations. Chunks may
In relations topology, reverse traversals are reoriented, including
self-relations. Chunks may
repeat a pair selected in both directions. Consumers must normalize these
pairs or request ``Task.batches(..., deduplicate_edges=True)`` to retain
the first occurrence per relation, at the cost of edge-array allocation.
Deduplication uses node positions, preserving separate examples even when
entity IDs repeat. Chunk order is not traversal provenance.
entity IDs repeat. Chunk order is not traversal provenance. Sampled
topology instead preserves native direction/order and exposes hop counts
on ColumnarGraph; deduplication is refused for that topology.
Sampled edges also carry the declared logical ``relation`` and traversal
``direction``. Those optional fields remain unset for relations output,
whose logical identity is already the mapping key and orientation is normalized.
"""

source_type: str
destination_type: str
index: pa.Table
relation: str | None = None
direction: Literal["forward", "reverse"] | None = None


@dataclass(frozen=True)
class ColumnarGraph:
"""CPU Arrow graph buffers emitted by ``Task.batches(output_format="columnar")``.

``nodes`` uses table names and ``edges`` uses declared relation names.
``nodes`` uses table names. With relations topology, ``edges`` uses declared
relation names. Sampled topology uses native physical relation-name strings,
retaining edge order and per-hop counts without reorienting or deduplicating
edges. Its edge values expose the logical ``relation`` and traversal
``direction``, alongside endpoint types. A model adapter can construct
``(edge.source_type, key, edge.destination_type)`` without changing arrays.
Counts use the same string keys as edges and align with
the native node and edge ordering, not a newly computed shortest-path order.
The first ``Batch.batch_size`` rows of ``nodes[root_type]`` are the roots,
in batch order, including repeated entities at distinct prediction times.
Isolated roots and empty declared node/edge types are retained.
Expand All @@ -71,3 +87,5 @@ class ColumnarGraph:
root_type: str
nodes: Mapping[str, ColumnarNodes]
edges: Mapping[str, ColumnarEdges]
num_sampled_nodes: Mapping[str, tuple[int, ...]] = field(default_factory=dict)
num_sampled_edges: Mapping[str, tuple[int, ...]] = field(default_factory=dict)
Loading
Loading