diff --git a/packages/experimental/kumo-pytorch/docs/04-engines-and-data.md b/packages/experimental/kumo-pytorch/docs/04-engines-and-data.md index 48752326f..91923622b 100644 --- a/packages/experimental/kumo-pytorch/docs/04-engines-and-data.md +++ b/packages/experimental/kumo-pytorch/docs/04-engines-and-data.md @@ -286,6 +286,13 @@ history capping/decoding and cross-component event merging are separate work. A selection spanning multiple split stages uses native stage batches. Each stage is shuffled independently; batches never mix split stages. The output covers the same anchors, while its order differs from the previous global Python shuffle. +Native graph output formats partition ranks, pad and drop partial +tails per stage. Refit does not combine partial tails from different splits. +With `drop_last=True`, either topology may drop a split completely when it is +smaller than one full batch. Padding uses a rank-owned anchor from another +selected stage if its current shard is empty; only a rank with no owned anchors +in the whole selection borrows marked padding from another rank. Padding-only +batches retain their source anchor's split and metadata. Encoder fitting remains a separate stage and is not made streaming here. The native model path avoids rebuilding a `SampleRequest` for each anchor. diff --git a/packages/experimental/kumo-pytorch/docs/05-training-batches.md b/packages/experimental/kumo-pytorch/docs/05-training-batches.md index ea4b8f384..2c05a831f 100644 --- a/packages/experimental/kumo-pytorch/docs/05-training-batches.md +++ b/packages/experimental/kumo-pytorch/docs/05-training-batches.md @@ -462,10 +462,70 @@ Native buffers are shared where compatible; restoring declared types may allocate. All buffers remain valid after iteration and task closure, and `Batch.to(...)` keeps them as CPU Arrow values. -This format currently supports one graph-only input, explicit Raw encodings, -fixed sample counts, and a selection contained in one split. It refuses +The default relations format supports one graph-only input, explicit Raw encodings, +fixed sample counts, and selections across split stages. It refuses request-backed tasks, non-DiskGraph engines, other sample values, targets, -entity filters, and availability checks before iteration. Select a single -split explicitly when anchors span several splits. Shuffle, rank sharding, -padding, drop-last, and iterator resume retain their existing behavior. +entity filters, and availability checks before iteration. Shuffle, rank sharding, +padding, drop-last, and iterator resume use the native stage-local behavior below. Existing sampling limits, including timeless destinations, still apply. + +## Sampled graphs and logical relationships + +`Graph(node_fields=..., topology="relations")` retains the existing output: +selected relationships in each declared relation's left-to-right orientation. +Model and inspectable output deduplicate pairs; columnar output keeps repeats +unless `deduplicate_edges=True`. Sampling direction and hop provenance are not +part of that graph. + +Use `Graph(node_fields=..., topology="sampled")` for a GNN consuming sampled +message-passing topology. The current implementation supports DiskGraph +AnchorTables with fixed-size batching. Topology describes the output layout; +native iteration is the currently supported delivery path, rather than an +intrinsic requirement of sampled graphs. During native iteration, refit may +select several splits; each stage shuffles and batches +independently, without joining partial tails across splits. Drop-last applies +per stage in either topology on this native path, including dropping a whole split smaller than +one batch. Use `drop_last=False` to retain partial tails. +Ranks pad from their own selected anchors when possible; a rank owning no +anchors in any selected stage may borrow only explicitly marked padding. +Model and inspectable output validate explicit `EntitySpec.available_from` +using the same strict-before-cutoff check as the request path. Columnar output +continues to refuse availability declarations before iteration. +Each input keeps its neighborhood's native node order, directed edge +occurrences and edge order. It does not deduplicate, symmetrize, reorder by +shortest-path distance or rebuild a graph in Python. Multiple neighborhoods +share the native anchor scheduler but retain their own topology and counts. + +Model/inspectable graph dictionaries retain `nodes` and `edges`. Sampled +`edges` keeps string keys: relations topology uses logical relation names, +while sampled topology uses unique native physical relation names. Sampled edge +values also expose `relation` (the declared logical name), `direction` +(`forward` or `reverse`), `source_type` and `destination_type`. Custom models +can use logical names and direction without parsing internal keys. A Kumo GNN +adapter constructs `(edge["source_type"], key, edge["destination_type"])` +per edge type, passing edge tensors through unchanged. The graph also exposes +`num_sampled_nodes` (per table, including seed count then successive hops) and +`num_sampled_edges` (using the same physical string keys, per sampling hop). These counts +refer to the returned native ordering and are lists, compatible with the +count-padding operations in Kumo's HeteroGNN. Nodes retain IDs, example ownership and +the declared field encodings. Each node occurrence belongs to one anchor; +shared global nodes are not merged across anchors. Edges are in native traversal orientation; a +consumer using PyG's incoming-message convention may transpose endpoints and +rename the corresponding reverse relation without recomputing topology. + +Columnar sampled graphs expose the same counts as tuples on `ColumnarGraph`, +preserving Arrow edge buffers. `deduplicate_edges=True` is refused in sampled topology: +coalescing would invalidate its hop accounting. Requests built in Python, +length-aware graph batching and the in-memory engine do not implement this +topology; they refuse it instead of silently reconstructing it. Existing +relation graphs and saved default contracts keep their behavior and identity. +The row-based `ServingAssembler.assemble` also does not implement sampled +topology. Relations-inspectable contracts that still use Python fallback +(for example, those with targets) retain their existing batching behavior; +the stage-local scheduling guarantees above apply to native iteration. + +Sampling hops are not shortest-path distances in the final relationship graph. +For example, traversing `A -> B -> C -> D -> A` discovers D on hop three. +Symmetrizing that cycle and recomputing BFS would place D one hop from A, +changing which messages a layer receives. Sampled output retains the original +discovery counts; it does not promise equivalence to that reconstructed graph. diff --git a/packages/experimental/kumo-pytorch/docs/api/README.md b/packages/experimental/kumo-pytorch/docs/api/README.md index 9b54cb972..97aecac7b 100644 --- a/packages/experimental/kumo-pytorch/docs/api/README.md +++ b/packages/experimental/kumo-pytorch/docs/api/README.md @@ -73,7 +73,7 @@ The [guide](../README.md) explains the concepts in order. The behavior spec, wit | [`Graph`](batches.md#graph) | `kumo.pytorch.api` | Node attributes use the same explicit field encodings as ordinary columns. | | [`ColumnarGraph`](batches.md#columnargraph) | `kumo.pytorch.api` | CPU Arrow graph buffers emitted by `Task.batches(output_format="columnar")`. | | [`ColumnarNodes`](batches.md#columnarnodes) | `kumo.pytorch.api` | Aligned columns for node occurrences, isolated per example. | -| [`ColumnarEdges`](batches.md#columnaredges) | `kumo.pytorch.api` | Selected edges in declared left-to-right relation orientation. | +| [`ColumnarEdges`](batches.md#columnaredges) | `kumo.pytorch.api` | Selected edges in the graph topology's declared orientation. | | [`Columns`](batches.md#columns) | `kumo.pytorch.api` | Keep selected encoded fields as separate typed columns. | | [`Tokens`](batches.md#tokens) | `kumo.pytorch.api` | Concatenate one row's encoded IDs in the declared field order. | | [`Composite`](batches.md#composite) | `kumo.pytorch.api` | One category ID for the ordered tuple of scalar component results. | diff --git a/packages/experimental/kumo-pytorch/docs/api/batches.md b/packages/experimental/kumo-pytorch/docs/api/batches.md index ce0e227d8..89ac66856 100644 --- a/packages/experimental/kumo-pytorch/docs/api/batches.md +++ b/packages/experimental/kumo-pytorch/docs/api/batches.md @@ -258,7 +258,10 @@ Encode one sampled field, optionally translating IDs into a final space. *class* · `kumo.pytorch.api.Graph` ```python -class Graph(node_fields: Mapping[str, Fields] = {}) +class Graph( + node_fields: Mapping[str, Fields] = {}, + topology: Literal['relations', 'sampled'] = 'relations', +) ``` Node attributes use the same explicit field encodings as ordinary columns. @@ -266,6 +269,7 @@ Node attributes use the same explicit field encodings as ordinary columns. **Parameters** - `node_fields`: table name to the `Fields` (scalar column encodings of physical columns) emitted for its nodes. Field-name IDs are not supported on graph nodes. +- `topology`: `"relations"` keeps each declared relation's orientation; model and inspectable output deduplicate pairs, while columnar output keeps repeats unless `deduplicate_edges=True`. `"sampled"` preserves DiskGraph's directed edge occurrences, ordering and per-hop counts. It is currently implemented for fixed-size native AnchorTable batches. Selected split stages shuffle and batch independently; it does not reconstruct topology. ### `ColumnarGraph` @@ -276,12 +280,21 @@ class ColumnarGraph( root_type: str, nodes: Mapping[str, ColumnarNodes], edges: Mapping[str, ColumnarEdges], + num_sampled_nodes: Mapping[str, tuple[int, ...]] = {}, + num_sampled_edges: Mapping[str, tuple[int, ...]] = {}, ) ``` CPU Arrow graph buffers emitted by `Task.batches(output_format="columnar")`. -`nodes` uses table names and `edges` uses declared relation names. +`nodes` uses table names. With relations topology, `edges` uses declared +relation names. Sampled topology uses native physical relation-name strings, +retaining edge order and per-hop counts without reorienting or deduplicating +edges. Its edge values expose the logical `relation` and traversal +`direction`, alongside endpoint types. A model adapter can construct +`(edge.source_type, key, edge.destination_type)` without changing arrays. +Counts use the same string keys as edges and align with +the native node and edge ordering, not a newly computed shortest-path order. The first `Batch.batch_size` rows of `nodes[root_type]` are the roots, in batch order, including repeated entities at distinct prediction times. Isolated roots and empty declared node/edge types are retained. @@ -316,21 +329,33 @@ the node count. All three outputs otherwise have the same row order. *class* · `kumo.pytorch.api.ColumnarEdges` ```python -class ColumnarEdges(source_type: str, destination_type: str, index: Table) +class ColumnarEdges( + source_type: str, + destination_type: str, + index: Table, + relation: str | None = None, + direction: Literal['forward', 'reverse'] | None = None, +) ``` -Selected edges in declared left-to-right relation orientation. +Selected edges in the graph topology's declared orientation. `index` is an Arrow table with non-null uint32 or uint64 `source` and `destination` columns, indexing rows in the corresponding node tables. Endpoints always belong to the same example. Only edges selected by the sampling layers are included, never all links among returned nodes. -Reverse traversals are reoriented, including self-relations. Chunks may +In relations topology, reverse traversals are reoriented, including +self-relations. Chunks may repeat a pair selected in both directions. Consumers must normalize these pairs or request `Task.batches(..., deduplicate_edges=True)` to retain the first occurrence per relation, at the cost of edge-array allocation. Deduplication uses node positions, preserving separate examples even when -entity IDs repeat. Chunk order is not traversal provenance. +entity IDs repeat. Chunk order is not traversal provenance. Sampled +topology instead preserves native direction/order and exposes hop counts +on ColumnarGraph; deduplication is refused for that topology. +Sampled edges also carry the declared logical `relation` and traversal +`direction`. Those optional fields remain unset for relations output, +whose logical identity is already the mapping key and orientation is normalized. ### `Columns` diff --git a/packages/experimental/kumo-pytorch/docs/api/lifecycle.md b/packages/experimental/kumo-pytorch/docs/api/lifecycle.md index 85aefeb1d..b266cd518 100644 --- a/packages/experimental/kumo-pytorch/docs/api/lifecycle.md +++ b/packages/experimental/kumo-pytorch/docs/api/lifecycle.md @@ -407,6 +407,17 @@ repeating one of its own anchors, and `Batch.metadata["is_padding"]` marks those rows so a loss can exclude them. `drop_last` applies to the final batch of each rank's pass, not to every chunk. +Native DiskGraph graph batches keep complete +split-stage batches for refit: shuffle, rank partitioning, padding and +`drop_last` apply separately to each selected split, without mixing +partial tails. A split smaller than one batch may be dropped completely +with `drop_last=True`, independently of graph topology. +An empty stage shard pads from +an anchor the rank owns in another selected split; only a rank owning +no selected anchors borrows marked padding from another rank. +Relations-inspectable contracts using Python fallback (for example, +those with targets) retain their existing scheduling instead. + Order is a pure function of `seed`, `epoch`, `shuffle`, `rank` and `world_size`; ranks are disjoint before padding. A token budget (`Batching.max_tokens`) with `world_size > 1` is refused. @@ -417,7 +428,7 @@ as Arrow arrays. Both emit the same `Batch` structure. `"columnar"` emits `ColumnarGraph` inputs with Arrow feature tables and selected edge indexes, without tensor encoding. It requires materialized DiskGraph anchors, one graph input with explicit Raw -fields, fixed sample counts, and a selection contained in one split. +fields and fixed sample counts. Both topologies support refit stages. Other sample values, targets, entity filters and availability checks are refused before iteration. Shuffle, sharding, padding, drop-last and resume retain their normal semantics. By default, selected edge pairs diff --git a/packages/experimental/kumo-pytorch/docs/api/samples.md b/packages/experimental/kumo-pytorch/docs/api/samples.md index d2aecae40..0b11b23a9 100644 --- a/packages/experimental/kumo-pytorch/docs/api/samples.md +++ b/packages/experimental/kumo-pytorch/docs/api/samples.md @@ -489,8 +489,12 @@ class NeighborhoodSpec(layers: tuple[GraphLayer, ...]) Rows belonging to one sampled neighborhood around the entity. -Layers expand in order from the entity. A row or edge reached more than once -appears once per sample. Node attributes are chosen in `BatchSpec` through +Layers expand in order from the entity. Relations topology emits one node +per sample/row and deduplicates edge pairs in model and inspectable output. +Columnar relations output can retain repeated pairs unless +`deduplicate_edges=True`. `Graph(topology="sampled")` retains native +directed edge occurrences, discovery order and hop counts. +Node attributes are chosen in `BatchSpec` through a `Graph` representation, not here. ### `GraphLayer` @@ -520,8 +524,9 @@ class GraphSample( One bounded relationship traversal used to select neighboring rows. -Emitted edges keep the relation's left-to-right orientation whatever the -traversal direction, self-relations included. +Relations-topology edges keep the declared left-to-right orientation whatever +the traversal direction, self-relations included. Sampled topology retains +the native traversal direction instead. **Parameters** diff --git a/packages/experimental/kumo-pytorch/src/kumo/pytorch/assembly.py b/packages/experimental/kumo-pytorch/src/kumo/pytorch/assembly.py index a544e100e..b42cf2765 100644 --- a/packages/experimental/kumo-pytorch/src/kumo/pytorch/assembly.py +++ b/packages/experimental/kumo-pytorch/src/kumo/pytorch/assembly.py @@ -2195,7 +2195,11 @@ def _collate( requests=(), columnar=False, deduplicate_edges=False, + encoding_model=None, ) -> Batch: + # Native buffers and the requested encoding surface are independent: + # inspectable output still permits Raw values without tensor encodings. + encoding_model = model if encoding_model is None else encoding_model declarations = { name: value.declarations for name, value in self._prepared_inputs.items() @@ -2214,6 +2218,7 @@ def _collate( graph = SimpleNamespace( source=prepared.spec.source.name, node_fields=prepared.spec.representation.node_fields, + topology=prepared.spec.representation.topology, ) if columnar: result[name] = self._source.columnar_graph_batch( @@ -2252,7 +2257,7 @@ def _collate( ) elif isinstance(prepared.spec.representation.row, Columns): result[name] = _column_output( - prepared, rows[name], model=model, state=self._input_state + prepared, rows[name], model=encoding_model, state=self._input_state ) else: result[name], produced = _token_output( @@ -2262,7 +2267,7 @@ def _collate( self._input_state, self._emitted, entity_ids, - model=model, + model=encoding_model, packer=getattr(self._source, "pack_token_stream", None) if model else None, seed=seed, epoch=epoch, @@ -2276,7 +2281,7 @@ def _collate( self._prepared_inputs, rows, values, - model=model, + model=encoding_model, ) ) if model: diff --git a/packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/batch/columnar.py b/packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/batch/columnar.py index c97be0a3b..3a02bd854 100644 --- a/packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/batch/columnar.py +++ b/packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/batch/columnar.py @@ -7,7 +7,8 @@ from __future__ import annotations from collections.abc import Mapping -from dataclasses import dataclass +from dataclasses import dataclass, field +from typing import Literal import pyarrow as pa @@ -33,30 +34,45 @@ class ColumnarNodes: @dataclass(frozen=True) class ColumnarEdges: - """Selected edges in declared left-to-right relation orientation. + """Selected edges in the graph topology's declared orientation. ``index`` is an Arrow table with non-null uint32 or uint64 ``source`` and ``destination`` columns, indexing rows in the corresponding node tables. Endpoints always belong to the same example. Only edges selected by the sampling layers are included, never all links among returned nodes. - Reverse traversals are reoriented, including self-relations. Chunks may + In relations topology, reverse traversals are reoriented, including + self-relations. Chunks may repeat a pair selected in both directions. Consumers must normalize these pairs or request ``Task.batches(..., deduplicate_edges=True)`` to retain the first occurrence per relation, at the cost of edge-array allocation. Deduplication uses node positions, preserving separate examples even when - entity IDs repeat. Chunk order is not traversal provenance. + entity IDs repeat. Chunk order is not traversal provenance. Sampled + topology instead preserves native direction/order and exposes hop counts + on ColumnarGraph; deduplication is refused for that topology. + Sampled edges also carry the declared logical ``relation`` and traversal + ``direction``. Those optional fields remain unset for relations output, + whose logical identity is already the mapping key and orientation is normalized. """ source_type: str destination_type: str index: pa.Table + relation: str | None = None + direction: Literal["forward", "reverse"] | None = None @dataclass(frozen=True) class ColumnarGraph: """CPU Arrow graph buffers emitted by ``Task.batches(output_format="columnar")``. - ``nodes`` uses table names and ``edges`` uses declared relation names. + ``nodes`` uses table names. With relations topology, ``edges`` uses declared + relation names. Sampled topology uses native physical relation-name strings, + retaining edge order and per-hop counts without reorienting or deduplicating + edges. Its edge values expose the logical ``relation`` and traversal + ``direction``, alongside endpoint types. A model adapter can construct + ``(edge.source_type, key, edge.destination_type)`` without changing arrays. + Counts use the same string keys as edges and align with + the native node and edge ordering, not a newly computed shortest-path order. The first ``Batch.batch_size`` rows of ``nodes[root_type]`` are the roots, in batch order, including repeated entities at distinct prediction times. Isolated roots and empty declared node/edge types are retained. @@ -71,3 +87,5 @@ class ColumnarGraph: root_type: str nodes: Mapping[str, ColumnarNodes] edges: Mapping[str, ColumnarEdges] + num_sampled_nodes: Mapping[str, tuple[int, ...]] = field(default_factory=dict) + num_sampled_edges: Mapping[str, tuple[int, ...]] = field(default_factory=dict) diff --git a/packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/batch/representations.py b/packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/batch/representations.py index 71b29e203..2bbeb7e4c 100644 --- a/packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/batch/representations.py +++ b/packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/batch/representations.py @@ -9,7 +9,7 @@ from collections.abc import Mapping from dataclasses import dataclass from dataclasses import field as dataclass_field -from typing import TypeAlias +from typing import Literal, TypeAlias from kumo.pytorch.contract.batch._common import _mapping from kumo.pytorch.contract.batch.encodings import ( @@ -123,11 +123,21 @@ class Graph: :param node_fields: table name to the ``Fields`` (scalar column encodings of physical columns) emitted for its nodes. Field-name IDs are not supported on graph nodes. + :param topology: ``"relations"`` keeps each declared relation's orientation; + model and inspectable output deduplicate pairs, while columnar output + keeps repeats unless ``deduplicate_edges=True``. ``"sampled"`` preserves + DiskGraph's directed edge occurrences, ordering and per-hop counts. It + is currently implemented for fixed-size native AnchorTable batches. + Selected split stages + shuffle and batch independently; it does not reconstruct topology. """ node_fields: Mapping[str, Fields] = dataclass_field(default_factory=dict) + topology: Literal["relations", "sampled"] = "relations" def __post_init__(self) -> None: + if self.topology not in {"relations", "sampled"}: + raise ValueError("graph topology must be 'relations' or 'sampled'") if any( not isinstance(value, Fields) or not isinstance(value.row, Columns) for value in self.node_fields.values() diff --git a/packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/sample.py b/packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/sample.py index 8ccaeafe2..fa042f9d5 100644 --- a/packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/sample.py +++ b/packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/sample.py @@ -701,8 +701,9 @@ class GraphSample: neighbors at the fanout limit are kept is unspecified (G-GRAPH-001). The DiskGraph engine supports ``"recent"`` only. - Emitted edges keep the relation's left-to-right orientation whatever the - traversal direction, self-relations included. + Relations-topology edges keep the declared left-to-right orientation whatever + the traversal direction, self-relations included. Sampled topology retains + the native traversal direction instead. :param time_column: overrides the destination table's declared clock. :param temporal: ``"inherit"`` filters by the resolved clock; ``"static"`` states that this traversal is deliberately unfiltered. @@ -741,8 +742,12 @@ def __post_init__(self) -> None: class NeighborhoodSpec: """Rows belonging to one sampled neighborhood around the entity. - Layers expand in order from the entity. A row or edge reached more than once - appears once per sample. Node attributes are chosen in ``BatchSpec`` through + Layers expand in order from the entity. Relations topology emits one node + per sample/row and deduplicates edge pairs in model and inspectable output. + Columnar relations output can retain repeated pairs unless + ``deduplicate_edges=True``. ``Graph(topology="sampled")`` retains native + directed edge occurrences, discovery order and hop counts. + Node attributes are chosen in ``BatchSpec`` through a ``Graph`` representation, not here. """ diff --git a/packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/diskgraph.py b/packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/diskgraph.py index 19373dc43..6223fb891 100644 --- a/packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/diskgraph.py +++ b/packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/diskgraph.py @@ -1231,6 +1231,10 @@ def sample_graph( seed: int, epoch: int, ) -> dict[str, Any]: + if getattr(spec, "topology", "relations") == "sampled": + raise ValueError( + "sampled graph topology requires fixed-size native AnchorTable batches" + ) del seed, epoch if self._sampler is None: raise RuntimeError("DiskGraph source is closed") @@ -1275,6 +1279,48 @@ def sample_graph_batch(self, sample, spec, state, runtime): } else: selected = runtime + if spec.topology == "sampled": + graph = self.columnar_graph_batch(sample, spec, selected) + return { + "nodes": { + table: { + "ids": _graph_node_ids( + node.ids.combine_chunks() + if isinstance(node.ids, self._pa.ChunkedArray) + else node.ids + ), + "batch": torch.from_numpy( + node.owners.to_numpy(zero_copy_only=False) + ).long(), + "features": { + name: encoder.encode_arrow(node.features[name]) + for name, encoder in state.get(table, {}).items() + }, + } + for table, node in graph.nodes.items() + }, + "edges": { + key: { + "index": torch.stack( + tuple( + torch.from_numpy(column.to_numpy(zero_copy_only=False)).long() + for column in edge.index.columns + ) + ), + "source_type": edge.source_type, + "destination_type": edge.destination_type, + "relation": edge.relation, + "direction": edge.direction, + } + for key, edge in graph.edges.items() + }, + "num_sampled_nodes": { + key: list(counts) for key, counts in graph.num_sampled_nodes.items() + }, + "num_sampled_edges": { + key: list(counts) for key, counts in graph.num_sampled_edges.items() + }, + } previous = getattr(self, "_runtime_batch", None) self._runtime_batch = selected try: @@ -1337,6 +1383,59 @@ def columnar_graph_batch(self, sample, spec, runtime, *, deduplicate_edges=False owners=owners, features=pa.table({name: columns[name] for name in names}), ) + if spec.topology == "sampled": + if deduplicate_edges: + raise ValueError("sampled graph topology preserves edge occurrences") + + counts = { + entry["edge_type"]["rel"]: tuple(entry["counts"].to_pylist()) + for entry in graph["num_sampled_edges_dict"] + if entry["edge_type"]["rel"] in physical_relations + } + edge_types = { + entry["edge_type"]["rel"]: entry["edge_type"] + for entry in (*graph["edge_index_dict"], *graph["num_sampled_edges_dict"]) + if entry["edge_type"]["rel"] in physical_relations + } + buffers = { + entry["edge_type"]["rel"]: pa.table( + {"source": entry["src"], "destination": entry["dst"]} + ) + for entry in graph["edge_index_dict"] + if entry["edge_type"]["rel"] in physical_relations + } + edges = {} + for physical, kind in edge_types.items(): + logical = physical_relations[physical] + index = buffers.get(physical) + if index is None: + index = pa.table( + { + "source": pa.array([], type=pa.uint64()), + "destination": pa.array([], type=pa.uint64()), + } + ) + edges[physical] = ColumnarEdges( + kind["src"], + kind["dst"], + index, + logical, + "forward" + if physical == self._relation_mapping[logical]["forward"] + else "reverse", + ) + return ColumnarGraph( + sample.entity.table, + nodes, + edges, + { + table: tuple(graph["num_sampled_nodes_dict"][table].to_pylist()) + if table in graph["num_sampled_nodes_dict"] + else (0,) + for table in nodes + }, + counts, + ) parts = defaultdict(list) for edge in graph["edge_index_dict"]: physical = edge["edge_type"]["rel"] @@ -1486,14 +1585,23 @@ def _validate_root_availability( ) -> None: if not requests or spec.entity.available_from is None: return + self._validate_root_ids_availability( + spec, + self._root_ids(spec.entity.table, requests), + [self._cutoff(request.as_of_time) for request in requests], + ) + + def _validate_root_ids_availability(self, spec, node_ids, cutoffs) -> None: + """Use the same availability rule with already resolved native seeds.""" + if not node_ids or spec.entity.available_from is None: + return table = spec.entity.table column = spec.entity.available_from - node_ids = self._root_ids(table, requests) times = self._feature_values(table, node_ids, column) - for request, value in zip(requests, times, strict=True): + for cutoff, value in zip(cutoffs, times, strict=True): if value is None: raise ValueError(f"{table!r} row has no timestamp in {column!r}") - if self._cutoff(value) >= self._cutoff(request.as_of_time): + if self._cutoff(value) >= cutoff: raise ValueError("entity did not exist before the requested sample time") def _root_ids( diff --git a/packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/in_memory/source.py b/packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/in_memory/source.py index bfc79a7dc..73744be93 100644 --- a/packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/in_memory/source.py +++ b/packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/in_memory/source.py @@ -147,6 +147,8 @@ def sample_graph( seed: int, epoch: int, ) -> Any: + if getattr(spec, "topology", "relations") == "sampled": + raise ValueError("sampled graph topology requires native DiskGraph AnchorTable batches") return self._graph.batch( sample.entity.table, self._neighborhood(sample, spec), diff --git a/packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/native_anchors.py b/packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/native_anchors.py index 32bb6ee96..6f3868515 100644 --- a/packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/native_anchors.py +++ b/packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/native_anchors.py @@ -60,7 +60,7 @@ class NativeAnchors: """An AnchorTask backed by a TT-NG seed extension, with bounded reads. Stage order is fixed at materialization. The native stage permutation owns - epoch shuffling; ranks partition that concatenation before marked padding. + epoch shuffling; each stage is partitioned between ranks before marked padding. Original row ordinals, rather than entity/time pairs, supply default IDs. """ @@ -777,6 +777,7 @@ def _typed_batches( drop_last, output_format, deduplicate_edges=False, + padding_anchor=None, ): """One native iterator call per batch; no request objects or re-batching.""" from dataclasses import replace @@ -872,26 +873,34 @@ def runtime_batches(): # batch, including repeated padding seeds and their histories. chosen = ids.to_pylist() + [first] * remaining runtime = next(self._iter(stage, **options, ids=chosen)) - yield runtime, remaining + yield runtime, remaining, stage remaining = 0 else: - yield runtime, 0 + yield runtime, 0, stage if length_planning is not None: self._last_length_planning_stats = iterator.length_planning_stats() if remaining: + padding_stage, padding_id = ( + padding_anchor() + if first is None and padding_anchor is not None + else (stage, 0 if first is None else first) + ) yield ( - next( - self._iter( - stage, **options, ids=[0 if first is None else first] * remaining - ) - ), + next(self._iter(padding_stage, **options, ids=[padding_id] * remaining)), remaining, + padding_stage, ) - for runtime, padded in runtime_batches(): - anchors = self._batch_values(stage, runtime) + for runtime, padded, runtime_stage in runtime_batches(): + anchors = self._batch_values(runtime_stage, runtime) if length_planning is None and anchors.batch_size < size and drop_last: continue + if sample.entity.available_from is not None: + source._validate_root_ids_availability( + sample, + runtime["task"]["seed_node_ids"].to_pylist(), + anchors.evaluation_times, + ) if padded: flags = [False] * (anchors.batch_size - padded) + [True] * padded anchors = replace( @@ -922,11 +931,18 @@ def runtime_batches(): key: runtime["history_merges"][str(i)] for i, key in enumerate(merge_keys) }, ) - yield _surface_padding( + batch = _surface_padding( # For graphs, model=True selects typed source buffers; Raw # fields keep the same Arrow representation as inspection. - assembler._collate(values, epoch=epoch, seed=seed, model=True) + assembler._collate( + values, + epoch=epoch, + seed=seed, + model=True, + encoding_model=output_format == "model", + ) ) + yield batch def _requests(self, stage: Any, runtime: Any) -> list[SampleRequest]: pa = self._pa @@ -1065,12 +1081,11 @@ def validate_columnar(self, assembler, selection): not self._can_batch_inspectable_graph(assembler) or policy.max_tokens is not None or policy.bucket_by_length - or len(self._selection(selection)) > 1 or assembler._source is not self._source ): raise ValueError( "columnar output requires one graph-only input, fixed-size batching, " - "one selected split, and no targets, entity filters or availability" + "and no targets, entity filters or availability" ) for prepared in assembler._prepared_inputs.values(): for fields in prepared.spec.representation.node_fields.values(): @@ -1143,6 +1158,19 @@ def batches( ] policy = assembler.spec.batching fixed = policy.max_tokens is None and not policy.bucket_by_length + sampled = any( + isinstance(prepared.spec.representation, Graph) + and prepared.spec.representation.topology == "sampled" + for prepared in assembler._prepared_inputs.values() + ) + if sampled and (not fixed or assembler._source is not self._source): + raise ValueError( + "sampled graph topology requires fixed-size native AnchorTable batches" + ) + if sampled and deduplicate_edges: + raise ValueError( + "sampled graph topology preserves edge occurrences; do not deduplicate" + ) maximum = batch_size if batch_size is not None else policy.max_samples size = maximum if fixed else chunk neighborhoods = { @@ -1211,16 +1239,43 @@ def assemble(requests: Sequence[SampleRequest], runtime: Any, final: bool): } stages = self._selection(selection) # Preserve mixed-stage ordering and validation outside graph-only contracts. - inspectable_graph = ( - output_format == "inspectable" - and len(stages) == 1 - and self._can_batch_inspectable_graph(assembler) + inspectable_graph = output_format == "inspectable" and self._can_batch_inspectable_graph( + assembler ) if ( - (output_format in {"model", "columnar"} or inspectable_graph) + (output_format in {"model", "columnar"} or inspectable_graph or sampled) and (fixed or output_format == "model" and world_size == 1 and not neighborhoods) and assembler._source is self._source ): + first_owned = None + + def padding_anchor(): + nonlocal first_owned + if first_owned is None: + for candidate in stages: + if candidate["count"] > rank: + runtime = next( + self._iter( + candidate, + batch_size=1, + seed=seed, + epoch=epoch, + shuffle=shuffle, + rank=rank, + world_size=world_size, + ) + ) + first_owned = ( + candidate, + runtime["task"]["tt_stage_node_ids"][0].as_py(), + ) + break + else: + # No owned anchor anywhere: equal steps require marked + # padding from another rank, as for a short single split. + first_owned = (next(stage for stage in stages if stage["count"]), 0) + return first_owned + projections[self._sample.entity.table].extend(entity_columns) for stage in stages: yield from self._typed_batches( @@ -1239,6 +1294,7 @@ def assemble(requests: Sequence[SampleRequest], runtime: Any, final: bool): drop_last=policy.drop_last if drop_last is None else drop_last, output_format=output_format, deduplicate_edges=deduplicate_edges, + padding_anchor=padding_anchor, ) return diff --git a/packages/experimental/kumo-pytorch/src/kumo/pytorch/output.py b/packages/experimental/kumo-pytorch/src/kumo/pytorch/output.py index d3f72ff50..37f692c3c 100644 --- a/packages/experimental/kumo-pytorch/src/kumo/pytorch/output.py +++ b/packages/experimental/kumo-pytorch/src/kumo/pytorch/output.py @@ -67,7 +67,7 @@ def kernel_metadata(value: Any) -> Any: return value result = {name: kernel_metadata(item) for name, item in value.items()} for name, item in value.items(): - if name != "offsets" and not name.endswith("_offsets"): + if not isinstance(name, str) or (name != "offsets" and not name.endswith("_offsets")): continue if not isinstance(item, torch.Tensor) or item.ndim != 1 or not item.numel(): continue diff --git a/packages/experimental/kumo-pytorch/src/kumo/pytorch/spec.py b/packages/experimental/kumo-pytorch/src/kumo/pytorch/spec.py index ba91e6373..105f75dc8 100644 --- a/packages/experimental/kumo-pytorch/src/kumo/pytorch/spec.py +++ b/packages/experimental/kumo-pytorch/src/kumo/pytorch/spec.py @@ -729,6 +729,9 @@ def _encode(value: Any) -> Any: if is_dataclass(value) and not isinstance(value, type): arguments = {item.name: _encode(getattr(value, item.name)) for item in fields(value)} + if isinstance(value, Graph) and value.topology == "relations": + # Keep existing artifacts and default graph fingerprints unchanged. + del arguments["topology"] if type(value).__name__ == "IdSpace" and not value.special_ids: del arguments["special_ids"] if type(value).__name__ == "Vocabulary" and value.values is not None: diff --git a/packages/experimental/kumo-pytorch/src/kumo/pytorch/task.py b/packages/experimental/kumo-pytorch/src/kumo/pytorch/task.py index b9d4a5689..788edfafb 100644 --- a/packages/experimental/kumo-pytorch/src/kumo/pytorch/task.py +++ b/packages/experimental/kumo-pytorch/src/kumo/pytorch/task.py @@ -226,6 +226,17 @@ def batches( marks those rows so a loss can exclude them. ``drop_last`` applies to the final batch of each rank's pass, not to every chunk. + Native DiskGraph graph batches keep complete + split-stage batches for refit: shuffle, rank partitioning, padding and + ``drop_last`` apply separately to each selected split, without mixing + partial tails. A split smaller than one batch may be dropped completely + with ``drop_last=True``, independently of graph topology. + An empty stage shard pads from + an anchor the rank owns in another selected split; only a rank owning + no selected anchors borrows marked padding from another rank. + Relations-inspectable contracts using Python fallback (for example, + those with targets) retain their existing scheduling instead. + Order is a pure function of ``seed``, ``epoch``, ``shuffle``, ``rank`` and ``world_size``; ranks are disjoint before padding. A token budget (`Batching.max_tokens`) with ``world_size > 1`` is refused. @@ -236,7 +247,7 @@ def batches( ``"columnar"`` emits `ColumnarGraph` inputs with Arrow feature tables and selected edge indexes, without tensor encoding. It requires materialized DiskGraph anchors, one graph input with explicit Raw - fields, fixed sample counts, and a selection contained in one split. + fields and fixed sample counts. Both topologies support refit stages. Other sample values, targets, entity filters and availability checks are refused before iteration. Shuffle, sharding, padding, drop-last and resume retain their normal semantics. By default, selected edge pairs diff --git a/packages/experimental/kumo-pytorch/tests/test_columnar_graph_batches.py b/packages/experimental/kumo-pytorch/tests/test_columnar_graph_batches.py index e34fd3a2e..913258328 100644 --- a/packages/experimental/kumo-pytorch/tests/test_columnar_graph_batches.py +++ b/packages/experimental/kumo-pytorch/tests/test_columnar_graph_batches.py @@ -137,7 +137,7 @@ def forbidden(*args, **kwargs): ) -@pytest.mark.parametrize("graph_task", ["mixed", "availability"], indirect=True) +@pytest.mark.parametrize("graph_task", ["availability"], indirect=True) def test_unsupported_columnar_contract_fails_before_iteration(graph_task, monkeypatch): def forbidden(*args, **kwargs): raise AssertionError("unsupported contract opened native iterator") diff --git a/packages/experimental/kumo-pytorch/tests/test_inspectable_graph_batches.py b/packages/experimental/kumo-pytorch/tests/test_inspectable_graph_batches.py index f4b868b10..d635a7ee0 100644 --- a/packages/experimental/kumo-pytorch/tests/test_inspectable_graph_batches.py +++ b/packages/experimental/kumo-pytorch/tests/test_inspectable_graph_batches.py @@ -239,7 +239,7 @@ def forbidden(*args, **kwargs): list(task.batches(output_format="model")) -@pytest.mark.parametrize("graph_task", ["mixed", "availability"], indirect=True) +@pytest.mark.parametrize("graph_task", ["availability"], indirect=True) def test_other_inspectable_contracts_keep_existing_path(graph_task, monkeypatch): task = graph_task with monkeypatch.context() as patch: diff --git a/packages/experimental/kumo-pytorch/tests/test_sampled_graph_topology.py b/packages/experimental/kumo-pytorch/tests/test_sampled_graph_topology.py new file mode 100644 index 000000000..3a5138402 --- /dev/null +++ b/packages/experimental/kumo-pytorch/tests/test_sampled_graph_topology.py @@ -0,0 +1,884 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. +# All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Sampled graphs retain the directed native buffers and hop accounting.""" + +from dataclasses import replace + +import pyarrow as pa +import pyarrow.parquet as pq +import pytest +import torch + +from kumo.pytorch import api +from kumo.pytorch.engines.native_anchors import NativeAnchors +from tests.test_native_anchor_handoff import native_task as native_task + + +@pytest.fixture +def sampled_plan(tmp_path): + schema = api.DataSchema( + tables=( + api.TableSpec( + "people", + "id", + { + "id": api.DType("int64"), + "created": api.DType("timestamp", unit="s", timezone="UTC"), + "value": api.DType("float64"), + "manager": api.DType("int64"), + }, + time_column="created", + ), + api.TableSpec( + "events", + "id", + { + "id": api.DType("int64"), + "owner": api.DType("int64"), + "time": api.DType("timestamp", unit="s", timezone="UTC"), + "value": api.DType("float64"), + }, + time_column="time", + ), + ), + relations=( + api.RelationSpec("owner", "events", "owner", "people", "id"), + api.RelationSpec("manager", "people", "manager", "people", "id"), + ), + ) + timestamp = pa.timestamp("s", tz="UTC") + paths = {name: tmp_path / f"{name}.parquet" for name in ("people", "events", "anchors")} + pq.write_table( + pa.table( + { + "id": [1, 2, 3], + "created": pa.array([0, 0, 0], type=timestamp), + "value": [1.0, 2.0, 3.0], + "manager": [2, 3, 3], + } + ), + paths["people"], + ) + pq.write_table( + pa.table( + { + "id": [10, 11, 12], + "owner": [1, 1, 2], + "time": pa.array([10, 20, 30], type=timestamp), + "value": [10.0, 11.0, 12.0], + } + ), + paths["events"], + ) + pq.write_table( + pa.table( + { + "entity": [1, 1, 3, 2], + "cutoff": pa.array([25, 15, 35, 35], type=timestamp), + "split": ["train"] * 4, + } + ), + paths["anchors"], + ) + sample = api.SampleSpec( + entity=api.EntitySpec("people"), + neighborhoods={ + "g": api.NeighborhoodSpec( + ( + api.GraphLayer((api.GraphSample("owner", 2, direction="reverse"),)), + api.GraphLayer((api.GraphSample("owner", 1),)), + ) + ) + }, + ) + contract = api.DataLoaderSpec( + schema, + sample, + api.BatchSpec( + inputs={ + "graph": api.Input( + api.NeighborhoodSource("g"), + api.Graph( + node_fields={ + name: api.Fields((api.Encode("value", api.Raw()),)) + for name in ("people", "events") + }, + topology="sampled", + ), + None, + ) + }, + batching=api.Batching(max_samples=2), + ), + ) + dataset = api.Dataset( + tables={name: api.Parquet(str(paths[name])) for name in ("people", "events")} + ) + anchors = api.AnchorTable(api.Parquet(str(paths["anchors"])), "entity", "cutoff", split="split") + with api.prepare( + contract, dataset, api.DiskGraphEngine(index=tmp_path / "index", num_threads=2) + ) as plan: + yield plan, anchors + + +@pytest.mark.parametrize("output_format", ["model", "inspectable"]) +def test_sampled_counts_consumed_by_heterognn(sampled_plan, output_format): + HeteroData = pytest.importorskip("torch_geometric.data").HeteroData + HeteroGNN = pytest.importorskip("kumo_core.modeling.gnn.hetero").HeteroGNN + plan, anchors = sampled_plan + with plan.materialize(anchors, fit_split="train") as task: + graph = next(task.batches(split="train", output_format=output_format)).inputs["graph"] + data = HeteroData() + for table, node in graph["nodes"].items(): + data[table].x = node["features"]["value"]["values"].float().reshape(-1, 1) + data[table].num_sampled_nodes = graph["num_sampled_nodes"][table] + for key, edge in graph["edges"].items(): + # Incoming-message convention: reverse endpoints, keeping native + # occurrence order and hop counts without reconstructing topology. + edge_type = (edge["destination_type"], key, edge["source_type"]) + data[edge_type].edge_index = edge["index"].flip(0) + data[edge_type].num_sampled_edges = graph["num_sampled_edges"][key] + model = HeteroGNN(1, 2, metadata=data.metadata(), norm=None).eval() + with torch.no_grad(): + output = model( + data.x_dict, + data.edge_index_dict, + data.num_sampled_nodes_dict, + data.num_sampled_edges_dict, + ) + for table, values in output.items(): + assert values.shape == (graph["num_sampled_nodes"][table][0], 1) + assert torch.isfinite(values).all() + + +@pytest.mark.parametrize("output_format", ["model", "inspectable", "columnar"]) +@pytest.mark.parametrize("batch_size", [1, 2]) +def test_sampled_buffers_match_native_without_reconstruction( + sampled_plan, monkeypatch, output_format, batch_size +): + plan, anchors = sampled_plan + with plan.materialize(anchors, fit_split="train") as task: + native = [] + iterate = task._anchors._iter + + def capture(*args, **kwargs): + for batch in iterate(*args, **kwargs): + native.append(batch) + yield batch + + def forbidden(*args, **kwargs): + pytest.fail("sampled output reconstructed or deduplicated native topology") + + monkeypatch.setattr(task._anchors, "_iter", capture) + monkeypatch.setattr(NativeAnchors, "_requests", forbidden) + monkeypatch.setattr(plan._source, "_runtime_graph_events", forbidden) + monkeypatch.setattr(plan._source, "_graph_output", forbidden) + monkeypatch.setattr("kumo.pytorch.engines.diskgraph._first_occurrences", forbidden) + actual = list( + task.batches(split="train", batch_size=batch_size, output_format=output_format) + ) + assert len(actual) == len(native) == 4 // batch_size + for result, raw in zip(actual, native, strict=True): + graph = result.inputs["graph"] + original = raw["graphs"]["main"] + edges = graph.edges if output_format == "columnar" else graph["edges"] + counts = ( + graph.num_sampled_edges + if output_format == "columnar" + else graph["num_sampled_edges"] + ) + nodes = ( + graph.num_sampled_nodes + if output_format == "columnar" + else graph["num_sampled_nodes"] + ) + count_values = tuple if output_format == "columnar" else list + for table, values in original["num_sampled_nodes_dict"].items(): + if table in {"people", "events"}: + assert type(nodes[table]) is count_values + assert nodes[table] == count_values(values.to_pylist()) + physical = plan._source._sampling_plan(plan.contract.sample.neighborhoods["g"])[1] + assert dict(counts) == { + kind["rel"]: count_values(entry["counts"].to_pylist()) + for entry in original["num_sampled_edges_dict"] + if (kind := entry["edge_type"])["rel"] in physical + } + assert all(type(values) is count_values for values in counts.values()) + selected = { + entry["edge_type"]["rel"]: entry + for entry in original["edge_index_dict"] + if entry["edge_type"]["rel"] in physical + } + kinds = { + entry["edge_type"]["rel"]: entry["edge_type"] + for entry in (*original["edge_index_dict"], *original["num_sampled_edges_dict"]) + if entry["edge_type"]["rel"] in physical + } + assert edges.keys() == kinds.keys() + for key, kind in kinds.items(): + assert isinstance(key, str) + edge = edges[key] + entry = selected.get(key) + left = [] if entry is None else entry["src"].to_pylist() + right = [] if entry is None else entry["dst"].to_pylist() + if output_format == "columnar": + assert edge.index["source"].to_pylist() == left + assert edge.index["destination"].to_pylist() == right + metadata = vars(edge) + else: + assert edge["index"].tolist() == [left, right] + metadata = edge + assert (metadata["source_type"], key, metadata["destination_type"]) == ( + kind["src"], + key, + kind["dst"], + ) + logical = physical[key] + assert metadata["relation"] == logical + assert metadata["direction"] == ( + "forward" + if key == plan._source._relation_mapping[logical]["forward"] + else "reverse" + ) + assert sum(counts[key]) == len(left) + # Both directions survive as separate native relation keys. + assert any( + kind["src"] == "people" and kind["dst"] == "events" for kind in kinds.values() + ) + assert any( + kind["src"] == "events" and kind["dst"] == "people" for kind in kinds.values() + ) + if output_format != "columnar": + assert graph["nodes"]["people"]["batch"].dtype == torch.long + + +def test_sampled_rejects_requests_and_deduplication(sampled_plan): + plan, anchors = sampled_plan + with plan.materialize(anchors, fit_split="train") as task: + requests = task._anchors.fetch(api.ALL_ANCHORS, [0, 1]) + with pytest.raises(ValueError, match="native AnchorTable"): + list(task.batches(requests, output_format="model")) + with pytest.raises(ValueError, match="preserves edge occurrences"): + list(task.batches(output_format="columnar", deduplicate_edges=True)) + + +def test_topology_validation_and_fingerprint(sampled_plan): + plan, _ = sampled_plan + with pytest.raises(ValueError, match="graph topology"): + api.Graph(topology="unknown") + graph = plan.contract.batch.inputs["graph"] + relations = replace(graph, representation=replace(graph.representation, topology="relations")) + changed = replace( + plan.contract, batch=replace(plan.contract.batch, inputs={"graph": relations}) + ) + assert changed.fingerprint != plan.contract.fingerprint + assert api.Graph().topology == "relations" + + +def test_multiple_sampled_neighborhoods_share_schedule(sampled_plan, monkeypatch): + plan, anchors = sampled_plan + contract = plan.contract + neighborhood = contract.sample.neighborhoods["g"] + contract = replace( + contract, + sample=replace(contract.sample, neighborhoods={"g": neighborhood, "h": neighborhood}), + batch=replace( + contract.batch, + inputs={ + name: replace(contract.batch.inputs["graph"], source=api.NeighborhoodSource(source)) + for name, source in (("graph", "g"), ("other", "h")) + }, + ), + ) + with api.prepare(contract, plan.dataset, plan.engine) as bound: + with bound.materialize(anchors, fit_split="train") as task: + + def forbidden(*args, **kwargs): + pytest.fail("multiple sampled neighborhoods used per-request sampling") + + monkeypatch.setattr(bound._source, "sample_graph", forbidden) + monkeypatch.setattr(task._anchors, "_requests", forbidden) + for batch in task.batches(split="train", output_format="model"): + left, right = batch.inputs["graph"], batch.inputs["other"] + assert left["num_sampled_nodes"] == right["num_sampled_nodes"] + assert left["num_sampled_edges"] == right["num_sampled_edges"] + for key in left["edges"]: + torch.testing.assert_close( + left["edges"][key]["index"], right["edges"][key]["index"] + ) + + +def test_sampled_sharding_padding_and_resume(sampled_plan): + plan, anchors = sampled_plan + with plan.materialize(anchors, fit_split="train") as task: + seen = [] + for rank in range(3): + options = dict( + split="train", + output_format="model", + rank=rank, + world_size=3, + shuffle=True, + seed=7, + epoch=1, + ) + batches = task.batches(**options) + first = next(batches) + state = batches.state_dict() + resumed = task.batches(**options) + resumed.load_state_dict(state) + assert list(resumed) == list(batches) == [] + seen.extend( + sid + for sid, pad in zip( + first.metadata["sample_ids"], first.metadata["is_padding"], strict=True + ) + if not pad + ) + graph = first.inputs["graph"] + for key, edge in graph["edges"].items(): + assert edge["index"].shape[1] == sum(graph["num_sampled_edges"][key]) + assert sorted(seen) == list(range(4)) + + +def test_sampled_self_relation_keeps_both_native_directions(sampled_plan, monkeypatch): + plan, anchors = sampled_plan + neighborhood = api.NeighborhoodSpec( + ( + api.GraphLayer((api.GraphSample("manager", 1),)), + api.GraphLayer((api.GraphSample("manager", 2, direction="reverse"),)), + ) + ) + contract = replace( + plan.contract, + sample=replace(plan.contract.sample, neighborhoods={"g": neighborhood}), + batch=replace( + plan.contract.batch, + inputs={ + "graph": replace( + plan.contract.batch.inputs["graph"], + representation=api.Graph( + { + "people": plan.contract.batch.inputs[ + "graph" + ].representation.node_fields["people"] + }, + topology="sampled", + ), + ) + }, + ), + ) + with api.prepare(contract, plan.dataset, plan.engine) as bound: + with bound.materialize(anchors, fit_split="train") as task: + + def forbidden(*args, **kwargs): + pytest.fail("sampled self-relationship was deduplicated") + + monkeypatch.setattr("kumo.pytorch.engines.diskgraph._first_occurrences", forbidden) + batch = next(task.batches(split="train", output_format="model")) + graph = batch.inputs["graph"] + assert len(graph["edges"]) == 2 + assert all(isinstance(key, str) for key in graph["edges"]) + assert all( + edge["source_type"] == edge["destination_type"] == "people" + for edge in graph["edges"].values() + ) + assert {edge["direction"] for edge in graph["edges"].values()} == {"forward", "reverse"} + for key, edge in graph["edges"].items(): + assert sum(graph["num_sampled_edges"][key]) == edge["index"].shape[1] + + +def test_sampled_artifact_retains_mode_and_buffers(sampled_plan, tmp_path): + plan, anchors = sampled_plan + with plan.materialize(anchors, fit_split="train") as task: + artifact = task.to_artifact() + artifact.save(tmp_path / "artifact") + loaded = api.Artifact.load(tmp_path / "artifact") + assert loaded.contract.batch.inputs["graph"].representation.topology == "sampled" + assert loaded.contract.fingerprint == artifact.contract.fingerprint + held = next(task.batches(split="train", output_format="columnar")) + pairs = {key: edge.index.to_pydict() for key, edge in held.inputs["graph"].edges.items()} + assert pairs == { + key: edge.index.to_pydict() for key, edge in held.inputs["graph"].edges.items() + } + + +def test_sampled_resume_retains_unconsumed_topology(sampled_plan): + plan, anchors = sampled_plan + with plan.materialize(anchors, fit_split="train") as task: + options = dict(split="train", batch_size=1, output_format="model", shuffle=True, seed=7) + batches = task.batches(**options) + next(batches) + state = batches.state_dict() + expected = list(batches) + resumed = task.batches(**options) + resumed.load_state_dict(state) + actual = list(resumed) + assert len(actual) == len(expected) == 3 + for left, right in zip(actual, expected, strict=True): + assert left.metadata["sample_ids"] == right.metadata["sample_ids"] + left_graph, right_graph = left.inputs["graph"], right.inputs["graph"] + torch.testing.assert_close(left_graph["nodes"], right_graph["nodes"]) + assert left_graph["num_sampled_nodes"] == right_graph["num_sampled_nodes"] + assert left_graph["num_sampled_edges"] == right_graph["num_sampled_edges"] + assert left_graph["edges"].keys() == right_graph["edges"].keys() + for key, edge in left_graph["edges"].items(): + other = right_graph["edges"][key] + assert (edge["source_type"], edge["destination_type"]) == ( + other["source_type"], + other["destination_type"], + ) + torch.testing.assert_close(edge["index"], other["index"]) + + +@pytest.mark.parametrize("encoding", [api.ZScore(), api.Buckets(bins=3)]) +def test_sampled_reuses_relation_field_encodings(sampled_plan, encoding): + plan, anchors = sampled_plan + outputs = [] + for topology in ("relations", "sampled"): + use = plan.contract.batch.inputs["graph"] + use = replace( + use, + representation=api.Graph( + { + table: api.Fields((api.Encode("value", encoding),)) + for table in ("people", "events") + }, + topology=topology, + ), + ) + contract = replace(plan.contract, batch=replace(plan.contract.batch, inputs={"graph": use})) + with api.prepare(contract, plan.dataset, plan.engine) as bound: + with bound.materialize(anchors, fit_split="train") as task: + outputs.append( + [ + batch.inputs["graph"]["nodes"] + for batch in task.batches(split="train", output_format="model") + ] + ) + torch.testing.assert_close(outputs[0], outputs[1]) + + +@pytest.mark.parametrize("output_format", ["model", "inspectable"]) +@pytest.mark.parametrize("cutoff", [-1, 0, 1]) +@pytest.mark.parametrize( + "topology,requests", [("sampled", False), ("relations", False), ("relations", True)] +) +def test_sampled_enforces_entity_availability( + sampled_plan, tmp_path, output_format, cutoff, topology, requests +): + plan, _ = sampled_plan + contract = replace( + plan.contract, + sample=replace( + plan.contract.sample, entity=api.EntitySpec("people", available_from="created") + ), + batch=replace( + plan.contract.batch, + inputs={ + "graph": replace( + plan.contract.batch.inputs["graph"], + representation=replace( + plan.contract.batch.inputs["graph"].representation, topology=topology + ), + ) + }, + ), + ) + path = tmp_path / "availability.parquet" + pq.write_table( + pa.table({"entity": [1], "cutoff": pa.array([cutoff], type=pa.timestamp("s", tz="UTC"))}), + path, + ) + anchors = api.AnchorTable(api.Parquet(str(path)), "entity", "cutoff") + with api.prepare(contract, plan.dataset, plan.engine) as bound: + with bound.materialize(anchors) as task: + arguments = {} + if requests: + arguments["requests"] = task._anchors.fetch(api.ALL_ANCHORS, [0]) + if cutoff <= 0: + with pytest.raises(ValueError, match="did not exist"): + next(task.batches(output_format=output_format, **arguments)) + else: + assert next(task.batches(output_format=output_format, **arguments)).inputs["graph"] + + +def _refit_anchors(tmp_path, counts): + splits = [name for name, count in zip(("train", "valid", "test"), counts) for _ in range(count)] + path = tmp_path / "refit.parquet" + pq.write_table( + pa.table( + { + "sid": list(range(len(splits))), + "entity": [1] * len(splits), + "cutoff": pa.array([35] * len(splits), type=pa.timestamp("s", tz="UTC")), + "split": splits, + } + ), + path, + ) + return api.AnchorTable( + api.Parquet(str(path)), "entity", "cutoff", sample_id="sid", split="split" + ), splits + + +@pytest.mark.parametrize("output_format", ["model", "inspectable", "columnar"]) +@pytest.mark.parametrize("world_size", [1, 2, 4]) +@pytest.mark.parametrize("counts", [(1, 2), (5, 3, 1)]) +def test_sampled_native_refit_coverage_padding_and_resume( + sampled_plan, tmp_path, monkeypatch, output_format, world_size, counts +): + plan, _ = sampled_plan + anchors, splits = _refit_anchors(tmp_path, counts) + with plan.materialize(anchors, fit_split=api.ALL_ANCHORS) as task: + + def forbidden(*args, **kwargs): + pytest.fail("native refit rebuilt per-anchor requests or resampled in Python") + + monkeypatch.setattr(task._anchors, "_requests", forbidden) + monkeypatch.setattr(plan._source, "sample_graph", forbidden) + orders = [] + for epoch in (0, 1, 0): + real = [] + epoch_order = [] + step_counts = [] + for rank in range(world_size): + options = dict( + output_format=output_format, + batch_size=2, + shuffle=True, + seed=19, + epoch=epoch, + rank=rank, + world_size=world_size, + drop_last=False, + ) + iterator = task.batches(**options) + first = next(iterator) + state = iterator.state_dict() + suffix = list(iterator) + resumed = task.batches(**options) + resumed.load_state_dict(state) + restored = list(resumed) + assert [b.metadata["sample_ids"] for b in restored] == [ + b.metadata["sample_ids"] for b in suffix + ] + batches = [first, *suffix] + step_counts.append(len(batches)) + owned, padded = set(), [] + for batch in batches: + row_splits = [] + for sid, pad, meta in zip( + batch.metadata["sample_ids"], + batch.metadata["is_padding"].tolist(), + batch.metadata["request_metadata"], + strict=True, + ): + original = int(sid.split("#padding")[0]) if pad else sid + assert meta["split"] == splits[original] + row_splits.append(meta["split"]) + (padded if pad else real).append(original) + if not pad: + owned.add(original) + epoch_order.append((rank, sid)) + assert len(set(row_splits)) == 1 + if owned: + assert set(padded) <= owned + else: + assert all(b.metadata["is_padding"].all() for b in batches) + assert len(set(step_counts)) == 1 + assert sorted(real) == list(range(len(splits))) + orders.append(epoch_order) + assert orders[0] == orders[2] + + +@pytest.mark.parametrize("output_format", ["model", "inspectable", "columnar"]) +@pytest.mark.parametrize("world_size", [1, 2]) +def test_sampled_refit_drop_last_is_stage_local(sampled_plan, tmp_path, output_format, world_size): + plan, _ = sampled_plan + anchors, splits = _refit_anchors(tmp_path, (5, 4)) + with plan.materialize(anchors, fit_split=api.ALL_ANCHORS) as task: + steps = [] + for rank in range(world_size): + batches = list( + task.batches( + output_format=output_format, + batch_size=2, + world_size=world_size, + rank=rank, + drop_last=True, + ) + ) + steps.append(len(batches)) + assert all(len(b.metadata["sample_ids"]) == 2 for b in batches) + assert all(len({splits[sid] for sid in b.metadata["sample_ids"]}) == 1 for b in batches) + assert len(set(steps)) == 1 + assert steps[0] == sum(((count + world_size - 1) // world_size) // 2 for count in (5, 4)) + + +@pytest.mark.parametrize("output_format", ["model", "inspectable", "columnar"]) +@pytest.mark.parametrize("counts", [(1, 2), (3, 3)]) +@pytest.mark.parametrize("world_size", [1, 2, 4]) +@pytest.mark.parametrize("drop_last", [False, True]) +def test_topologies_share_native_refit_schedule( + sampled_plan, tmp_path, monkeypatch, output_format, counts, world_size, drop_last +): + plan, _ = sampled_plan + anchors, splits = _refit_anchors(tmp_path, counts) + schedules = [] + for topology in ("relations", "sampled"): + graph = plan.contract.batch.inputs["graph"] + contract = replace( + plan.contract, + batch=replace( + plan.contract.batch, + inputs={ + "graph": replace( + graph, representation=replace(graph.representation, topology=topology) + ) + }, + ), + ) + with api.prepare(contract, plan.dataset, plan.engine) as bound: + with bound.materialize(anchors, fit_split=api.ALL_ANCHORS) as task: + + def forbidden(*args, **kwargs): + pytest.fail("native graph batches reconstructed anchor requests") + + monkeypatch.setattr(task._anchors, "_requests", forbidden) + ranks = [] + for rank in range(world_size): + batches = list( + task.batches( + output_format=output_format, + batch_size=2, + shuffle=True, + seed=7, + epoch=1, + rank=rank, + world_size=world_size, + drop_last=drop_last, + ) + ) + ranks.append( + [ + ( + b.metadata["sample_ids"], + b.metadata["splits"], + b.metadata["is_padding"].tolist(), + b.metadata["request_metadata"], + ) + for b in batches + ] + ) + if drop_last: + assert len(batches) == sum( + ((n + world_size - 1) // world_size) // 2 for n in counts + ) + else: + owned, padded = set(), [] + for batch in batches: + for sid, pad, meta in zip( + batch.metadata["sample_ids"], + batch.metadata["is_padding"].tolist(), + batch.metadata["request_metadata"], + strict=True, + ): + original = int(sid.split("#padding")[0]) if pad else sid + assert meta["split"] == splits[original] + if pad: + padded.append(original) + else: + owned.add(original) + if owned: + assert set(padded) <= owned + assert len({len(rank) for rank in ranks}) == 1 + schedules.append(ranks) + assert schedules[0] == schedules[1] + + +def test_sampled_hops_preserve_discovery_in_a_cycle(tmp_path): + timestamp = api.DType("timestamp", unit="s", timezone="UTC") + schema = api.DataSchema( + ( + api.TableSpec( + "people", + "id", + {"id": api.DType("int64"), "manager": api.DType("int64"), "created": timestamp}, + time_column="created", + ), + ), + (api.RelationSpec("manager", "people", "manager", "people", "id"),), + ) + path = tmp_path / "cycle.parquet" + pq.write_table( + pa.table( + { + "id": [1, 2, 3, 4], + "manager": [2, 3, 4, 1], + "created": pa.array([0] * 4, type=timestamp.to_arrow()), + } + ), + path, + ) + contract = api.DataLoaderSpec( + schema, + api.SampleSpec( + api.EntitySpec("people"), + neighborhoods={ + "g": api.NeighborhoodSpec( + tuple(api.GraphLayer((api.GraphSample("manager", 1),)) for _ in range(4)) + ) + }, + ), + api.BatchSpec( + inputs={ + "graph": api.Input(api.NeighborhoodSource("g"), api.Graph(topology="sampled"), None) + }, + batching=api.Batching(max_samples=1), + ), + ) + anchor_path = tmp_path / "anchor.parquet" + pq.write_table(pa.table({"entity": [1], "cutoff": [1]}), anchor_path) + with api.prepare( + contract, + api.Dataset({"people": api.Parquet(str(path))}), + api.DiskGraphEngine(index=tmp_path / "index", num_threads=2), + ) as plan: + with plan.materialize( + api.AnchorTable(api.Parquet(str(anchor_path)), "entity", "cutoff") + ) as task: + graph = next(task.batches(output_format="model")).inputs["graph"] + assert graph["nodes"]["people"]["ids"].tolist() == [1, 2, 3, 4] + assert graph["num_sampled_nodes"]["people"] == [1, 1, 1, 1, 0] + edge = next(edge for edge in graph["edges"].values() if edge["index"].numel()) + assert edge["index"].tolist() == [[0, 1, 2, 3], [1, 2, 3, 0]] + # Adding reverse edges creates a shortcut from root A to D. Recovering BFS + # depths would move D from discovery hop three to hop one. + distances = {0: 0} + pairs = edge["index"].T.tolist() + for hop in range(4): + frontier = {node for node, distance in distances.items() if distance == hop} + for left, right in pairs + [pair[::-1] for pair in pairs]: + if left in frontier: + distances.setdefault(right, hop + 1) + assert distances[3] == 1 + + +@pytest.mark.parametrize("topology", ["relations", "sampled"]) +def test_native_graph_inspectable_keeps_raw_string_inputs_and_targets( + native_task, tmp_path, topology +): + original, _, anchors = native_task + inputs = dict(original.contract.batch.inputs) + graph = inputs["graph"] + inputs["graph"] = replace( + graph, representation=replace(graph.representation, topology=topology) + ) + inputs["entity"] = api.Input( + api.EntitySource(), api.Fields((api.Encode("raw_id", api.Raw()),)), api.Separate() + ) + contract = replace( + original.contract, + sample=replace( + original.contract.sample, + entity_values={"raw_id": api.Field("id")}, + targets={"label": api.ExternalTarget(api.DType("string"))}, + ), + batch=replace(original.contract.batch, inputs=inputs), + ) + rows = pq.read_table(anchors.source.path) + rows = rows.set_column( + rows.schema.get_field_index("label"), + "label", + pa.array([f"label{i}" for i in range(len(rows))]), + ) + path = tmp_path / "string-labels.parquet" + pq.write_table(rows, path) + anchors = replace(anchors, source=api.Parquet(str(path))) + with api.prepare( + contract, + original.dataset, + api.DiskGraphEngine(index=tmp_path / f"raw-{topology}", num_threads=1), + ) as plan: + with plan.materialize(anchors, fit_split="train") as task: + batches = list(task.batches(split="train", output_format="inspectable")) + assert [ + x for b in batches for x in b.inputs["entity"]["raw_id"]["values"].to_pylist() + ] == ["u1", "u1", "u2", "u1"] + assert [x for b in batches for x in b.targets["label"]["values"].to_pylist()] == [ + "label0", + "label1", + "label3", + "label5", + ] + with pytest.raises(ValueError, match="Raw string"): + next(task.batches(split="train", output_format="model")) + + +@pytest.mark.parametrize("topology", ["relations", "sampled"]) +@pytest.mark.parametrize("output_format", ["model", "inspectable"]) +def test_native_graph_iterators_restore_runtime_before_yield(sampled_plan, topology, output_format): + original, anchors = sampled_plan + graph = original.contract.batch.inputs["graph"] + contract = replace( + original.contract, + batch=replace( + original.contract.batch, + inputs={ + "graph": replace( + graph, representation=replace(graph.representation, topology=topology) + ) + }, + ), + ) + with api.prepare(contract, original.dataset, original.engine) as plan: + with plan.materialize(anchors, fit_split="train") as task: + source = task._assembler._source + first = task.batches(split="train", output_format=output_format) + second = task.batches(split="train", output_format=output_format) + next(first) + assert getattr(source, "_runtime_batch", None) is None + next(second) + assert getattr(source, "_runtime_batch", None) is None + list(first) + list(second) + assert getattr(source, "_runtime_batch", None) is None + + +def test_sampled_count_only_edge_type_has_empty_buffer(sampled_plan, monkeypatch): + plan, anchors = sampled_plan + with plan.materialize(anchors, fit_split="train") as task: + source = task._assembler._source + original = source.columnar_graph_batch + removed = [] + + def count_only(sample, spec, runtime, **kwargs): + graph = dict(runtime["graphs"]["main"]) + _, selected = source._sampling_plan(sample.neighborhoods[spec.source]) + entry = next( + item + for item in graph["num_sampled_edges_dict"] + if item["edge_type"]["rel"] in selected + ) + physical = entry["edge_type"]["rel"] + removed.append(physical) + graph["num_sampled_edges_dict"] = [ + {**item, "counts": pa.array([0] * len(item["counts"]), type=item["counts"].type)} + if item["edge_type"]["rel"] == physical + else item + for item in graph["num_sampled_edges_dict"] + ] + graph["edge_index_dict"] = [ + item for item in graph["edge_index_dict"] if item["edge_type"]["rel"] != physical + ] + return original(sample, spec, {**runtime, "graphs": {"main": graph}}, **kwargs) + + monkeypatch.setattr(source, "columnar_graph_batch", count_only) + graph = next(task.batches(output_format="columnar")).inputs["graph"] + assert removed[0] in graph.edges + assert graph.edges[removed[0]].index.num_rows == 0 + assert sum(graph.num_sampled_edges[removed[0]]) == 0