Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 92 additions & 0 deletions packages/diskgraph/diskgraph-pybind/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,98 @@ The fluent materializer windows the selected destination frame and rejects
temporal intermediate tables. This does not change PQL's first-hop window policy
or add arbitrary multi-hop temporal support.

### Columnar requests for external loaders

Callers that own an anchor table can keep its columns out of Python row objects:

```python
import json
from collections import defaultdict

import diskgraph
import pyarrow as pa

engine = diskgraph.SamplerEngine("/path/to/index", num_threads=4)
# Bound each request upstream; this is one batch, not the entire anchor table.
keys = pa.array(["u1", "u2", "u1"])
cutoffs = pa.array([1609545600, 1609632000, 1609718400], type=pa.int64())

# The caller supplies its normal SamplingPlan JSON.
events = engine.sample_from_keys(
{"users": keys}, plan_json, fused=True,
seed_timestamps={"users": cutoffs},
seed_timestamp_unit=diskgraph.TimeUnit.Seconds,
swap_edge_direction=False,
)
batches = defaultdict(list)
for event in events:
if event["type"] != "stats":
kind = event["type"]
type_id = event["rel_id"] if kind == "edge" else event["type_id"]
batches[kind, type_id].append(event["batch"])
tables = {key: pa.Table.from_batches(parts) for key, parts in batches.items()}
```

`sample_from_keys` resolves the supplied typed Arrow primary keys and invokes
the existing sampler. `fused=True` also fetches features in that same request.
It returns the same node, edge, feature and stats events as `sample`; it does
not change fanouts, randomness, temporal selection or edge direction. Repeated
keys retain separate anchor positions and aligned cutoffs. A missing key raises
`DiskgraphDataError` with its type and batch offset before sampling starts;
keys are never silently dropped. Invalid/null keys keep `lookup_node_ids`'s
existing errors. No second key map or per-anchor Python request is constructed.

For reusable/materialized dense IDs, resolve once and use `sample` directly:

```python
ids = engine.lookup_node_ids("users", keys, output_format="model")
# Missing keys are nulls in the uint64 array. Apply the caller's missing-key
# policy to all aligned columns before sampling; sample rejects null IDs.
events = engine.sample(
{"users": ids}, plan_json, fused=True,
seed_timestamps={"users": cutoffs},
seed_timestamp_unit=diskgraph.TimeUnit.Seconds,
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
swap_edge_direction=False,
)
```

`lookup_node_ids` defaults to the existing Python list result;
`output_format="model"` returns nullable Arrow `uint64`. Both preserve input
order, repetitions and missing-key positions. IDs belong to the opened index;
persisted IDs need the caller's snapshot provenance checks.

`sample`, `fetch_features`, `canonical_node_mask`, `query_temporal` and
`query_temporal_lengths` accept dense IDs as Arrow `uint64` or nonnegative
`int64` arrays as well as their existing Python sequences. Cutoffs accept Arrow
`int64` or the existing sequences. Arrays are imported through Arrow's C data
interface, including sliced arrays, without Python scalar iteration. Pass one
array per column; combine a `ChunkedArray` explicitly. Nulls and incompatible
physical types are rejected, including timestamp-typed arrays: cast cutoffs
explicitly to the API's declared integer time unit. For `sample` and
`sample_from_keys`, the existing default unit is **nanoseconds**; temporal
queries always take **seconds**. Supplied sampling cutoff columns must match
their seed type and length; omitted types retain the existing unbounded behavior.
The engine still copies these input buffers into owned native vectors for the
request lifetime. This is not an end-to-end zero-copy guarantee.

Keep the returned Arrow batches through downstream gathering/encoding. In node
events, `id_a` is the owner seed position and `id_b` is the dense graph ID;
edge endpoints are local positions in the corresponding node tables, subject
to `swap_edge_direction`. Features retain the existing node-row alignment.
An FM payload builder can gather these columns with Arrow and convert only at
its wire boundary; a tensor consumer can use numeric Arrow/NumPy buffers.
Check nulls/dtypes and copy when mutable tensor storage is required. Python
event dictionaries remain per batch, not per node or feature value.

For relational deep learning (RDL) loaders, this supplies the native boundary
for passing anchor columns without Python scalar lists and combining key lookup,
sampling and feature retrieval in one call. Loader integration also requires
wiring materialized anchors into these calls, vectorizing permutations while
preserving their exact order, and retaining columnar topology/features through
tensor assembly.
No task policy, model dependency, full-task materialization, native iterator,
or FM-specific wire schema is added here.

### Direct temporal rows

`SamplerEngine.query_temporal(seed_nodes, seed_timestamps, plan_json,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES.
# All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Matched request-boundary probe; not an RDL epoch or model benchmark.

Run with an installed extension from this checkout:
python diskgraph-pybind/python/tests/bench_columnar_requests.py --output result.json

Uses a fresh synthetic index, one native thread, warm caches, batch 512, and
alternating path order. Ingestion, lookup precomputation and exact Arrow-output
checks are outside timing. All paths consume the same fused Arrow event stream.
"""

import argparse
from collections import defaultdict
import hashlib
import json
from pathlib import Path
import statistics
import tempfile
import time

import diskgraph
import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq


def fixture(root):
users, per_user = 2048, 16
pq.write_table(pa.table({"id": np.arange(users, dtype=np.int64)}), root / "users.parquet")
pq.write_table(pa.table({
"id": np.arange(users * per_user, dtype=np.int64),
"owner": np.repeat(np.arange(users, dtype=np.int64), per_user),
"amount": np.arange(users * per_user, dtype=np.float64) / 4,
}), root / "events.parquet")
tables = []
for name, columns in [("users", [("id", "int64")]),
("events", [("id", "int64"), ("owner", "int64"), ("amount", "float64")])]:
table = {"name": name, "primary_key": "id",
"columns": [{"name": col, "ingested_dtype": dtype} for col, dtype in columns],
"source": {"type": "parquet", "paths": [str(root / f"{name}.parquet")]}}
if name == "events":
table["fkeys"] = [{"src_column": "owner", "dst_table": "users", "reverse": {}}]
tables.append(table)
diskgraph.ingest(json.dumps({"dataset_name": "columnar_probe", "num_shards": 1,
"tables": tables}), str(root / "index"), col_stats=False)
return diskgraph.SamplerEngine(str(root / "index"), random_seed=41, num_threads=1)


def consume(events, verify):
batches = defaultdict(list)
rows = 0
for event in events:
if event["type"] == "stats":
continue
rows += event["batch"].num_rows
if verify:
batches[event["type"], event.get("rel_id", event.get("type_id"))].append(event["batch"])
digest = hashlib.sha256()
for key, parts in sorted(batches.items()):
table = pa.Table.from_batches(parts).combine_chunks()
sink = pa.BufferOutputStream()
with pa.ipc.new_stream(sink, table.schema) as writer:
writer.write_table(table)
digest.update(str(key).encode())
digest.update(sink.getvalue())
return rows, digest.hexdigest()


def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--anchors", type=int, nargs="+", default=[32000, 400000])
parser.add_argument("--repeats", type=int, default=5)
args = parser.parse_args()
results = []
with tempfile.TemporaryDirectory(prefix="columnar-probe-") as tmp:
engine = fixture(Path(tmp))
relation = next(r["name"] for r in json.loads(engine.get_type_mapping())["edge_relations"]
if r["src_type"] == "users")
plan = json.dumps({"layers": [{"fanouts": {relation: {
"fanout": 16, "mode": "deterministic_uniform", "absent_key_policy": "uniform_fallback",
}}, "default_fanout": None}]})
paths = ["lookup_lists", "lookup_fused", "materialized_lists", "materialized_arrays"]
for count in args.anchors:
keys = pa.array(np.arange(count, dtype=np.int64) % 2048)
ids = engine.lookup_node_ids("users", keys, output_format="model")
cutoffs = pa.array(np.full(count, 1700000000, dtype=np.int64))
def run(path, verify=False):
result = []
for start in range(0, count, 512):
chunk = keys.slice(start, 512)
times = cutoffs.slice(start, 512)
options = dict(fused=True, seed_timestamp_unit=diskgraph.TimeUnit.Seconds)
if path == "lookup_fused":
events = engine.sample_from_keys({"users": chunk}, plan,
seed_timestamps={"users": times}, **options)
else:
roots = (engine.lookup_node_ids("users", chunk) if path == "lookup_lists"
else ids.slice(start, 512))
if path == "materialized_lists":
roots = roots.to_pylist()
if path != "materialized_arrays":
times = times.to_pylist()
events = engine.sample({"users": roots}, plan,
seed_timestamps={"users": times}, **options)
result.append(consume(events, verify))
return result
# Every batch on every path is checked, including per-anchor ownership.
expected = run(paths[0], verify=True)
for path in paths[1:]:
assert run(path, verify=True) == expected, path
for path in paths:
run(path) # untimed warmup
samples = {path: [] for path in paths}
for repeat in range(args.repeats):
for path in paths if repeat % 2 == 0 else paths[::-1]:
start = time.perf_counter()
run(path)
samples[path].append(time.perf_counter() - start)
results.append({"anchors": count, "batch_size": 512, "exact_output_parity": True,
"seconds": samples,
"medians": {p: statistics.median(v) for p, v in samples.items()}})
args.output.write_text(json.dumps(results, indent=2) + "\n")
print(json.dumps(results[-1]), flush=True)


if __name__ == "__main__":
main()
Loading
Loading