Skip to content

Expose Arrow graph batches for foundation-model consumers - #365

Merged
am-porter merged 9 commits into
mainfrom
codex/columnar-graph-batches
Nov 17, 2026
Merged

am-porter merged 9 commits into
mainfrom
codex/columnar-graph-batches

Conversation

@am-porter

@am-porter am-porter commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

A foundation-model user may need Arrow tables to assemble model-specific payloads. The inspectable Graph output first encodes sampled nodes into tensor/mask dictionaries, requiring a columnar consumer to reconstruct Arrow columns. Add Task.batches(output_format="columnar") to return public ColumnarGraph, ColumnarNodes, and ColumnarEdges directly from native Arrow buffers.

  • Preserve declared feature types and NULLs, example ownership, isolated/repeated roots, and selected edge endpoints in declared relation orientation.
  • Reuse the native anchor iterator and existing metadata, padding, sharding and resume logic. Refuse unsupported contracts before iteration.
  • Keep native edge chunks by default. Offer deduplicate_edges=True for unique pairs per relation in first-occurrence order, preserving example isolation. Document the additional allocation/work and consumer normalization contract. Existing inspectable/model outputs remain unchanged.

Stacked on #340. No native/Rust changes.

Benchmark

A foundation-model payload consumer was evaluated using the same indexes, native binary and frozen 10k-seed batches for both output formats. Each format received one warmup and five alternating measured draws. Columnar measurements use the default deduplicate_edges=False; the opt-in deduplication path has not been benchmarked. Medians include the consumer's complete payload assembly and JSON encoding; preparation and validation are excluded.

Workload #340 inspectable Columnar Latency reduction
Commerce, 2 hops 1.632s 1.016s 37.8%
Commerce, 4 hops 20.084s 13.305s 33.8%
Streaming, 2 hops 0.681s 0.465s 31.8%
Delivery, 2 hops 1.943s 1.361s 29.9%

AMD EPYC 9654P, 16 sampler threads. All measured payloads are byte-identical across formats. Full 10k native/Python encoder parity and independent DuckDB source validation on 128 distributed anchors per workload pass. These measurements retain the native binary used for the #340 baseline and isolate the output-format change.

Testing

237 focused engine/API tests pass, including 56 columnar tests, existing graph direction/uniqueness/temporal/native-anchor suites, and generated API-reference checks. Tests cover both deduplication settings, stable uniqueness within/across chunks, empty/singleton edges, uint32/uint64 indexes, repeated entities in separate examples, reverse/self-relations, and rejection of changed deduplication settings on resume. Buffer-address and closed-task lifetime tests cover Arrow sharing; sharding, padding, shuffle and rejection tests cover the public contract. Consumer integration checks additionally cover asymmetric sampling radii, shuffled-anchor mapping and payload equality. Ruff and diff checks pass.

Change map

#365 (comment)

@copy-pr-bot

copy-pr-bot Bot commented Nov 16, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@am-porter
am-porter temporarily deployed to gha-kumo-workflow-tests November 16, 2026 22:59 — with GitHub Actions Inactive
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Adds a columnar output format to graph batches. Eligible materialized DiskGraph anchors produce Arrow-backed node features and relation-oriented edge indexes. The API supports optional per-relation edge deduplication and validates columnar requests before iteration. New public data types describe the output, and tests and documentation cover its behavior.

Sequence Diagram(s)

sequenceDiagram
  participant Task as Task.batches
  participant Anchors as NativeAnchors
  participant Collate as PreparedBatchPlan._collate
  participant DiskGraph as DiskGraphSource.columnar_graph_batch
  Task->>Anchors: pass columnar request and deduplication option
  Anchors->>Anchors: validate request
  Anchors->>Collate: collate eligible runtime batch
  Collate->>DiskGraph: assemble columnar graph data
  DiskGraph-->>Collate: return Arrow-backed ColumnarGraph
Loading
Merge Risk: Minimal · up to 0f550

This adds an opt-in columnar Arrow output format for eligible DiskGraph batches. Existing output formats are unchanged and unsupported columnar requests are refused before iteration. No concrete merge-blocking risk was found. Native buffer ownership after the task is closed could not be inspected, but a PR test covers readability after close.

Security architecture risk: Minimal · up to 0f550

The opt-in format changes how selected graph data is returned. Unsupported configurations are rejected before iteration. No introduced or worsened security, ownership, or recovery failure was established in the inspected paths.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The demonstrated exposure is a local caller receiving graph data from its already-bound task and source. The inspected route adds a representation choice, not a new remote endpoint or identity grant. Downstream serialization and deployment-specific access controls were not assessed.

Trust Boundaries and Controls

  • observed — Validation occurs at the public task entrypoint and again in native batching. Graph-only validation excludes auxiliary collation inputs, while columnar conversion receives runtime buffers explicitly instead of installing the shared runtime-feature scope used by the other branch.
Pre-merge checks: Passed 2 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Change Map Design Review Inconclusive The check reviews the PR conversation comment that starts with <!-- kumo-pr-change-map -->. That comment is not available to this review. The PR description links to it (issuecomment-6931953284), bu… Provide the text of the PR conversation comment that starts with <!-- kumo-pr-change-map --> (comment 6931953284 on NVIDIA/kumo-engine PR #365), or enable a PR comment retrieval capability for this check. Then re-run the check. The re-run…
Passed Passed checks: 2 passed

Check name Status Explanation
Pr Size Passed The PR size check fails only when the changed file count exceeds 40. git diff --name-only refs/coderabbit/pre-merge-check/base refs/coderabbit/pre-merge-check/head lists 13 changed files: 11 modifie…
Public Resource Access Passed The check covers only external resource references (URLs, repository locations, package source locations) that this PR adds or modifies. I scanned all added lines in the reviewed range (refs/coderabbi…
Full details: Pr Size

Explanation

The PR size check fails only when the changed file count exceeds 40. git diff --name-only refs/coderabbit/pre-merge-check/base refs/coderabbit/pre-merge-check/head lists 13 changed files: 11 modified and 2 added (contract/batch/columnar.py and tests/test_columnar_graph_batches.py). There are no renames or deletions. Thirteen is well under the threshold of 40, so the check passes. Line counts were ignored, as the check requires.

Full details: Public Resource Access

Explanation

The check covers only external resource references (URLs, repository locations, package source locations) that this PR adds or modifies. I scanned all added lines in the reviewed range (refs/coderabbit/pre-merge-check/base..head) for http(s), www, GitHub/GitLab, object-storage, file, ssh, registry, index-URL, wheel and tarball references. None were found. The only added link targets are repository-relative Markdown anchors in packages/experimental/kumo-pytorch/docs/api/README.md: batches.md#columnargraph, batches.md#columnarnodes and batches.md#columnaredges. These resolve inside the same repository and need no external access. The PR description's change-map link points to a PR comment, but it is PR metadata, not a reference added to the repository, so it is outside this check's scope. Other added content is Python code, tests and documentation prose without external references.

Full details: Change Map Design Review

Explanation

The check reviews the PR conversation comment that starts with &lt;!-- kumo-pr-change-map --&gt;. That comment is not available to this review. The PR description links to it (issuecomment-6931953284), but no tool here fetches PR conversation comments. The only tools available are the primary-author lookup, a read-only checkout, and the review-scoped Git refs. A search of the checkout and git history found the marker only in docs/development/review.md, which describes the expected comment. It does not contain the change map itself. The changed-file inventory confirms 13 files changed (git diff --name-status over refs/coderabbit/pre-merge-check/base..head). That inventory shows file coverage only and cannot evaluate the design criteria: necessity, abstraction layer, preserved behavior, observed failure, smallest sufficient change, focused verification, or stopping condition. The review produced zero actionable findings, but that does not evaluate the change map. Without the comment, the design review cannot be assessed. Missing evidence cannot establish PASS.

Resolution

Provide the text of the PR conversation comment that starts with &lt;!-- kumo-pr-change-map --&gt; (comment 6931953284 on NVIDIA/kumo-engine PR #365), or enable a PR comment retrieval capability for this check. Then re-run the check. The re-run should test each Change map row against the diff. Required checks: whether the observed failure or release requirement is stated; whether existing behavior was inspected; whether each change is necessary and placed at the right layer (for example, the native-anchor validation in native_anchors.py versus task.py, and the DiskGraph-only restriction); whether the deduplication path and its first-occurrence ordering are described accurately; and whether the benchmark-only or opt-in behavior is stated as material omission or scope.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 28335cd and 0f550da.

Files selected for processing: 13

  • packages/experimental/kumo-pytorch/docs/05-training-batches.md
  • packages/experimental/kumo-pytorch/docs/api/README.md
  • packages/experimental/kumo-pytorch/docs/api/batches.md
  • packages/experimental/kumo-pytorch/docs/api/lifecycle.md
  • packages/experimental/kumo-pytorch/src/kumo/pytorch/assembly.py
  • packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/batch/__init__.py
  • packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/batch/columnar.py
  • packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/batch/spec.py
  • packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/diskgraph.py
  • packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/native_anchors.py
  • packages/experimental/kumo-pytorch/src/kumo/pytorch/task.py
  • packages/experimental/kumo-pytorch/tests/api_reference.py
  • packages/experimental/kumo-pytorch/tests/test_columnar_graph_batches.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@am-porter

am-porter commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Change map

Requested behavior: expose graph columns that a foundation-model user can consume directly. Previously, native Arrow buffers became inspectable tensor/mask dictionaries that a columnar payload consumer must convert back into Arrow. The change adds one explicitly bounded public output format and preserves existing outputs. Consumers can normalize pairs during payload assembly, or request stable per-relation edge deduplication with deduplicate_edges=True; the default retains native chunks. Verification: 237 engine/API tests, downstream consumer integration tests, same-index byte equality on four 10k workloads, native/Python encoding parity, and independent source checks on 128 anchors per workload. Stop condition: a usable public format and measured end-to-end improvement; broader sampling and mixed-target support are outside this PR.

File Function / scope What changed Why
packages/experimental/kumo-pytorch/docs/05-training-batches.md Arrow graph guide Document public usage, ownership, edge orientation, duplicate chunks, optional stable deduplication and supported scope. Let payload consumers use the format correctly.
packages/experimental/kumo-pytorch/docs/api/README.md Generated reference Regenerate the API catalog and affected type/lifecycle documentation. Keep documentation synchronized with authored exports and docstrings.
packages/experimental/kumo-pytorch/docs/api/batches.md Generated reference Regenerate the API catalog and affected type/lifecycle documentation. Keep documentation synchronized with authored exports and docstrings.
packages/experimental/kumo-pytorch/docs/api/lifecycle.md Generated reference Regenerate the API catalog and affected type/lifecycle documentation. Keep documentation synchronized with authored exports and docstrings.
packages/experimental/kumo-pytorch/src/kumo/pytorch/assembly.py _collate Dispatch the columnar graph builder and deduplication option while retaining common batch metadata. Avoid duplicating metadata and padding semantics.
packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/batch/__init__.py Public batch exports Export the three graph output types through kumo.pytorch.api. Keep consumers on the supported public API.
packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/batch/columnar.py ColumnarGraph, ColumnarNodes, ColumnarEdges Define public Arrow graph output types and ownership/lifetime/edge contracts, including optional uniqueness and its allocation cost. A foundation-model user needs selected graph columns without tensor/mask encoding.
packages/experimental/kumo-pytorch/src/kumo/pytorch/contract/batch/spec.py Batch documentation Describe ColumnarGraph inputs. Make output shapes explicit.
packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/diskgraph.py columnar_graph_batch Expose aligned feature, owner and selected-edge Arrow buffers; restore declared feature types and orient reverse/self relations. Optionally reuse stable edge deduplication after combining logical-relation chunks. Eliminate per-node scalar/tensor graph construction without changing sampled links.
packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/native_anchors.py validate_columnar, _typed_batches, batches Validate supported contracts early and route native anchor buffers and the deduplication option to columnar assembly. Preserve ordering, padding and selection while avoiding request reconstruction.
packages/experimental/kumo-pytorch/src/kumo/pytorch/task.py Task.batches, _anchor_batches Accept and validate the columnar format, and columnar-only boolean deduplication option, including resume configuration fingerprints. Expose the format through the existing public task lifecycle.
packages/experimental/kumo-pytorch/tests/api_reference.py API_PAGES Include the new output types in generated reference pages. Keep authored docstrings as the reference source.
packages/experimental/kumo-pytorch/tests/test_columnar_graph_batches.py Columnar regressions Test parity, Arrow buffer sharing and lifetime, sharding/padding/shuffle/resume, reverse/self-relations, stable deduplication within/across chunks, repeated entities across examples, both endpoint widths, empty/singleton results and rejection paths. Catch semantic and allocation regressions at the public output boundary.

@Victsm

Victsm commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

The Arrow output makes sense when the downstream RFM repo owns model-batch construction. Could you clarify the separate edge-deduplication tradeoff?

Existing graph output deduplicates logical edge pairs, while ColumnarEdges permits repeated pairs across chunks. Does RFM need those repetitions, or does its payload builder already deduplicate them? Is skipping deduplication here primarily to preserve native Arrow buffers and avoid allocations?

If RFM needs unique edges, please identify where that normalization occurs and confirm the reported full-payload timings include it. This would clarify the consumer contract and whether each downstream model otherwise needs to implement the same topology normalization.

@am-porter
am-porter force-pushed the codex/kumo-pytorch-inspectable-performance branch from 04bdf1c to 0fffd73 Compare November 17, 2026 00:05
@am-porter
am-porter temporarily deployed to gha-kumo-workflow-tests November 17, 2026 04:15 — with GitHub Actions Inactive
@am-porter

Copy link
Copy Markdown
Contributor Author

The Arrow output makes sense when the downstream RFM repo owns model-batch construction. Could you clarify the separate edge-deduplication tradeoff?

Existing graph output deduplicates logical edge pairs, while ColumnarEdges permits repeated pairs across chunks. Does RFM need those repetitions, or does its payload builder already deduplicate them? Is skipping deduplication here primarily to preserve native Arrow buffers and avoid allocations?

If RFM needs unique edges, please identify where that normalization occurs and confirm the reported full-payload timings include it. This would clarify the consumer contract and whether each downstream model otherwise needs to implement the same topology normalization.

A foundation-model consumer already normalizes these pairs while building its final payload (it maps nodes to their parents so duplicates get assigned to the same map slot repeatedly), so deduplicating upstream adds redundant work. That downstream normalization is included in the benchmark timings.

I’ve added deduplicate_edges=True for consumers that want unique edges directly, and to make it more explicit that deduplication is not happening by default. It retains the first occurrence of each pair per relation and preserves example isolation. The default remains False to retain native Arrow chunks; the comments now make the normalization contract and opt-in allocation cost explicit.

@am-porter
am-porter temporarily deployed to gha-kumo-workflow-tests November 17, 2026 17:34 — with GitHub Actions Inactive
Base automatically changed from codex/kumo-pytorch-inspectable-performance to main November 17, 2026 17:42
@am-porter
am-porter temporarily deployed to gha-kumo-workflow-tests November 17, 2026 17:45 — with GitHub Actions Inactive
@am-porter

Copy link
Copy Markdown
Contributor Author

/ok to test 0f550da

@Victsm Victsm left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the native Arrow output path and the updated edge-deduplication contract. Native AnchorTable iteration, example isolation, edge orientation and existing output defaults are preserved; the new format is explicitly guarded to its supported scope. Earlier deduplication question is addressed, regression coverage and CI are green, and I found no remaining correctness blocker.

@am-porter
am-porter merged commit b9a00e5 into main Nov 17, 2026
124 checks passed
@am-porter
am-porter deleted the codex/columnar-graph-batches branch November 17, 2026 22:32

This branch was previously deployed

1 inactive deployment
gha-kumo-workflow-tests — 0f550da9 Deployed Nov 17, 2026 by am-porter via import #2487
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants