Skip to content

perf(kumo-pytorch): deduplicate graph edges with a packed key - #383

Merged
Vedaant-J merged 1 commit into
mainfrom
perf/first-occurrences-packed-key
Nov 19, 2026
Merged

Vedaant-J merged 1 commit into
mainfrom
perf/first-occurrences-packed-key

Conversation

@Vedaant-J

@Vedaant-J Vedaant-J commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

_first_occurrences deduplicated edge columns with np.unique(pairs, axis=1), which sorts opaque two-column records. When all indices are non-negative and (max source + 1) × (max destination + 1) fits in int64, it now deduplicates one packed int64 key per edge instead. First-occurrence order, dtype and the unchanged-input return are preserved; other inputs use the previous path.

Microbenchmark: int64 pairs with about 20% duplicate columns, median of 9 runs. Python 3.12 on a CPU-only Linux dev host.

Columns Before After Speedup
50k 40.8 ms 5.3 ms 7.7×
100k 90.1 ms 14.0 ms 6.4×
250k 262 ms 38.1 ms 6.9×
500k 501 ms 79.7 ms 6.3×

On a synthetic multi-hop graph workload (4 hops, fanout 10), this function took about 48% of batch time, and loader throughput rose about 1.69× with an equivalent prototype of this change.

Testing Done

  • New test_first_occurrences_keeps_first_unique_columns_in_order compares against a pure-Python first-occurrence reference for random int64 and repeated uint64 pairs, plus fallback cases (negative values, a packed key that overflows int64, uint64 values above the int64 range) and an empty input.
  • From packages/experimental/kumo-pytorch, with DiskGraph built from this tree: pytest tests/test_diskgraph_graph_output.py tests/test_columnar_graph_batches.py tests/test_diskgraph_backend.py — 96 passed, 1 skipped (needs optional torch_geometric).
  • The benchmark script also checks 300 random int64/uint64 inputs against the previous implementation, including dtype, contiguity and returning the input when nothing is removed.
  • ruff check and ruff format --check pass on the changed files.

Not tested: GPU and end-to-end training. No API, dependency or Rust changes.

Change map

File-by-file rationale.

Checklist

  • I listed all the breaking changes introduced by the PR (none)
  • I made sure to update examples and documentation with my changes (if necessary) (not needed)
  • I have added and modified unit tests required to cover changes made by this PR.

🤖 Generated with Claude Code

np.unique(axis=1) sorts opaque two-column records. When the indices are
non-negative and source * (max destination + 1) + destination fits int64,
deduplicate one packed int64 key per edge instead, keeping the existing
first-occurrence order; other inputs use the previous path.

Co-Authored-By: Claude <noreply@anthropic.com>
@Vedaant-J

Copy link
Copy Markdown
Contributor Author

Change map

Observed issue: on a synthetic multi-hop graph workload, _first_occurrences took about 48% of batch time. Existing behavior: np.unique(pairs, axis=1) sorts each edge as an opaque 16-byte record. Smallest sufficient change: when indices are non-negative and source * (max destination + 1) + destination fits in int64, run np.unique(..., return_index=True) on that packed key; keep the previous path otherwise. Focused check: a new reference-comparison test covering the packed path, each fallback condition and empty input, plus existing graph-output, columnar-batch and DiskGraph backend tests. Stop point: this helper and its test; callers and sampling are unchanged.

File Function / scope What changed Why
packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/diskgraph.py _first_occurrences Deduplicate a packed int64 (source, destination) key when it cannot overflow; otherwise keep np.unique(axis=1). Sorting one int64 key is 6–8× faster than sorting two-column records at 50k–500k edges, with the same first-occurrence result.
packages/experimental/kumo-pytorch/tests/test_diskgraph_graph_output.py test_first_occurrences_keeps_first_unique_columns_in_order Add six parametrized cases checking order and dtype against a pure-Python reference. Cover the packed path, the negative, overflow and large-uint64 fallbacks, and empty input.

@copy-pr-bot

copy-pr-bot Bot commented Nov 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Vedaant-J
Vedaant-J temporarily deployed to gha-kumo-workflow-tests November 17, 2026 23:34 — with GitHub Actions Inactive
@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

_first_occurrences now uses encoded scalar pair keys when all values are nonnegative and the computed key width is below 2**63. Otherwise, it retains axis-based pair deduplication. Parameterized tests check first-occurrence order and dtype across varied values and shapes, including empty input.

Merge Risk: Minimal · up to d59cd

This is a localized performance optimization to graph edge deduplication, and the previous implementation remains as a fallback for inputs outside the fast path's safe bounds. No concrete merge-blocking risk was found.

Security architecture risk: Minimal · up to d59cd

The change optimizes an existing graph-edge calculation while preserving its integer-input behavior and fallback. No introduced or worsened security, failure-containment, rollback, or ownership risk was established.

Pre-merge checks: Passed 2 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Change Map Design Review Inconclusive The check assesses the PR conversation comment that starts with <!-- kumo-pr-change-map -->. That comment is the only source for the Change map, and it was not available to me. Its text is not in th… To complete this check: 1. Make the <!-- kumo-pr-change-map --> PR conversation comment (issuecomment-6964174125) available to the reviewer. Provide its text in the review context, or allow the review to read PR conversation comments. 2. …
Passed Passed checks: 2 passed

Check name Status Explanation
Pr Size Passed The PR size check counts files changed in this pull request. The review-scoped range (refs/coderabbit/pre-merge-check/base 9637e9c..head d59cd2d) contains one commit and changes 2 files, both modifi…
Public Resource Access Passed The check applies only to external resource references (URLs, repository locations, package source locations) that this PR adds or modifies. The reviewed range (refs/coderabbit/pre-merge-check/base..h…
Full details: Pr Size

Explanation

The PR size check counts files changed in this pull request. The review-scoped range (refs/coderabbit/pre-merge-check/base 9637e9c..head d59cd2d) contains one commit and changes 2 files, both modified (M): packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/diskgraph.py and packages/experimental/kumo-pytorch/tests/test_diskgraph_graph_output.py. The diff stat reports 2 files changed, 27 insertions and 2 deletions. The count is 2, which does not exceed the limit of 40. Provider-reported primary PR: NVIDIA/kumo-engine #383, author Vedaant-J, head perf/first-occurrences-packed-key, base main. The standing FAIL is refuted. It counted 142 files against an older base, 9c9cba5. That commit is an ancestor of the review-selected merge base 9637e9c, so the upstream-synced commits it counted are already in the base. The check uses the review-selected base, which gives a count of 2. Line counts were ignored as required. Prior failure refuted: The prior run counted 142 files against origin/main 9c9cba5. That commit is an ancestor of the review-selected merge base 9637e9c, so the prior count included upstream commits already in the base. The review-scoped diff base..head (9637e9c..d59cd2d) has 1 commit and 2 changed files, both M, with diff stat '2 files changed'. Count 2 is within the limit of 40.

Full details: Public Resource Access

Explanation

The check applies only to external resource references (URLs, repository locations, package source locations) that this PR adds or modifies. The reviewed range (refs/coderabbit/pre-merge-check/base..head) changes two files: packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/diskgraph.py and packages/experimental/kumo-pytorch/tests/test_diskgraph_graph_output.py. Scanning every added and removed line for http(s), git, ssh, file, www, and common registry patterns found no matches. The added lines are code only: a _first_occurrences change using a packed int64 key with a fallback to np.unique(pairs, axis=1), a new numpy import, a _first_occurrences import, and a parametrized test. The PR description contains a link to a PR comment, but that text is not part of the changed code, and the check covers changed code references only. No added or modified external resource reference exists, so the check passes.

Full details: Change Map Design Review

Explanation

The check assesses the PR conversation comment that starts with &lt;!-- kumo-pr-change-map --&gt;. That comment is the only source for the Change map, and it was not available to me. Its text is not in the supplied context. The repository checkout has no copy of it, and I have no tool to read PR conversation comments. The supplied review material contains no CodeRabbit review threads, and it does not include that comment. The PR description links to the comment at #issuecomment-6964174125, but I could not fetch it. Without the map text, I cannot check whether each proposed change is necessary, whether it fits its abstraction layer, whether it preserves behavior, or whether its explanation omits anything material. Code evidence I did verify: - The git range refs/coderabbit/pre-merge-check/base..head changes two files: packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/diskgraph.py and packages/experimental/kumo-pytorch/tests/test_diskgraph_graph_output.py. - In _first_occurrences (diskgraph.py around line 2561), the new packed-key branch runs when lowest &gt;= 0 and (source_max + 1) * width &lt; 2**63. It casts to int64 and deduplicates source * width + destination. The fallback calls the original np.unique(pairs, axis=1, return_index=True). The first-occurrence reorder code is unchanged. - The PR description and the summary both state this behavior. I found no code-level contradiction. The check does not ask for a code defect review, so I did not assess the code beyond this. This verdict does not show that the change map is wrong or correct. It only shows that the required evidence was unavailable.

Resolution

To complete this check: 1. Make the &lt;!-- kumo-pr-change-map --&gt; PR conversation comment (issuecomment-6964174125) available to the reviewer. Provide its text in the review context, or allow the review to read PR conversation comments. 2. Re-run the check. Assess the Change map against the observed failure or requirement, the inspected existing behavior, the smallest sufficient change, the focused verification, and the stopping condition. 3. For the packed-key change, confirm in the map that the fallback covers negative indices, overflow of the (source_max + 1) * width &lt; 2**63 bound, and uint64 values above the int64 range. Also confirm that the map states the Rust and API scope.

Finishing Touches

Fix failing CI checks

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 9637e9c and d59cd2d.

Files selected for processing: 2

  • packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/diskgraph.py
  • packages/experimental/kumo-pytorch/tests/test_diskgraph_graph_output.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 8 remain after this review.

@Vedaant-J
Vedaant-J marked this pull request as ready for review November 18, 2026 01:00
@Victsm

Victsm commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

Interaction with #390: this optimization remains useful for relations output paths that call _first_occurrences to produce unique logical edge pairs. #390 adds opt-in sampled output that preserves native directed edges, occurrences and hop counts, and bypasses this deduplication entirely.

The PRs are complementary: #383 accelerates the existing relations contract; #390 serves models requiring the original sampled topology without deduplication/reconstruction. #383’s speedup should not be added to #390’s sampled-path speedup, since that path never calls this helper.

@Vedaant-J

Copy link
Copy Markdown
Contributor Author

@Victsm yes, 383 and 390 are complementary, I think the relations codepath should just become a view or post process of the sampled output path, but that is for another PR, and even then this PR is useful

could you review this as speedup of the relations path?

@Victsm Victsm left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Packed-key bounds and the fallback preserve stable first-occurrence order, dtype and unchanged-input behavior. Independent verification passed 97 focused tests and 1,000 randomized equivalence cases. Applies to relations paths that request library deduplication; sampled output remains unaffected.

@Vedaant-J
Vedaant-J merged commit 2db0f9e into main Nov 19, 2026
170 of 171 checks passed
@Vedaant-J
Vedaant-J deleted the perf/first-occurrences-packed-key branch November 19, 2026 03:43

This branch was previously deployed

1 inactive deployment
gha-kumo-workflow-tests — d59cd2d1 Deployed Nov 17, 2026 by Vedaant-J via import #2539
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants