You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
perf(kumo-pytorch): deduplicate graph edges with a packed key - #383
_first_occurrences deduplicated edge columns with np.unique(pairs, axis=1), which sorts opaque two-column records. When all indices are non-negative and (max source + 1) × (max destination + 1) fits in int64, it now deduplicates one packed int64 key per edge instead. First-occurrence order, dtype and the unchanged-input return are preserved; other inputs use the previous path.
Microbenchmark: int64 pairs with about 20% duplicate columns, median of 9 runs. Python 3.12 on a CPU-only Linux dev host.
Columns
Before
After
Speedup
50k
40.8 ms
5.3 ms
7.7×
100k
90.1 ms
14.0 ms
6.4×
250k
262 ms
38.1 ms
6.9×
500k
501 ms
79.7 ms
6.3×
On a synthetic multi-hop graph workload (4 hops, fanout 10), this function took about 48% of batch time, and loader throughput rose about 1.69× with an equivalent prototype of this change.
Testing Done
New test_first_occurrences_keeps_first_unique_columns_in_order compares against a pure-Python first-occurrence reference for random int64 and repeated uint64 pairs, plus fallback cases (negative values, a packed key that overflows int64, uint64 values above the int64 range) and an empty input.
From packages/experimental/kumo-pytorch, with DiskGraph built from this tree: pytest tests/test_diskgraph_graph_output.py tests/test_columnar_graph_batches.py tests/test_diskgraph_backend.py — 96 passed, 1 skipped (needs optional torch_geometric).
The benchmark script also checks 300 random int64/uint64 inputs against the previous implementation, including dtype, contiguity and returning the input when nothing is removed.
ruff check and ruff format --check pass on the changed files.
Not tested: GPU and end-to-end training. No API, dependency or Rust changes.
np.unique(axis=1) sorts opaque two-column records. When the indices are
non-negative and source * (max destination + 1) + destination fits int64,
deduplicate one packed int64 key per edge instead, keeping the existing
first-occurrence order; other inputs use the previous path.
Co-Authored-By: Claude <noreply@anthropic.com>
Observed issue: on a synthetic multi-hop graph workload, _first_occurrences took about 48% of batch time. Existing behavior: np.unique(pairs, axis=1) sorts each edge as an opaque 16-byte record. Smallest sufficient change: when indices are non-negative and source * (max destination + 1) + destination fits in int64, run np.unique(..., return_index=True) on that packed key; keep the previous path otherwise. Focused check: a new reference-comparison test covering the packed path, each fallback condition and empty input, plus existing graph-output, columnar-batch and DiskGraph backend tests. Stop point: this helper and its test; callers and sampling are unchanged.
No actionable comments were generated in the recent review.
Walkthrough
_first_occurrences now uses encoded scalar pair keys when all values are nonnegative and the computed key width is below 2**63. Otherwise, it retains axis-based pair deduplication. Parameterized tests check first-occurrence order and dtype across varied values and shapes, including empty input.
Merge Risk: Minimal · up to d59cd
This is a localized performance optimization to graph edge deduplication, and the previous implementation remains as a fallback for inputs outside the fast path's safe bounds. No concrete merge-blocking risk was found.
Security architecture risk: Minimal · up to d59cd
The change optimizes an existing graph-edge calculation while preserving its integer-input behavior and fallback. No introduced or worsened security, failure-containment, rollback, or ownership risk was established.
Pre-merge checks: 2 · 1
Failed checks: 1 inconclusive
Check name
Status
Explanation
Resolution
Change Map Design Review
The check assesses the PR conversation comment that starts with <!-- kumo-pr-change-map -->. That comment is the only source for the Change map, and it was not available to me. Its text is not in th…
To complete this check: 1. Make the <!-- kumo-pr-change-map --> PR conversation comment (issuecomment-6964174125) available to the reviewer. Provide its text in the review context, or allow the review to read PR conversation comments. 2. …
Passed checks: 2 passed
Check name
Status
Explanation
Pr Size
The PR size check counts files changed in this pull request. The review-scoped range (refs/coderabbit/pre-merge-check/base 9637e9c..head d59cd2d) contains one commit and changes 2 files, both modifi…
Public Resource Access
The check applies only to external resource references (URLs, repository locations, package source locations) that this PR adds or modifies. The reviewed range (refs/coderabbit/pre-merge-check/base..h…
Full details: Pr Size
Explanation
The PR size check counts files changed in this pull request. The review-scoped range (refs/coderabbit/pre-merge-check/base 9637e9c..head d59cd2d) contains one commit and changes 2 files, both modified (M): packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/diskgraph.py and packages/experimental/kumo-pytorch/tests/test_diskgraph_graph_output.py. The diff stat reports 2 files changed, 27 insertions and 2 deletions. The count is 2, which does not exceed the limit of 40. Provider-reported primary PR: NVIDIA/kumo-engine #383, author Vedaant-J, head perf/first-occurrences-packed-key, base main. The standing FAIL is refuted. It counted 142 files against an older base, 9c9cba5. That commit is an ancestor of the review-selected merge base 9637e9c, so the upstream-synced commits it counted are already in the base. The check uses the review-selected base, which gives a count of 2. Line counts were ignored as required. Prior failure refuted: The prior run counted 142 files against origin/main 9c9cba5. That commit is an ancestor of the review-selected merge base 9637e9c, so the prior count included upstream commits already in the base. The review-scoped diff base..head (9637e9c..d59cd2d) has 1 commit and 2 changed files, both M, with diff stat '2 files changed'. Count 2 is within the limit of 40.
Full details: Public Resource Access
Explanation
The check applies only to external resource references (URLs, repository locations, package source locations) that this PR adds or modifies. The reviewed range (refs/coderabbit/pre-merge-check/base..head) changes two files: packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/diskgraph.py and packages/experimental/kumo-pytorch/tests/test_diskgraph_graph_output.py. Scanning every added and removed line for http(s), git, ssh, file, www, and common registry patterns found no matches. The added lines are code only: a _first_occurrences change using a packed int64 key with a fallback to np.unique(pairs, axis=1), a new numpy import, a _first_occurrences import, and a parametrized test. The PR description contains a link to a PR comment, but that text is not part of the changed code, and the check covers changed code references only. No added or modified external resource reference exists, so the check passes.
Full details: Change Map Design Review
Explanation
The check assesses the PR conversation comment that starts with <!-- kumo-pr-change-map -->. That comment is the only source for the Change map, and it was not available to me. Its text is not in the supplied context. The repository checkout has no copy of it, and I have no tool to read PR conversation comments. The supplied review material contains no CodeRabbit review threads, and it does not include that comment. The PR description links to the comment at #issuecomment-6964174125, but I could not fetch it. Without the map text, I cannot check whether each proposed change is necessary, whether it fits its abstraction layer, whether it preserves behavior, or whether its explanation omits anything material. Code evidence I did verify: - The git range refs/coderabbit/pre-merge-check/base..head changes two files: packages/experimental/kumo-pytorch/src/kumo/pytorch/engines/diskgraph.py and packages/experimental/kumo-pytorch/tests/test_diskgraph_graph_output.py. - In _first_occurrences (diskgraph.py around line 2561), the new packed-key branch runs when lowest >= 0 and (source_max + 1) * width < 2**63. It casts to int64 and deduplicates source * width + destination. The fallback calls the original np.unique(pairs, axis=1, return_index=True). The first-occurrence reorder code is unchanged. - The PR description and the summary both state this behavior. I found no code-level contradiction. The check does not ask for a code defect review, so I did not assess the code beyond this. This verdict does not show that the change map is wrong or correct. It only shows that the required evidence was unavailable.
Resolution
To complete this check: 1. Make the <!-- kumo-pr-change-map --> PR conversation comment (issuecomment-6964174125) available to the reviewer. Provide its text in the review context, or allow the review to read PR conversation comments. 2. Re-run the check. Assess the Change map against the observed failure or requirement, the inspected existing behavior, the smallest sufficient change, the focused verification, and the stopping condition. 3. For the packed-key change, confirm in the map that the fallback covers negative indices, overflow of the (source_max + 1) * width < 2**63 bound, and uint64 values above the int64 range. Also confirm that the map states the Rust and API scope.
Finishing Touches
Fix failing CI checks
Commit to this branch
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 8 remain after this review.
Interaction with #390: this optimization remains useful for relations output paths that call _first_occurrences to produce unique logical edge pairs. #390 adds opt-in sampled output that preserves native directed edges, occurrences and hop counts, and bypasses this deduplication entirely.
The PRs are complementary: #383 accelerates the existing relations contract; #390 serves models requiring the original sampled topology without deduplication/reconstruction. #383’s speedup should not be added to #390’s sampled-path speedup, since that path never calls this helper.
@Victsm yes, 383 and 390 are complementary, I think the relations codepath should just become a view or post process of the sampled output path, but that is for another PR, and even then this PR is useful
could you review this as speedup of the relations path?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_first_occurrencesdeduplicated edge columns withnp.unique(pairs, axis=1), which sorts opaque two-column records. When all indices are non-negative and(max source + 1) × (max destination + 1)fits in int64, it now deduplicates one packed int64 key per edge instead. First-occurrence order, dtype and the unchanged-input return are preserved; other inputs use the previous path.Microbenchmark: int64 pairs with about 20% duplicate columns, median of 9 runs. Python 3.12 on a CPU-only Linux dev host.
On a synthetic multi-hop graph workload (4 hops, fanout 10), this function took about 48% of batch time, and loader throughput rose about 1.69× with an equivalent prototype of this change.
Testing Done
test_first_occurrences_keeps_first_unique_columns_in_ordercompares against a pure-Python first-occurrence reference for random int64 and repeated uint64 pairs, plus fallback cases (negative values, a packed key that overflows int64, uint64 values above the int64 range) and an empty input.packages/experimental/kumo-pytorch, with DiskGraph built from this tree:pytest tests/test_diskgraph_graph_output.py tests/test_columnar_graph_batches.py tests/test_diskgraph_backend.py— 96 passed, 1 skipped (needs optionaltorch_geometric).ruff checkandruff format --checkpass on the changed files.Not tested: GPU and end-to-end training. No API, dependency or Rust changes.
Change map
File-by-file rationale.
Checklist
🤖 Generated with Claude Code