Skip to content

[kumo-pytorch] Vectorize exact positional anchor shuffling - #334

Draft
am-porter wants to merge 3 commits into
mainfrom
codex/kumo-pytorch-vector-shuffle
Draft

am-porter wants to merge 3 commits into
mainfrom
codex/kumo-pytorch-vector-shuffle

Conversation

@am-porter

@am-porter am-porter commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Carry the remaining shuffle optimization from closed rdl-platform #140 into kumo.pytorch.

Explicit-request and positional-anchor paths still evaluate the default permutation one row at a time. Evaluate the same integer mixing and cycle walking in chunks of at most 4,096 positions, preserving exact seed/epoch/rank order. Tiny batches and very large integer domains retain scalar evaluation.

Merged #282 already provides native anchor/model batch delivery; its native stage scheduling stays unchanged. This PR needs neither #164 nor its columnar sampling APIs. No public API, output ordering, dependencies, guarantees or native code change.

The 4,096-position bound applies only to temporary permutation evaluation in shard_positions. Explicit-request delivery in task.py::request_batches and the assembly.py::batches path still materialize the full epoch order with list(...). Streaming these consumers is a separate optional follow-up.

Testing Done

  • Review-fix focused suite: 378 passed (tests/test_anchor_permutation.py tests/test_anchor_tables.py), including 45 new exact-parity cases that forbid scalar fallback: 4,096-position windows at the start, middle and end of domains covering every half-width from 17 through 31. Existing chunk/power-of-two boundaries, huge-domain fallback, bounded evaluation, rank sequences and task delivery also pass. Changed-file Ruff check/format and git diff --check pass.
  • Original implementation validation (not rerun for this test/docstring-only follow-up): package Ruff check/format, repository license-header check, source/wheel builds and git diff --check passed.
  • In-memory conformance: 521 passed, 19 failed because the local environment lacks DiskGraph, now required by encoder statistics fitting. Unmodified base 9d155981 has the identical 19 failures. Full native conformance was not run locally.
  • Local macOS arm64 / Python 3.12 / NumPy 2.5.4 permutation probe: 400,000 anchors, three alternating rounds, exact parity each round; median 2.551 s scalar vs 0.145 s vector (~17.5×). This measures permutation evaluation only, not native loading or training.

Change map

File-by-file change map.

Checklist

  • No breaking changes.
  • Existing documented behavior is preserved.
  • Regression coverage added for the changed path.

@copy-pr-bot

copy-pr-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@am-porter
am-porter temporarily deployed to gha-kumo-workflow-tests November 13, 2026 21:03 — with GitHub Actions Inactive
@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Pre-merge checks: Passed 2 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Change Map Design Review Warning (previously flagged, unaddressed) Summary: The change map (comment 6864860037, marker <!-- kumo-pr-change-map -->) is present. Its design is sound at the core: vectorizing the existing permutation in anchors.py is a necessary,… Correct the change map and tests as follows: 1. Scope the bounded-storage claim. In the anchors.py row and the Scope/verification text, state that chunking bounds only shard_positions evaluation. State that explicit-request delivery…
Passed Passed checks: 2 passed

Check name Status Explanation
Pr Size Passed PR size check: the pull request changes 2 files, which is at or below the threshold of 40, so the check passes. Evidence: - The PR head is de41932. Its branch merges current origin/main (9c9cba5) vi…
Public Resource Access Passed The check applies only to external resource references added or modified by this PR. The PR patch was taken against the merge base with origin/main (9c9cba5), which is the branch's point of divergenc…
Full details: Pr Size

Explanation

PR size check: the pull request changes 2 files, which is at or below the threshold of 40, so the check passes. Evidence: - The PR head is de41932. Its branch merges current origin/main (9c9cba5) via merge commit 4e82cbf. - git diff --name-status origin/main...HEAD (the PR-style diff against the base branch) lists exactly 2 files: - M packages/experimental/kumo-pytorch/src/kumo/pytorch/anchors.py - A packages/experimental/kumo-pytorch/tests/test_anchor_permutation.py - The PR-authored commits are 6c9a912 (implementation, 2 files) and de41932 (test follow-up, same 2 files). Caveat: the PR description names 9d15598 as its base. A raw diff from 9d15598 to HEAD shows 197 files. That count comes from the stale base, because main advanced and was merged into the branch. It does not reflect the PR's own changes. The provider-style diff against the base branch is 2 files, so the check passes on that basis. Line counts were ignored, as the check requires. The PR author is am-porter (provider-reported); author identity does not affect this check.

Full details: Public Resource Access

Explanation

The check applies only to external resource references added or modified by this PR. The PR patch was taken against the merge base with origin/main (9c9cba5), which is the branch's point of divergence from main. That diff touches two files: packages/experimental/kumo-pytorch/src/kumo/pytorch/anchors.py (+32/-2) and packages/experimental/kumo-pytorch/tests/test_anchor_permutation.py (+103). A search of every added line in that patch found no URL, git+ or ssh source, GitHub/PyPI/GitLab/Bitbucket location, or package source reference. The only links (rdl-platform #140, #282, #164, and the change-map comment) appear in the PR description, not in changed repository files, and are not needed to build, test, or use the changed code. No added external resource reference exists, so the check passes. Note: a diff against the older base 9d15598 shows many unrelated UI files. Those changes come from main history that the branch merged in, not from this PR.

Full details: Change Map Design Review

Explanation

Summary: The change map (comment 6864860037, marker &lt;!-- kumo-pr-change-map --&gt;) is present. Its design is sound at the core: vectorizing the existing permutation in anchors.py is a necessary, correctly placed change, and it preserves the scalar sequence. The map still omits material behavior and overstates its bounded-storage and verification claims. That is a substantive issue under this check. Verified as accurate - Scope and problem: the scalar per-anchor loop in shard_positions existed before the PR (anchors.py lines 444–449 before this change). The PR adds _Permutation.many (lines 387–410) and a chunked loop (lines 445–449). many applies the same _mix rounds and cycle walk as __call__ (lines 371–385). - Thresholds: scalar fallback below 64 positions and when _half &gt; 31 matches the code. - Chunk bound: step = 4096 * world_size yields at most 4096 positions per rank per chunk. This matches the map and the PR description. - Dependencies: NumPy is already declared in pyproject.toml (line 32), so the map's "no dependency change" claim holds. - Explicit-request path: PositionalAnchors.order routes to shard_positions, so the vector kernel covers that path. - #282 exists in history (6042710f). #140 and #164 are external and could not be checked locally. Substantive issues (proven) 1. Omitted material behavior: full epoch materialization on delivery paths. The map says the change keeps "bounded temporary storage." That holds only for the shard_positions iterator. The explicit-request path still builds the full order in memory: - task.py lines 443–456 (request_batches): chosen = list(padded_positions(anchors.order(...), ...)), then anchors.fetch(...) over all positions. - assembly.py lines 2414–2423 (batches): order = list(shard_positions(...)). The map names explicit-request delivery as covered, but it does not disclose this. The "bounded" rationale is therefore misleading for end-to-end delivery. 2. Overstated verification. The map says the tests compare "across boundaries and large domains." In tests/test_anchor_permutation.py: - Exact vector-versus-scalar comparisons use sizes up to 10003, so _half is at most 7. - The large-domain test compares only 5 positions. Those fall below 64, so they use the scalar fallback and never reach the vector kernel. - The 2**40 bounded-chunk test checks lengths only, not values. So exact parity for the vector kernel is not shown for _half in 17..31. I found no defect in the arithmetic. This is a verification gap. Optional refinements (kept separate) - Streaming the two consumers above would fix issue 1. That is a separate change, not a requirement of this PR, unless bounded storage is a stated requirement. - The 64-position threshold and the 4096-per-rank chunk size have no stated rationale in the map. - many returns a list on the scalar branch and a uint64 ndarray on the vector branch. shard_positions tolerates both, so this is a design smell, not a behavior defect. I did not run the code. The arithmetic conclusions rest on reading the source.

Resolution

Correct the change map and tests as follows: 1. Scope the bounded-storage claim. In the anchors.py row and the Scope/verification text, state that chunking bounds only shard_positions evaluation. State that explicit-request delivery (task.py request_batches, lines 443–456) and the assembly.py batches path (lines 2414–2423) still materialize the full epoch order with list(...). Remove "bounded temporary storage" from any end-to-end claim. 2. Decide the consumer behavior. Either (a) record streaming those consumers as a separate, optional follow-up outside this PR, or (b) if bounded storage is required for explicit requests, change those two call sites to consume the iterator in chunks in this PR, with a test that checks it. 3. Close the verification gap. Add an exact-parity test that forces the vector kernel (at least 64 positions, no scalar fallback) at a domain where _half is between 17 and 31, for example n = 2**32 + 1 compared with the scalar __call__ on a 4096-position window. Also correct the map's "large domains" wording to match what the tests exercise. Alternatively, state the gap explicitly. 4. Optional. Give the 64-position threshold and 4096-per-rank chunk rationale. Make many return one type on both branches.

Latest assessment: Change Map Design Review

Latest assessment: The latest assessment was inconclusive; the previous failure remains unresolved.

Latest explanation: The check assesses the PR conversation comment that starts with &lt;!-- kumo-pr-change-map --&gt;. That comment is not in the checkout. The only marker hit is the rule in docs/development/review.md. Its text is available only through the prior failure's quotation, so I cannot confirm whether the map was edited after that run. Code evidence at current HEAD: - Verification gap (prior issue 2): addressed in code. Commit de41932d added a test in tests/test_anchor_permutation.py. It compares exact vector-versus-scalar output for half 17 through 31, using 4,096-position start, middle and end windows. It also monkeypatches scalar __call__ to fail, so the vector kernel must run. Whether the map's "large domains" wording was corrected is unverified. - Bounded-storage claim (prior issue 1): still applies in code. anchors.py lines 445–449 chunk only the shard_positions iterator. task.py lines 443–456 (request_batches) still call list(padded_positions(anchors.order(...))). assembly.py lines 2425–2434 (batches) still call order = list(shard_positions(...)). The PR description states this limit. The map must scope "bounded" to shard_positions, and I cannot confirm that it does. - Design: Placing the vector kernel in _Permutation.many and the chunk loop in shard_positions fits the existing abstraction. NumPy is already a declared dependency (pyproject.toml line 32). The scalar fallback for fewer than 64 positions and for _half &gt; 31 is present. many returns a list on the scalar branch and a uint64 ndarray on the vector branch. This is a design smell, not a proven defect. I did not find a proven defect in the arithmetic or the changed code.

Latest resolution: To reach a verdict, provide the text of comment 6864860037 (or the current &lt;!-- kumo-pr-change-map --&gt; comment), or enable PR-comment retrieval for this check. Then verify these items in the map: 1. In the anchors.py row and the Scope/verification text, state that chunking bounds only shard_positions evaluation. State that task.py request_batches (lines 443–456) and assembly.py batches (lines 2425–2434) still materialize the full epoch order with list(...). Remove "bounded temporary storage" from any end-to-end claim. 2. Either record streaming those two consumers as a separate optional follow-up, or change them in this PR with a test. 3. Cite the de41932d large-domain test (half 17–31, 4,096-position windows, scalar fallback disabled) as the verification for the vector kernel. Correct "large domains" wording to match the tests. 4. Optional: give the 64-position threshold and the 4,096-per-rank chunk rationale. Make many return one type on both branches.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@am-porter

am-porter commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Scope and verification

The default positional shuffle still performs scalar Python mixing for every anchor. This PR batches that arithmetic while preserving exact seed/epoch/rank order. Merged #282 supplies native anchor/model batching; this optimization is independent of #164.

The 4,096-position bound applies only to temporary evaluation in shard_positions; _Permutation.many relies on its caller to bound inputs. Explicit-request delivery in task.py::request_batches still builds chosen = list(padded_positions(...)) and fetches the selected requests. assembly.py::batches still builds order = list(shard_positions(...)). Neither consumer has bounded end-to-end epoch storage. Streaming these consumers is a separate optional follow-up outside this PR.

The threshold of 64 avoids array setup for tiny inputs; chunks of 4,096 amortize vector operations while limiting temporary arrays independently of epoch size. These are engineering choices, not measured universal optima. Scalar fallback also preserves arbitrary-size integer domains. The private helper retains list/ndarray returns; its caller accepts either, and return-type normalization is optional.

Review-fix validation: PYTHONPATH=src python -m pytest -q tests/test_anchor_permutation.py tests/test_anchor_tables.py: 378 passed. The 45 added cases compare 4,096-position start/middle/end windows against the scalar oracle at every half-width 17–31, and patch scalar evaluation to fail during the vector call. Changed-file Ruff check/format and git diff --check pass.

Original implementation evidence: package lint/format, license headers and distribution builds passed. In-memory conformance had 521 passes and 19 missing-DiskGraph errors, identical on its unmodified base; full native conformance was not run locally. Those broader checks and the original 400,000-anchor permutation-only timing were not rerun for this test/docstring follow-up.

Change map

File Function / scope What changed Why
packages/experimental/kumo-pytorch/src/kumo/pytorch/anchors.py _Permutation.many, shard_positions Evaluate the same permutation in NumPy chunks of at most 4,096 positions per rank; retain scalar fallback below 64 positions and for very large domains. Clarify that the caller bounds helper inputs. Reduce Python mixing work while preserving order. Temporary storage is bounded only within shuffle evaluation; consumers still materialize full epoch lists.
packages/experimental/kumo-pytorch/tests/test_anchor_permutation.py Scalar parity, large vector domains, bounded evaluation, sharding and task delivery Check exact vector parity across small boundaries and 45 large-domain windows with scalar fallback forbidden; retain huge-domain fallback, chunk-bound and delivery checks. Detect order drift, overflow, accidental scalar fallback, excessive shuffle evaluation and incorrect rank delivery.

@am-porter
am-porter temporarily deployed to gha-kumo-workflow-tests November 16, 2026 23:06 — with GitHub Actions Inactive
@am-porter
am-porter temporarily deployed to gha-kumo-workflow-tests November 16, 2026 23:57 — with GitHub Actions Inactive
@Victsm

Victsm commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Could you make the limited applicability explicit in the summary/performance discussion? This optimizes Python positional/request-based shuffling; it does not change the DG-native batch iterator used by the primary kumo.pytorch graph and sequence loading paths, including native length-aware planning. The 17.5x measurement is permutation arithmetic only, not loader throughput.

Our direction is to move production consumers to native AnchorTable iteration and remove the public task.batches(requests) API. Please identify which retained callers benefit after that migration, and avoid introducing new request-based consumers to demonstrate this optimization. This can still be useful for retained positional/reference paths, but should not be presented as an improvement to native graph/sequence iteration.

This branch was previously deployed

1 inactive deployment
gha-kumo-workflow-tests — de41932d Deployed Nov 16, 2026 by am-porter via import #2460
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants