You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Use smaller native work partitions for history minibatches - #389
A 64-anchor sequence minibatch currently creates only one native history job, even with eight configured threads. Use an eight-anchor partition target for pure-history plans, retaining the existing two-jobs-per-thread cap and single-thread bypass. Aggregate/mixed-plan scheduling and output order are unchanged; no new configuration.
On a retained 123M-event graph, 16,384 anchors and eight native threads, median warm loader latency over three repeats improves:
Batch size
Before
After
Reduction
32
2.094s
1.855s
11.4%
64
1.641s
1.249s
23.9%
128
1.166s
0.971s
16.7%
256
0.888s
0.818s
7.8%
These are CPU loader measurements including tensor adaptation, not training speedups. More parallel work consumes more CPU: the batch-64 traversal uses 1.739→2.918 CPU-seconds. Empty histories are 3–8% slower, with a worst measured overhead of 158µs per batch. Single-thread behavior is unchanged.
The same history executor serves fixed batches, cache-miss length-planning pools and final dynamic mini-batches. Pool size and actual delivered batch size control their separate job counts; fully cached planning epochs skip the length-query retrieval. The reported benchmark does not establish the same speedup for dynamic batches.
Validation: 290 crate unit tests and two focused integration tests pass. The real-index matrix covers normal/one-event/empty/repeated-root histories, batches 32/64/128/256 and 1/2/8 threads. All 48 matching combinations preserve exact tensors, order and event counts. Follow-up integration asserts independent per-anchor event/cap counts, nonempty witnesses and contextual failure messages. Identical compiler/build flags were used for all native variants.
No actionable comments were generated in the recent review.
Walkthrough
execute_prepared_parallel now uses an 8-anchor threshold and chunk divisor when a plan has sequences and no aggregates. Other plans retain the 64-anchor threshold and divisor. The integration test compares prepared-history results across 1-, 2-, and 8-thread pools for varied limits, batch sizes, roots, and cutoff inputs.
Merge Risk: Low · up to 11fbd
The change lowers the parallel threshold for history-only plans. It looks functionally safe, but the new parity test could miss a dropped sequence or a populated history at exactly 8 anchors. These are small test follow-ups, not blockers.
Architecture risk: Low · up to 11fbd
The change affects 1 system.
Changed systems:packages/diskgraph
Architecture concerns
No architecture-level concerns identified.
Review details
Systems and components
observed — packages/diskgraph (library) was modified; 2 changed files map to changed impact.
Before / after behavior
observed — Modified behavior in packages/diskgraph/diskgraph-train-table/tests/ttgen_integration.rs: Adds a parallel-history integration test that checks per-anchor row counts and compares sequence names, offsets, node IDs, timestamps, and batches across 1-, 2-, and 8-thread pools. Cases vary sequence limits, batch sizes, repeated roots, and empty or partially populated cutoffs.
observed — Modified behavior in packages/diskgraph/diskgraph-train-table/src/batch_aggregate.rs: The parallelization threshold and chunk divisor are now 8 when a plan has sequences but no aggregates; all other plans continue to use 64. Root/cutoff length validation and serial fallback behavior are unchanged.
Pre-merge checks: 2 · 1
Failed checks: 1 inconclusive
Check name
Status
Explanation
Resolution
Change Map Design Review
Summary: I could not read the required <!-- kumo-pr-change-map --> comment. The PR description links to it (issuecomment-6973097535), but no tool here fetches PR conversation comments, and the m…
Provide the text of the <!-- kumo-pr-change-map --> comment, or enable PR comment retrieval, so the Change map table can be reviewed. Then re-run the check. When the table is reviewed, make these corrections where needed: 1. Describe the …
Passed checks: 2 passed
Check name
Status
Explanation
Pr Size
The PR changes 2 files, which is below the threshold of 40. The changed-file inventory from git diff --name-status refs/coderabbit/pre-merge-check/base refs/coderabbit/pre-merge-check/head lists two…
Public Resource Access
The check applies only to external resource references (URLs, repository locations, package source locations) that this PR adds or modifies in the repository diff. The diff touches two files: `package…
Full details: Pr Size
Explanation
The PR changes 2 files, which is below the threshold of 40. The changed-file inventory from git diff --name-status refs/coderabbit/pre-merge-check/base refs/coderabbit/pre-merge-check/head lists two modified files: - packages/diskgraph/diskgraph-train-table/src/batch_aggregate.rs (M) - packages/diskgraph/diskgraph-train-table/tests/ttgen_integration.rs (M) git diff --name-only returns the same two paths. The check ignores line counts, so the size of the edits does not affect the result. The PR does not exceed the file-count limit, so no split recommendation applies.
Full details: Public Resource Access
Explanation
The check applies only to external resource references (URLs, repository locations, package source locations) that this PR adds or modifies in the repository diff. The diff touches two files: packages/diskgraph/diskgraph-train-table/src/batch_aggregate.rs and packages/diskgraph/diskgraph-train-table/tests/ttgen_integration.rs. A grep over every added line for http(s)://, git+, git@, github.com, crates.io, registry, path =, git =, ssh://, file://, and package-index hosts found no matches. No removed lines contained such references either. The change-map URL (github.com/NVIDIA/kumo-engine/pull/389#issuecomment-...) appears only in the PR description, not in the repository diff, so it falls outside the check's scope. The check therefore introduces no added or modified external resource reference, and it passes.
Full details: Change Map Design Review
Explanation
Summary: I could not read the required <!-- kumo-pr-change-map --> comment. The PR description links to it (issuecomment-6973097535), but no tool here fetches PR conversation comments, and the marker is not in the checkout. The pr_objectives section paraphrases the comment, but it omits the table itself. The table is the artifact the check reviews, so I cannot finish the design review. I checked the code and the paraphrased claims against the diff. Verified code facts (PR #389, batch_aggregate.rsexecute_prepared_parallel): - The change is 8 anchors per chunk for plans with sequences and no aggregates. Other plans keep 64. Single-thread bypass and length checks are unchanged. - The chunk count is still min(ceil(n / anchors_per_chunk), 2 × threads). So the 8-anchor target holds only while n ≤ 16 × threads. With 8 threads, a 256-anchor batch gives 16 chunks of 16 anchors. The 2 × threads cap overrides the 8-anchor target. The PR description discloses the cap but calls the target "eight-anchor." A table that says "8-anchor partitions" would be misleading for batch 256. - The executor is PreparedTemporalPlan::execute. It is called from runtime_batch_iterator.rs (cache-miss length planning and delivery). The PR's claim that all three callers share it is consistent with the code. The benchmark covers fixed batches only, and the PR says so. - "Fully cached planning epochs skip length-query retrieval" is not in this diff. The diff changes only batch_aggregate.rs (+7/−2) and ttgen_integration.rs (+71). If the table mentions that behavior, it must be marked as pre-existing. - The new integration test asserts per-anchor row counts, which match the paraphrased [4,2,1,3,0] oracle. It also compares 1, 2, and 8 thread results. Design questions (uncertain, not proven defects): - The 8-anchor target adds up to 3–8% overhead on empty histories. It also applies to dynamic minibatches, which the benchmark does not measure. Whether that trade-off is acceptable is a design judgment. - The aggregates.is_empty() test is the proxy for "pure history." Whether that proxy covers every non-history output is not verified. Result: No proven defect was found. The design cannot be fully assessed without the comment's table text.
Resolution
Provide the text of the <!-- kumo-pr-change-map --> comment, or enable PR comment retrieval, so the Change map table can be reviewed. Then re-run the check. When the table is reviewed, make these corrections where needed: 1. Describe the partition size precisely: 8 anchors when n ≤ 16 × threads, otherwise ceil(n / (2 × threads)). Do not describe every partition as 8 anchors. 2. Label the "skip length-query retrieval on fully cached epochs" behavior as pre-existing, not part of this diff. 3. State that the dynamic-minibatch and cache-miss-planning effects are not measured by the reported benchmark.
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
Observed issue: a 64-anchor sequence minibatch creates only one native history job even when the loader has several worker threads. The requested outcome is to distribute independent history retrieval over the existing pool without changing batch contents or adding another iterator.
Use an eight-anchor work-partition target for pure-history plans, retaining the existing two-jobs-per-thread ceiling, serial single-thread execution, and aggregate/mixed-plan scheduling.
Let common sequence minibatches use the already configured native thread pool.
Compare complete sequence outputs across one, two and eight threads, empty/partly empty histories, repeated roots, ascending/descending order, capped histories and boundary batch sizes. Independently assert exact per-anchor counts/caps and nonempty results with contextual failure messages.
Prove that partitioning preserves ragged offsets, row order, node IDs, timestamps and Arrow values.
Stopping condition: focused parity, existing prepared-history regression and crate unit tests pass; bounded real-index qualification measures the scheduling tradeoff. Broader encoder, aggregation, graph-topology and Python-adapter changes are separate.
Follow-up verification: cargo test -p diskgraph-train-table --test ttgen_integration integ_parallel_history_preserves_ragged_batches -- --exact passes. The reviewed oracle checks uncapped [4,2,1,3,0] source-event counts for the five fixture users and min(count,1) for capped output. The scheduling implementation and measured native binary are unchanged by this test-only update.
Make the new test check the plan's behavior, not only thread-count equivalence.
The test compares the 2-thread and 8-thread results with the 1-thread result. The 1-thread pool always takes the serial path, so it is a valid baseline. The test does not check that any chunked execution happens. It also does not check that the baseline has rows. With stop_seconds at 0 and cutoffs of 0 or 5000, many cases may produce empty or trivially equal results.
Add an assertion that some baseline sequences[0].node_ids is non-empty. This confirms the comparison covers real rows. The limit variants also do not check the expected counts per anchor.
Learnings favor table-driven tests for many similar cases. The nested loops over limit, size, repeated and empty do match that guidance. Failure messages do not identify the failing case, so add context to the assertions, for example assert_eq!(..., "limit={limit:?} size={size} repeated={repeated} empty={empty}").
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
10711-10711: Minor · Functional Correctness · Quick win
Check nonempty history at the eight-anchor threshold.
At size 8, every cutoff is zero, even when empty is false. The test therefore passes if the executor drops populated histories only at the new threshold. Keep the empty size-8 case and add one with an eligible event. Check its expected row count before comparing the pool results.
10709-10709: Minor · Functional Correctness · Quick win
Assert that the result contains both requested sequences.
If all three pools omit recent, the row-count loop checks only history and the parity assertions still pass. If all three pools return no sequences, both loops skip their checks. Assert that the serial result contains history and recent before checking row counts. The existing parity assertions will then detect omissions in either parallel result.
Add an upper-cutoff case to the parallel history parity test.
The matrix at ttgen_integration.rs:10705 uses only the cutoffs 0 and 5000. The latest fixture order has timestamp 3300, so no event falls on either cutoff. If upper-bound inclusion regressed on the prepared-history path, this test would still pass.
Add a sequence-only case with at least eight anchors, a multithreaded pool, and cutoff 1300 for the user whose fixture orders include timestamp 1300. Assert that the event at 1300 is counted, which matches the inclusive upper bound used at ttgen_integration.rs:3223-3224. Compare the result with the single-thread output, as the existing parity test does.
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A 64-anchor sequence minibatch currently creates only one native history job, even with eight configured threads. Use an eight-anchor partition target for pure-history plans, retaining the existing two-jobs-per-thread cap and single-thread bypass. Aggregate/mixed-plan scheduling and output order are unchanged; no new configuration.
On a retained 123M-event graph, 16,384 anchors and eight native threads, median warm loader latency over three repeats improves:
These are CPU loader measurements including tensor adaptation, not training speedups. More parallel work consumes more CPU: the batch-64 traversal uses 1.739→2.918 CPU-seconds. Empty histories are 3–8% slower, with a worst measured overhead of 158µs per batch. Single-thread behavior is unchanged.
The same history executor serves fixed batches, cache-miss length-planning pools and final dynamic mini-batches. Pool size and actual delivered batch size control their separate job counts; fully cached planning epochs skip the length-query retrieval. The reported benchmark does not establish the same speedup for dynamic batches.
Validation: 290 crate unit tests and two focused integration tests pass. The real-index matrix covers normal/one-event/empty/repeated-root histories, batches 32/64/128/256 and 1/2/8 threads. All 48 matching combinations preserve exact tensors, order and event counts. Follow-up integration asserts independent per-anchor event/cap counts, nonempty witnesses and contextual failure messages. Identical compiler/build flags were used for all native variants.
Change map
#389 (comment)