Skip to content

Use smaller native work partitions for history minibatches - #389

Merged
Victsm merged 4 commits into
mainfrom
codex/history-task-granularity-20261118
Nov 19, 2026
Merged

Victsm merged 4 commits into
mainfrom
codex/history-task-granularity-20261118

Conversation

@Victsm

@Victsm Victsm commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

A 64-anchor sequence minibatch currently creates only one native history job, even with eight configured threads. Use an eight-anchor partition target for pure-history plans, retaining the existing two-jobs-per-thread cap and single-thread bypass. Aggregate/mixed-plan scheduling and output order are unchanged; no new configuration.

On a retained 123M-event graph, 16,384 anchors and eight native threads, median warm loader latency over three repeats improves:

Batch size Before After Reduction
32 2.094s 1.855s 11.4%
64 1.641s 1.249s 23.9%
128 1.166s 0.971s 16.7%
256 0.888s 0.818s 7.8%

These are CPU loader measurements including tensor adaptation, not training speedups. More parallel work consumes more CPU: the batch-64 traversal uses 1.739→2.918 CPU-seconds. Empty histories are 3–8% slower, with a worst measured overhead of 158µs per batch. Single-thread behavior is unchanged.

The same history executor serves fixed batches, cache-miss length-planning pools and final dynamic mini-batches. Pool size and actual delivered batch size control their separate job counts; fully cached planning epochs skip the length-query retrieval. The reported benchmark does not establish the same speedup for dynamic batches.

Validation: 290 crate unit tests and two focused integration tests pass. The real-index matrix covers normal/one-event/empty/repeated-root histories, batches 32/64/128/256 and 1/2/8 threads. All 48 matching combinations preserve exact tensors, order and event counts. Follow-up integration asserts independent per-anchor event/cap counts, nonempty witnesses and contextual failure messages. Identical compiler/build flags were used for all native variants.

Change map

#389 (comment)

@copy-pr-bot

copy-pr-bot Bot commented Nov 18, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@Victsm
Victsm temporarily deployed to gha-kumo-workflow-tests November 18, 2026 08:21 — with GitHub Actions Inactive
@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

execute_prepared_parallel now uses an 8-anchor threshold and chunk divisor when a plan has sequences and no aggregates. Other plans retain the 64-anchor threshold and divisor. The integration test compares prepared-history results across 1-, 2-, and 8-thread pools for varied limits, batch sizes, roots, and cutoff inputs.

Merge Risk: Low · up to 11fbd

The change lowers the parallel threshold for history-only plans. It looks functionally safe, but the new parity test could miss a dropped sequence or a populated history at exactly 8 anchors. These are small test follow-ups, not blockers.

Architecture risk: Low · up to 11fbd

The change affects 1 system.

Changed systems: packages/diskgraph

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — packages/diskgraph (library) was modified; 2 changed files map to changed impact.

Before / after behavior

  • observed — Modified behavior in packages/diskgraph/diskgraph-train-table/tests/ttgen_integration.rs: Adds a parallel-history integration test that checks per-anchor row counts and compares sequence names, offsets, node IDs, timestamps, and batches across 1-, 2-, and 8-thread pools. Cases vary sequence limits, batch sizes, repeated roots, and empty or partially populated cutoffs.
  • observed — Modified behavior in packages/diskgraph/diskgraph-train-table/src/batch_aggregate.rs: The parallelization threshold and chunk divisor are now 8 when a plan has sequences but no aggregates; all other plans continue to use 64. Root/cutoff length validation and serial fallback behavior are unchanged.
Pre-merge checks: Passed 2 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Change Map Design Review Inconclusive Summary: I could not read the required <!-- kumo-pr-change-map --> comment. The PR description links to it (issuecomment-6973097535), but no tool here fetches PR conversation comments, and the m… Provide the text of the <!-- kumo-pr-change-map --> comment, or enable PR comment retrieval, so the Change map table can be reviewed. Then re-run the check. When the table is reviewed, make these corrections where needed: 1. Describe the …
Passed Passed checks: 2 passed

Check name Status Explanation
Pr Size Passed The PR changes 2 files, which is below the threshold of 40. The changed-file inventory from git diff --name-status refs/coderabbit/pre-merge-check/base refs/coderabbit/pre-merge-check/head lists two…
Public Resource Access Passed The check applies only to external resource references (URLs, repository locations, package source locations) that this PR adds or modifies in the repository diff. The diff touches two files: `package…
Full details: Pr Size

Explanation

The PR changes 2 files, which is below the threshold of 40. The changed-file inventory from git diff --name-status refs/coderabbit/pre-merge-check/base refs/coderabbit/pre-merge-check/head lists two modified files: - packages/diskgraph/diskgraph-train-table/src/batch_aggregate.rs (M) - packages/diskgraph/diskgraph-train-table/tests/ttgen_integration.rs (M) git diff --name-only returns the same two paths. The check ignores line counts, so the size of the edits does not affect the result. The PR does not exceed the file-count limit, so no split recommendation applies.

Full details: Public Resource Access

Explanation

The check applies only to external resource references (URLs, repository locations, package source locations) that this PR adds or modifies in the repository diff. The diff touches two files: packages/diskgraph/diskgraph-train-table/src/batch_aggregate.rs and packages/diskgraph/diskgraph-train-table/tests/ttgen_integration.rs. A grep over every added line for http(s)://, git+, git@, github.com, crates.io, registry, path =, git =, ssh://, file://, and package-index hosts found no matches. No removed lines contained such references either. The change-map URL (github.com/NVIDIA/kumo-engine/pull/389#issuecomment-...) appears only in the PR description, not in the repository diff, so it falls outside the check's scope. The check therefore introduces no added or modified external resource reference, and it passes.

Full details: Change Map Design Review

Explanation

Summary: I could not read the required &lt;!-- kumo-pr-change-map --&gt; comment. The PR description links to it (issuecomment-6973097535), but no tool here fetches PR conversation comments, and the marker is not in the checkout. The pr_objectives section paraphrases the comment, but it omits the table itself. The table is the artifact the check reviews, so I cannot finish the design review. I checked the code and the paraphrased claims against the diff. Verified code facts (PR #389, batch_aggregate.rs execute_prepared_parallel): - The change is 8 anchors per chunk for plans with sequences and no aggregates. Other plans keep 64. Single-thread bypass and length checks are unchanged. - The chunk count is still min(ceil(n / anchors_per_chunk), 2 × threads). So the 8-anchor target holds only while n ≤ 16 × threads. With 8 threads, a 256-anchor batch gives 16 chunks of 16 anchors. The 2 × threads cap overrides the 8-anchor target. The PR description discloses the cap but calls the target "eight-anchor." A table that says "8-anchor partitions" would be misleading for batch 256. - The executor is PreparedTemporalPlan::execute. It is called from runtime_batch_iterator.rs (cache-miss length planning and delivery). The PR's claim that all three callers share it is consistent with the code. The benchmark covers fixed batches only, and the PR says so. - "Fully cached planning epochs skip length-query retrieval" is not in this diff. The diff changes only batch_aggregate.rs (+7/−2) and ttgen_integration.rs (+71). If the table mentions that behavior, it must be marked as pre-existing. - The new integration test asserts per-anchor row counts, which match the paraphrased [4,2,1,3,0] oracle. It also compares 1, 2, and 8 thread results. Design questions (uncertain, not proven defects): - The 8-anchor target adds up to 3–8% overhead on empty histories. It also applies to dynamic minibatches, which the benchmark does not measure. Whether that trade-off is acceptable is a design judgment. - The aggregates.is_empty() test is the proxy for "pure history." Whether that proxy covers every non-history output is not verified. Result: No proven defect was found. The design cannot be fully assessed without the comment's table text.

Resolution

Provide the text of the &lt;!-- kumo-pr-change-map --&gt; comment, or enable PR comment retrieval, so the Change map table can be reviewed. Then re-run the check. When the table is reviewed, make these corrections where needed: 1. Describe the partition size precisely: 8 anchors when n ≤ 16 × threads, otherwise ceil(n / (2 × threads)). Do not describe every partition as 8 anchors. 2. Label the "skip length-query retrieval on fully cached epochs" behavior as pre-existing, not part of this diff. 3. State that the dynamic-minibatch and cache-miss-planning effects are not measured by the reported benchmark.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between e7b5ac3 and 11fbdf5.

Files selected for processing: 1

  • packages/diskgraph/diskgraph-train-table/src/batch_aggregate.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@Victsm

Victsm commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator Author

Change map

Observed issue: a 64-anchor sequence minibatch creates only one native history job even when the loader has several worker threads. The requested outcome is to distribute independent history retrieval over the existing pool without changing batch contents or adding another iterator.

File Function / scope What changed Why
packages/diskgraph/diskgraph-train-table/src/batch_aggregate.rs execute_prepared_parallel Use an eight-anchor work-partition target for pure-history plans, retaining the existing two-jobs-per-thread ceiling, serial single-thread execution, and aggregate/mixed-plan scheduling. Let common sequence minibatches use the already configured native thread pool.
packages/diskgraph/diskgraph-train-table/tests/ttgen_integration.rs integ_parallel_history_preserves_ragged_batches Compare complete sequence outputs across one, two and eight threads, empty/partly empty histories, repeated roots, ascending/descending order, capped histories and boundary batch sizes. Independently assert exact per-anchor counts/caps and nonempty results with contextual failure messages. Prove that partitioning preserves ragged offsets, row order, node IDs, timestamps and Arrow values.

Stopping condition: focused parity, existing prepared-history regression and crate unit tests pass; bounded real-index qualification measures the scheduling tradeoff. Broader encoder, aggregation, graph-topology and Python-adapter changes are separate.

Follow-up verification: cargo test -p diskgraph-train-table --test ttgen_integration integ_parallel_history_preserves_ragged_batches -- --exact passes. The reviewed oracle checks uncapped [4,2,1,3,0] source-event counts for the five fixture users and min(count,1) for capped output. The scheduling implementation and measured native binary are unchanged by this test-only update.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpick comments: 1

packages/diskgraph/diskgraph-train-table/tests/ttgen_integration.rs: 1

10683-10722: Trivial · Maintainability & Code Quality · Quick win

Make the new test check the plan's behavior, not only thread-count equivalence.

The test compares the 2-thread and 8-thread results with the 1-thread result. The 1-thread pool always takes the serial path, so it is a valid baseline. The test does not check that any chunked execution happens. It also does not check that the baseline has rows. With stop_seconds at 0 and cutoffs of 0 or 5000, many cases may produce empty or trivially equal results.

Add an assertion that some baseline sequences[0].node_ids is non-empty. This confirms the comparison covers real rows. The limit variants also do not check the expected counts per anchor.

Learnings favor table-driven tests for many similar cases. The nested loops over limit, size, repeated and empty do match that guidance. Failure messages do not identify the failing case, so add context to the assertions, for example assert_eq!(..., "limit={limit:?} size={size} repeated={repeated} empty={empty}").

Source: Learnings

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between b53781b and 2624ffc.

Files selected for processing: 2

  • packages/diskgraph/diskgraph-train-table/src/batch_aggregate.rs
  • packages/diskgraph/diskgraph-train-table/tests/ttgen_integration.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@Victsm
Victsm temporarily deployed to gha-kumo-workflow-tests November 18, 2026 15:46 — with GitHub Actions Inactive

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

Other comments: 2

packages/diskgraph/diskgraph-train-table/tests/ttgen_integration.rs: 2

10711-10711: Minor · Functional Correctness · Quick win

Check nonempty history at the eight-anchor threshold.

At size 8, every cutoff is zero, even when empty is false. The test therefore passes if the executor drops populated histories only at the new threshold. Keep the empty size-8 case and add one with an eligible event. Check its expected row count before comparing the pool results.


10709-10709: Minor · Functional Correctness · Quick win

Assert that the result contains both requested sequences.

If all three pools omit recent, the row-count loop checks only history and the parity assertions still pass. If all three pools return no sequences, both loops skip their checks. Assert that the serial result contains history and recent before checking row counts. The existing parity assertions will then detect omissions in either parallel result.

Nitpick comments: 1

packages/diskgraph/diskgraph-train-table/tests/ttgen_integration.rs: 1

10711-10711: Trivial · Functional Correctness · Quick win

Add an upper-cutoff case to the parallel history parity test.

The matrix at ttgen_integration.rs:10705 uses only the cutoffs 0 and 5000. The latest fixture order has timestamp 3300, so no event falls on either cutoff. If upper-bound inclusion regressed on the prepared-history path, this test would still pass.

Add a sequence-only case with at least eight anchors, a multithreaded pool, and cutoff 1300 for the user whose fixture orders include timestamp 1300. Assert that the event at 1300 is counted, which matches the inclusive upper bound used at ttgen_integration.rs:3223-3224. Compare the result with the single-thread output, as the existing parity test does.

Source: Coding guidelines

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 2624ffc and 882dbdd.

Files selected for processing: 1

  • packages/diskgraph/diskgraph-train-table/tests/ttgen_integration.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@Victsm
Victsm temporarily deployed to gha-kumo-workflow-tests November 18, 2026 21:29 — with GitHub Actions Inactive
@Victsm
Victsm deployed to gha-kumo-workflow-tests November 19, 2026 01:07 — with GitHub Actions Active
@Victsm
Victsm enabled auto-merge (squash) November 19, 2026 01:07
@Victsm
Victsm disabled auto-merge November 19, 2026 03:06
@Victsm
Victsm merged commit 8d33799 into main Nov 19, 2026
111 checks passed
@Victsm
Victsm deleted the codex/history-task-granularity-20261118 branch November 19, 2026 03:06

This branch was successfully deployed

1 active deployment
gha-kumo-workflow-tests — 11fbdf53 Deployed Nov 19, 2026 by Victsm via import #2658
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants