Found while working on: PR #107 (performance prototypes; segmented dependency-counter factorization bench/fact_fused.jl)
What: the offline placement of contribution blocks on the update stack (build_layout, src/symbolic/layout.jl, PR #54) derives the lifetime of a block from the schedule steps of its front and its parent, i.e. it assumes level-synchronous execution. Any out-of-level-order execution (dependency counters, concurrent streams, persistent or fused kernels, which is experiment 5 of PERFORMANCE.md and the subject of #82) reuses a slot while its block is still live, and a global zero pass does the same; the failures surface as mid-column pivot errors far from the cause. The prototype gives every non-subtree front a private block slot: 0.14 GB on the 78k-bus condensed KKT against 34 MB for the reusing stack. Related: concurrent extend-add of a parent's children needs atomic adds (the owner-pull order of extend_add! serializes children; a fused grid does not), and full cross-stream overlap deadlocks structurally on that tree (62 wide fronts from level 3, 31.5k dependent blocks saturate the SMs ahead of the vendor stream); the segmented form (fused within segments, vendor fronts between segments) is the deadlock-free shape.
Why it is out of scope: #107 is bench-level; the layout is T07/#54 code and the schedule is T25's. Blocks nothing before T25; it is the memory price of experiment 5.
Suggested fix or plan change: when T25 implements level merging, either compute block lifetimes from the DAG the fused kernel actually executes (interval graph colouring over the execution order, not the level order), or budget private slots and report them in memory_estimates. PLAN §2.3 step 6 and §3.4 updated in the docs PR that cites this issue.
Generated by Claude Code
Found while working on: PR #107 (performance prototypes; segmented dependency-counter factorization
bench/fact_fused.jl)What: the offline placement of contribution blocks on the update stack (
build_layout,src/symbolic/layout.jl, PR #54) derives the lifetime of a block from the schedule steps of its front and its parent, i.e. it assumes level-synchronous execution. Any out-of-level-order execution (dependency counters, concurrent streams, persistent or fused kernels, which is experiment 5 ofPERFORMANCE.mdand the subject of #82) reuses a slot while its block is still live, and a global zero pass does the same; the failures surface as mid-column pivot errors far from the cause. The prototype gives every non-subtree front a private block slot: 0.14 GB on the 78k-bus condensed KKT against 34 MB for the reusing stack. Related: concurrent extend-add of a parent's children needs atomic adds (the owner-pull order ofextend_add!serializes children; a fused grid does not), and full cross-stream overlap deadlocks structurally on that tree (62 wide fronts from level 3, 31.5k dependent blocks saturate the SMs ahead of the vendor stream); the segmented form (fused within segments, vendor fronts between segments) is the deadlock-free shape.Why it is out of scope: #107 is bench-level; the layout is T07/#54 code and the schedule is T25's. Blocks nothing before T25; it is the memory price of experiment 5.
Suggested fix or plan change: when T25 implements level merging, either compute block lifetimes from the DAG the fused kernel actually executes (interval graph colouring over the execution order, not the level order), or budget private slots and report them in
memory_estimates. PLAN §2.3 step 6 and §3.4 updated in the docs PR that cites this issue.Generated by Claude Code