Found while working on: experiment 0 of PERFORMANCE.md (PR #81)
What: PR #81 splits regime-A subtrees by flops. After that, the pglib KKT dumps no longer run on one workgroup, but their refactorization and solve are bound by the number of tree levels. The profile comes from bench/profile_phases.jl on an RTX 4080 and is in bench/profile/phase_split.md on that branch.
| matrix |
refactorization |
kernels |
solve |
kernels |
cuDSS refactorization / solve |
| case1354 condensed, Cholesky |
2.9 ms |
59 |
1.1 ms |
62 |
0.59 / 0.27 ms |
| case1354 condensed, LDLᵀ |
9.7 ms |
59 |
1.2 ms |
63 |
0.68 / 0.36 ms |
| case1354 K2, LDLᵀ |
7.2 ms |
58 |
1.5 ms |
85 |
0.82 / 0.42 ms |
On these rows, the longest kernel takes under 15% of GPU busy time. Busy time still equals wall time, so the host is not the limit. The cost is one launch per schedule level and step. Each regime-B launch (factorize_fronts_b! / front_ldlt_kernel, and the per-level sweep kernels in src/solve/sweeps.jl) has few fronts, often one or two thread blocks. The remaining gap to cuDSS on case1354 is 5× (Cholesky) and 9–14× (LDLᵀ) for refactorization, and 3–6× for the solve.
Why it is out of scope: PR #81 only fixes the regime-A selection. The work belongs to T25 (performance pass: "CUDA sync-free forward sweep, CUDA graph capture of refactorize+solve, level merging") and is not blocking. It is experiments 5 and 6 of PERFORMANCE.md. The LDLᵀ rows also contain the one-front-per-workgroup cost of #75.
Suggested fix or plan change: three options.
- Level merging: one persistent kernel over a window of levels, with per-front dependency counters instead of one launch per level.
- Graph capture: capture refactorization and solve in one CUDA graph, replayed through a cached exec rather than
@captured.
- Wider per-level launches: more than one block per small front, or several fronts per block, for the sweeps.
Measure each with bench/profile_phases.jl --only=kkt_pglib_opf_case1354_pegase_k2_1,....
Found while working on: experiment 0 of
PERFORMANCE.md(PR #81)What: PR #81 splits regime-A subtrees by flops. After that, the pglib KKT dumps no longer run on one workgroup, but their refactorization and solve are bound by the number of tree levels. The profile comes from
bench/profile_phases.jlon an RTX 4080 and is inbench/profile/phase_split.mdon that branch.On these rows, the longest kernel takes under 15% of GPU busy time. Busy time still equals wall time, so the host is not the limit. The cost is one launch per schedule level and step. Each regime-B launch (
factorize_fronts_b!/front_ldlt_kernel, and the per-level sweep kernels insrc/solve/sweeps.jl) has few fronts, often one or two thread blocks. The remaining gap to cuDSS on case1354 is 5× (Cholesky) and 9–14× (LDLᵀ) for refactorization, and 3–6× for the solve.Why it is out of scope: PR #81 only fixes the regime-A selection. The work belongs to T25 (performance pass: "CUDA sync-free forward sweep, CUDA graph capture of refactorize+solve, level merging") and is not blocking. It is experiments 5 and 6 of
PERFORMANCE.md. The LDLᵀ rows also contain the one-front-per-workgroup cost of #75.Suggested fix or plan change: three options.
@captured.Measure each with
bench/profile_phases.jl --only=kkt_pglib_opf_case1354_pegase_k2_1,....