Compile time: fewer kernel specialisations per (T, INT)

[michel2323]

Context: PR #90 (CI time). After #87–#89 the test suite is dominated by compilation, not by tests. CI went from ~21 min to ~52 min. With the parallel runner and the CPU job on kkt it is ~34 min, and the floor is the cold compile of one worker.

Measurements (owner's machine, CPU backend unless noted)

• First solver factorization through the API, per (T, INT): 42–84 s after perf: cooperative pivot search in the device LDLᵀ kernels (#75) #87–perf: regime-C LDLᵀ through blocked pivot steps and GEMMs (#75) #89 (27–47 s before). With the Cholesky/LDLᵀ barrier from PR CI: parallel test runner, CPU job on kkt, LDLᵀ compiled only when used #90: 20–36 s. An LDLᵀ call afterwards costs ~28 s more.
• Full CPU suite in one process: ~18.5 min, of which ~12 min is the cold compile (the first file pays it, the rest runs warm).
• On kkt (CI) the per-worker cold compile is 25–31 min (slow per-core compile).
• CUDA: test_numeric_ldlt takes 1155–1787 s on CI (351 s before perf: cooperative pivot search in the device LDLᵀ kernels (#75) #87–perf: regime-C LDLᵀ through blocked pivot steps and GEMMs (#75) #89). This is GPU kernel compilation of the new LDLᵀ variants.
• Splitting a test file by element type only partly helps, because part of the compile doesn't depend on T. test_numeric_cholesky_c: one element type alone takes 157 s, all four 168 s. test_numeric_ldlt: 814 s → 226 s per element type.

Where the specialisations come from

Kernels are specialised on T (4) × INT (2) and, in addition, on several compile-time classes chosen with explicit Val branches:

• _with_width_class (src/numeric/front.jl): regime-B width classes W ∈ (8, 16, 32, 64), used by the Cholesky (front.jl) and LDLᵀ (ldlt.jl) fused kernels. The branches are static, so the launcher compiles all four variants with its caller, even if the matrix only has one width class.
• _with_local_bytes (src/numeric/subtree.jl): 4 regime-A budget classes (SUBTREE_LOCAL_SIZES), Cholesky and LDLᵀ.
• Further Vals in the LDLᵀ kernels: HERM, the packed/unpacked panel (_lt_pk(Val(W))), Val(ASM), Val(NL), Val(W*(W+1)÷2), and the three pivot passes Val(1..3).
• Solve sweeps (src/solve/sweeps.jl): 3 solve kinds × sub/det/forward variants, also through explicit branches.
• Dense implementations: :generic, :vendor, :ka, plus strided-batched variants.

On the CPU backend a KA kernel is an ordinary Julia function, and every statically reachable variant compiles with its caller. On GPUs a kernel compiles at its first launch, so only the variants actually launched are compiled. But each of them is a separate GPUCompiler/LLVM run per T, INT.

Possible levers (not sure any is acceptable)

1. Keep @localmem sizes compile-time (required by KA), but don't compile unused classes on the CPU backend. For example, put a dynamic barrier at the class branch, like the structure barrier in PR CI: parallel test runner, CPU job on kkt, LDLᵀ compiled only when used #90. It must not allocate in the numeric phase: the argument tuples hold immutable Numeric/Symbolic, which get boxed. A mutable carrier or a precomputed launcher table could avoid that.
2. Fewer classes. On the CPU backend, a single width class (64) and a single budget class would do. Cost: the CPU tests would no longer exercise the smaller-class code paths that the GPU uses.
3. Merge the Val flags that only select small code paths (HERM, pivot pass, packed panel) into run-time branches where the GPU cost is negligible. That needs benchmarking per flag (bench/compare.jl).
4. Test side: not every testset needs all four element types and both index types. That would change the test conventions in AGENTS.md, so it's the owner's decision.

Open question: whether 1–3 can be done without losing GPU performance. The width-class and local-memory specialisations are the core of the regime-A/B design (PLAN §2.4), so this may turn out not to be worth it.

[michel2323]

PR #103 adds SDS_TEST_ELTYPES (in test/backends.jl, default all four types). Pull-request and push CI run Float64,ComplexF32; the weekly schedule and a manual run (default) run all four. Tests and CI configuration only.

Measured before (run 37693884475 of #100) and after (#103 on cc98eaf):

job	before	after	summed part time
CUDA, test step	18m20s	19m03s	−41%
CPU (kkt, 32 workers), test step	21m40s	20m59s	−36%
CPU local, 4 workers	21m14s	11m53s	−44%

Result: the CI jobs do not get shorter. Total work drops by about 40%, but with 12 or 32 workers the slowest single part sets the wall time, and that part is a complex part kept in the selection. On CUDA it is test_numeric_ldlt[ComplexF32] (1132 s; before, [ComplexF64] 1089 s). On kkt every complex part takes about 1100 s. The selection pays off only where the run is throughput-bound (local, small runners) and in runner occupancy.

To shorten the CI wall time, the heavy files have to be split below the element type, e.g. test_numeric_ldlt, test_solve and test_numeric_cholesky_c/_a per testset group. Lever 3 (folding the small Val flags into run-time branches) would also help, since it cuts the kernel compilations inside each part. That is T25 work, so this issue stays open for it. Full per-file table in #103.

---

Generated by Claude Code