From b190d09c5944f9c3ba0ccf074942ab5d4150536e Mon Sep 17 00:00:00 2001 From: Sungho Shin Date: Tue, 13 Oct 2026 11:40:47 -0400 Subject: [PATCH 1/5] KernelAbstractions 0.10 compat: admit it, keep the 0.9 contract exact KA 0.10.0 runs this package unchanged and FASTER on the CPU backend (78k condensed KKT, same tree, ABBA: refactorization 1554 -> 1270-1323 ms, solve 138-143 -> 106-115 ms; the full CPU suite wall drops 14m39s -> 5m53s). Three 0.10 behaviors needed accommodation, none numerical: * its CPU backend heap-allocates ~230 B per kernel launch where 0.9 was allocation-free: ka_cpu_alloc_budget carries a 320 B/launch allowance on 0.10 only (the budgets still scale with each phase's launch count; the 0.9 zero-allocation contract stays exact). Surfacing that also exposed a real undercount: the LDL factorization makes ~3 launches per group (measured), not 1 -- corrected. * its CPU backend announces caught in-kernel exceptions on stderr (noise in the negative tests; harmless). * it keeps worker tasks alive at exit, so Aqua's persistent-tasks check reports the backend, not the package: skipped on >= 0.10 with the upstream reason documented, unchanged on 0.9. The GPU stacks are unaffected today: every released CUDA.jl/AMDGPU.jl pins KA 0.9, so GPU environments keep resolving 0.9 until upstream ships 0.10 support -- at which point the fused-kernel A/B is worth re-running. Validated: full CPU suite green on BOTH versions (0.9.45: 72006 incl. the persistent-tasks check; 0.10.0: 72005/72005, all four element types). Co-Authored-By: Claude Fable 5 --- AGENTS.md | 7 ++++--- Project.toml | 2 +- test/Project.toml | 2 +- test/test_aqua.jl | 5 ++++- test/utils.jl | 11 ++++++++++- 5 files changed, 20 insertions(+), 7 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 2ff3be8b..8ba686ad 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -42,9 +42,10 @@ Three documents define the work. Read them in this order before touching code: §1.2–§1.4 and §3.1. They mirror `../CUDSS.jl` so MadNLP can migrate mechanically. * New dependencies need a one-line justification in the Report. Never `dev` - unregistered or local checkouts into the project (there is a - `../KernelAbstractions.jl` 0.10-dev checkout on this machine; the package - uses the registered KernelAbstractions 0.9.x). + unregistered or local checkouts into the project (the package admits the + registered KernelAbstractions 0.9.x and 0.10.x; the GPU stacks still resolve + 0.9 until CUDA.jl/AMDGPU.jl ship 0.10 support, and the test allocation + budgets carry a per-launch allowance on 0.10 -- see `ka_cpu_alloc_budget`). * The repository is `github.com/exanauts/SparseDirectSolver.jl` (remote `origin`, branch `main`). Commit at the end of the task with a message starting with the task id, e.g. `T05: symbolic pattern, ordering, etree, diff --git a/Project.toml b/Project.toml index f34afbe7..fdbc0290 100644 --- a/Project.toml +++ b/Project.toml @@ -39,7 +39,7 @@ CUDACore = "6" CliqueTrees = "1" GPUArrays = "11, 12" GPUArraysCore = "0.2" -KernelAbstractions = "0.9" +KernelAbstractions = "0.9, 0.10" Krylov = "0.10" LinearAlgebra = "1.13" MadNLP = "0.8, 0.9, 0.10" diff --git a/test/Project.toml b/test/Project.toml index d3ff28ce..cd6f158b 100644 --- a/test/Project.toml +++ b/test/Project.toml @@ -16,7 +16,7 @@ Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40" Adapt = "4" Aqua = "0.8" DelimitedFiles = "1" -KernelAbstractions = "0.9" +KernelAbstractions = "0.9, 0.10" Krylov = "0.10" LinearAlgebra = "1.13" ParallelTestRunner = "2" diff --git a/test/test_aqua.jl b/test/test_aqua.jl index 57db5401..2d792c5c 100644 --- a/test/test_aqua.jl +++ b/test/test_aqua.jl @@ -9,5 +9,8 @@ # undefined_exports is excluded HERE only: SDSSolver/SDSSolverOptions are declared # globals the MadNLP extension assigns, and MadNLP is not in the test environment. # CI's Aqua workflow loads MadNLP and runs that check with the exports assigned. +# persistent_tasks is skipped under KernelAbstractions >= 0.10: its CPU backend keeps +# worker tasks alive at exit, so the check reports the backend, not this package +# (upstream; re-enable when KA's task pool shuts down cleanly) Aqua.test_all(SparseDirectSolver; ambiguities = false, undefined_exports = false, - persistent_tasks = (tmax = 120,)) + persistent_tasks = (pkgversion(KernelAbstractions) >= v"0.10" ? false : (tmax = 120,))) diff --git a/test/utils.jl b/test/utils.jl index 9860bcb6..1f827483 100644 --- a/test/utils.jl +++ b/test/utils.jl @@ -247,6 +247,12 @@ barrier; that allowance went with the Julia 1.10 support.) """ function ka_cpu_alloc_budget(launches::Integer, localmem_bytes::Integer) b = 1024 + # KernelAbstractions 0.10's CPU backend heap-allocates per launch (~230 B measured + # with a trivial precompiled kernel; the new backend's task machinery), where 0.9 + # was allocation-free. 320 B of per-launch headroom keeps the budgets meaningful -- + # they still scale with the phase's launch count -- until the upstream allocation + # is fixed; the 0.9 contract stays exact. + pkgversion(KernelAbstractions) >= v"0.10" && (b += 320 * launches) Base.JLOptions().code_coverage != 0 && (b += localmem_bytes + 64 * launches) return b end @@ -483,7 +489,10 @@ reductions; `@localmem` of the largest regime-A kernel (48 KiB) per group plus the two reductions (T15). """ function ldlt_alloc_budget(S) - launches = length(S.schedule.groups) + 2 + # ~3 launches per group measured on the LDL path (subtree kernel + per-group pivot + # and statistics passes), not 1: the undercount was invisible while KA 0.9 launches + # were allocation-free, and surfaced through the 0.10 per-launch allowance + launches = 3 * length(S.schedule.groups) + 2 localmem = launches * maximum(SparseDirectSolver.SUBTREE_LOCAL_SIZES) + SparseDirectSolver.STATS_WORKGROUP * SparseDirectSolver.FRONT_STATS_FIELDS * sizeof(Int64) return ka_cpu_alloc_budget(launches, localmem) From dc89cd616243c7045b7e30cd0bbe8624781ea435 Mon Sep 17 00:00:00 2001 From: "claude[bot]" <41898282+claude[bot]@users.noreply.github.com> Date: Tue, 13 Oct 2026 16:24:20 +0000 Subject: [PATCH 2/5] KA 0.10 compat: address review round 1 * ldlt_alloc_budget(S, N): count the launches from N.plan (blocked-path groups per chunk and block of nb pivot columns) instead of a constant 3 per group; the coverage-mode localmem term is back to the pre-PR fused-launch count. lu = true for factorize_lu!. * ka_cpu_alloc_budget docstring lists the KA >= 0.10 per-launch term. * test_aqua.jl: the persistent-tasks skip reason names both failure modes (tasks alive at exit; pocl_standalone_jll precompile failure seen on CI). The matching gate in .github/workflows/Aqua.yml cannot be pushed by the GitHub App (no workflows permission); the patch is on the PR for the owner. Co-Authored-By: Claude Opus 5.5 --- test/test_aqua.jl | 8 +++--- test/test_numeric_ldlt.jl | 2 +- test/test_numeric_lu.jl | 2 +- test/utils.jl | 53 ++++++++++++++++++++++++++++----------- 4 files changed, 46 insertions(+), 19 deletions(-) diff --git a/test/test_aqua.jl b/test/test_aqua.jl index 2d792c5c..0da32e29 100644 --- a/test/test_aqua.jl +++ b/test/test_aqua.jl @@ -9,8 +9,10 @@ # undefined_exports is excluded HERE only: SDSSolver/SDSSolverOptions are declared # globals the MadNLP extension assigns, and MadNLP is not in the test environment. # CI's Aqua workflow loads MadNLP and runs that check with the exports assigned. -# persistent_tasks is skipped under KernelAbstractions >= 0.10: its CPU backend keeps -# worker tasks alive at exit, so the check reports the backend, not this package -# (upstream; re-enable when KA's task pool shuts down cleanly) +# persistent_tasks is skipped under KernelAbstractions >= 0.10, where it reports the +# backend, not this package: KA 0.10's CPU backend keeps worker tasks alive at exit, +# and in Aqua's fresh temporary project its pocl_standalone_jll dependency fails to +# precompile ("Cannot load SLEEF_jll into module pocl_standalone_jll", seen on CI). +# Upstream; re-enable when both are fixed. CI's Aqua workflow applies the same gate. Aqua.test_all(SparseDirectSolver; ambiguities = false, undefined_exports = false, persistent_tasks = (pkgversion(KernelAbstractions) >= v"0.10" ? false : (tmax = 120,))) diff --git a/test/test_numeric_ldlt.jl b/test/test_numeric_ldlt.jl index 35397459..90a3783b 100644 --- a/test/test_numeric_ldlt.jl +++ b/test/test_numeric_ldlt.jl @@ -304,7 +304,7 @@ end # no allocation in the numeric phase and the solve (CPU backend) if backend isa CPU SDS.factorize!(Nd, Sd, nz; opts) - @test (@allocated SDS.factorize!(Nd, Sd, nz; opts)) <= ldlt_alloc_budget(S) + @test (@allocated SDS.factorize!(Nd, Sd, nz; opts)) <= ldlt_alloc_budget(S, Nd) x = similar(b) SDS.sweep_solve!(x, ws, Sd, Nd, b) @test (@allocated SDS.sweep_solve!(x, ws, Sd, Nd, b)) <= solve_alloc_budget(S, ws; ldlt = true) diff --git a/test/test_numeric_lu.jl b/test/test_numeric_lu.jl index 3001d7ee..a1a71a4c 100644 --- a/test/test_numeric_lu.jl +++ b/test/test_numeric_lu.jl @@ -201,7 +201,7 @@ end # no allocation in the numeric phase and the solve (CPU backend) if backend isa CPU SDS.factorize!(Nd, Sd, nz; opts) - @test (@allocated SDS.factorize!(Nd, Sd, nz; opts)) <= ldlt_alloc_budget(S) + @test (@allocated SDS.factorize!(Nd, Sd, nz; opts)) <= ldlt_alloc_budget(S, Nd; lu = true) x = similar(b) for transpose in (false, true) SDS.sweep_solve!(x, ws, Sd, Nd, b; transpose) diff --git a/test/utils.jl b/test/utils.jl index 1f827483..331ba1bb 100644 --- a/test/utils.jl +++ b/test/utils.jl @@ -241,7 +241,9 @@ kernel launches whose `@localmem` buffers total `localmem_bytes`, on top of the 1024 B allowed to an allocation-free phase (PLAN §3.9). Measured (T09 review round 1): 0 without coverage (the local runs, Julia ≥ 1.13 is required); with `--code-coverage` (CI's `julia-runtest`) the `@localmem` `MArray`s are -heap-allocated, hence `localmem_bytes` plus 64 B of header per launch. (Julia +heap-allocated, hence `localmem_bytes` plus 64 B of header per launch. Under +KernelAbstractions ≥ 0.10 the CPU backend heap-allocates on every launch +(~230 B measured), hence 320 B per launch on that version only. (Julia ≤ 1.11 also boxed the arguments of every launch behind KA's `__run` inference barrier; that allowance went with the Julia 1.10 support.) """ @@ -481,19 +483,42 @@ only on generators without element growth). d_error(Nh, Nr) = maximum(abs, Nh.d - Nr.d) / maximum(abs, Nr.d) """ - ldlt_alloc_budget(S) -> Int - -[`ka_cpu_alloc_budget`](@ref) for one LDLᵀ/LDLᴴ `factorize!` on the CPU -backend: one launch per launch group, the `max |aᵢⱼ|` and statistics -reductions; `@localmem` of the largest regime-A kernel (48 KiB) per group plus -the two reductions (T15). -""" -function ldlt_alloc_budget(S) - # ~3 launches per group measured on the LDL path (subtree kernel + per-group pivot - # and statistics passes), not 1: the undercount was invisible while KA 0.9 launches - # were allocation-free, and surfaced through the 0.10 per-launch allowance - launches = 3 * length(S.schedule.groups) + 2 - localmem = launches * maximum(SparseDirectSolver.SUBTREE_LOCAL_SIZES) + + ldlt_alloc_budget(S, N; nb = SparseDirectSolver.LDLT_C_NB, lu = false) -> Int + +[`ka_cpu_alloc_budget`](@ref) for one LDLᵀ/LDLᴴ `factorize!` of the numeric +storage `N` (its launch plan `N.plan`) on the CPU backend, counting the +launches the driver makes: one per subtree launch and per fused regime-A/B +launch group; for a group on the blocked path (`ldlt_blocked_path`) the three +assembly launches and, per chunk (`ldlt_c_chunk_end`) and batch member, one +`panel_ldlt_kernel!` and up to two GEMMs per front per block of `nb` pivot +columns, the finish kernel and one contribution-block pack per front; plus the +`max |aᵢⱼ|` and statistics reductions. `lu = true` budgets `factorize_lu!` +instead: every launch group is one fused launch, plus the Schur assembly. +`@localmem`: the largest regime-A kernel (48 KiB) per subtree or fused launch +plus the two reductions (T15). +""" +function ldlt_alloc_budget(S, N; nb::Integer = SparseDirectSolver.LDLT_C_NB, lu::Bool = false) + plan, sc = N.plan, S.schedule + members = N.nbatch + fused = length(plan.sub_first) + blocked = lu ? 1 : 0 + for k in eachindex(plan.group_first) + a, b = plan.group_first[k], plan.group_last[k] + if !lu && (plan.group_width[k] == 0 || SparseDirectSolver.ldlt_blocked_path(sc, sc.group_nodes[a])) + blocked += 3 + q = a + while q <= b + e = SparseDirectSolver.ldlt_c_chunk_end(sc, S.layout.cb_len, q, b, S.layout.work_len) + nblocks = maximum(cld(sc.width[sc.group_nodes[r]], nb) for r in q:e) + blocked += members * (nblocks * (1 + 2 * (e - q + 1)) + 1 + (e - q + 1)) + q = e + 1 + end + else + fused += 1 + end + end + launches = fused + blocked + 2 + localmem = (fused + 2) * maximum(SparseDirectSolver.SUBTREE_LOCAL_SIZES) + SparseDirectSolver.STATS_WORKGROUP * SparseDirectSolver.FRONT_STATS_FIELDS * sizeof(Int64) return ka_cpu_alloc_budget(launches, localmem) end From 2b1ff4c74aa68d34b23db64b8f0841024755e2ea Mon Sep 17 00:00:00 2001 From: Sungho Shin Date: Tue, 13 Oct 2026 12:41:57 -0400 Subject: [PATCH 3/5] Aqua workflow: gate persistent_tasks off under KernelAbstractions >= 0.10 The review's blocking finding: test/test_aqua.jl gained the version-gated skip but the standalone Aqua workflow still ran the check unconditionally, and its fresh ubuntu-latest environment resolves KA 0.10.0, so the build job went red (pocl_standalone_jll fails to precompile in Aqua's temporary project: "Cannot load SLEEF_jll"). Apply the same gate there, with both upstream failure modes documented at both sites. Co-Authored-By: Claude Fable 5 --- .github/workflows/Aqua.yml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/workflows/Aqua.yml b/.github/workflows/Aqua.yml index af731a04..60657141 100644 --- a/.github/workflows/Aqua.yml +++ b/.github/workflows/Aqua.yml @@ -16,5 +16,9 @@ jobs: - name: Aqua.jl # MadNLP is loaded so the SparseDirectSolverMadNLPExt extension assigns the # exported SDSSolver/SDSSolverOptions globals: the undefined-exports check - # then validates them for real instead of flagging the declared-only state - run: julia --color=yes -e 'using Pkg; Pkg.add(["Aqua", "MadNLP"]); Pkg.develop(path="."); using Aqua, MadNLP, SparseDirectSolver; Aqua.test_all(SparseDirectSolver, ambiguities=false)' + # then validates them for real instead of flagging the declared-only state. + # persistent_tasks is gated off under KernelAbstractions >= 0.10 for the same + # two upstream reasons as in test/test_aqua.jl: KA 0.10's CPU backend keeps + # worker tasks alive at exit, and Aqua's fresh temporary project fails to + # precompile pocl_standalone_jll ("Cannot load SLEEF_jll"). + run: julia --color=yes -e 'using Pkg; Pkg.add(["Aqua", "MadNLP", "KernelAbstractions"]); Pkg.develop(path="."); using Aqua, MadNLP, KernelAbstractions, SparseDirectSolver; Aqua.test_all(SparseDirectSolver, ambiguities=false, persistent_tasks=(pkgversion(KernelAbstractions) >= v"0.10" ? false : true))' From 73b397bcc198f889fa3ed0a0e3f96c079bf965ce Mon Sep 17 00:00:00 2001 From: "claude[bot]" <41898282+claude[bot]@users.noreply.github.com> Date: Tue, 13 Oct 2026 17:18:43 +0000 Subject: [PATCH 4/5] KA 0.10 compat: address review round 2 ldlt_alloc_budget: keep the 48 KiB coverage-mode localmem allowance per blocked group as before; the LU Schur assembly is two launches. (The Aqua.yml YAML fix needs the owner: the App token lacks the workflows permission.) Co-Authored-By: Claude Opus 5.5 --- test/utils.jl | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/test/utils.jl b/test/utils.jl index 331ba1bb..dd7a3215 100644 --- a/test/utils.jl +++ b/test/utils.jl @@ -493,19 +493,22 @@ assembly launches and, per chunk (`ldlt_c_chunk_end`) and batch member, one `panel_ldlt_kernel!` and up to two GEMMs per front per block of `nb` pivot columns, the finish kernel and one contribution-block pack per front; plus the `max |aᵢⱼ|` and statistics reductions. `lu = true` budgets `factorize_lu!` -instead: every launch group is one fused launch, plus the Schur assembly. -`@localmem`: the largest regime-A kernel (48 KiB) per subtree or fused launch -plus the two reductions (T15). +instead: every launch group is one fused launch, plus the two Schur assembly +launches. `@localmem` (heap-allocated only under `--code-coverage` on KA 0.9): +the largest regime-A kernel (48 KiB) per subtree launch and per launch group, +blocked groups included as before, plus the two reductions (T15). """ function ldlt_alloc_budget(S, N; nb::Integer = SparseDirectSolver.LDLT_C_NB, lu::Bool = false) plan, sc = N.plan, S.schedule members = N.nbatch fused = length(plan.sub_first) - blocked = lu ? 1 : 0 + blocked = lu ? 2 : 0 + nblocked = 0 for k in eachindex(plan.group_first) a, b = plan.group_first[k], plan.group_last[k] if !lu && (plan.group_width[k] == 0 || SparseDirectSolver.ldlt_blocked_path(sc, sc.group_nodes[a])) blocked += 3 + nblocked += 1 q = a while q <= b e = SparseDirectSolver.ldlt_c_chunk_end(sc, S.layout.cb_len, q, b, S.layout.work_len) @@ -518,7 +521,7 @@ function ldlt_alloc_budget(S, N; nb::Integer = SparseDirectSolver.LDLT_C_NB, lu: end end launches = fused + blocked + 2 - localmem = (fused + 2) * maximum(SparseDirectSolver.SUBTREE_LOCAL_SIZES) + + localmem = (fused + nblocked + 2) * maximum(SparseDirectSolver.SUBTREE_LOCAL_SIZES) + SparseDirectSolver.STATS_WORKGROUP * SparseDirectSolver.FRONT_STATS_FIELDS * sizeof(Int64) return ka_cpu_alloc_budget(launches, localmem) end From 7811198ee6854ea87d55ae015da4364c71a58875 Mon Sep 17 00:00:00 2001 From: Sungho Shin Date: Tue, 13 Oct 2026 13:49:45 -0400 Subject: [PATCH 5/5] Aqua workflow: write the persistent_tasks gate without a ternary The previous gate's ' ? false : true' put ': ' inside the plain YAML scalar, which is invalid YAML: the Aqua workflow run had zero jobs on the last two heads and the check silently vanished from the PR instead of failing. Same semantics via comparison: persistent_tasks=(pkgversion(KernelAbstractions) < v"0.10"). Parse-checked the YAML and the Julia expression locally. Co-Authored-By: Claude Fable 5 --- .github/workflows/Aqua.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/Aqua.yml b/.github/workflows/Aqua.yml index 60657141..0de75e4a 100644 --- a/.github/workflows/Aqua.yml +++ b/.github/workflows/Aqua.yml @@ -21,4 +21,4 @@ jobs: # two upstream reasons as in test/test_aqua.jl: KA 0.10's CPU backend keeps # worker tasks alive at exit, and Aqua's fresh temporary project fails to # precompile pocl_standalone_jll ("Cannot load SLEEF_jll"). - run: julia --color=yes -e 'using Pkg; Pkg.add(["Aqua", "MadNLP", "KernelAbstractions"]); Pkg.develop(path="."); using Aqua, MadNLP, KernelAbstractions, SparseDirectSolver; Aqua.test_all(SparseDirectSolver, ambiguities=false, persistent_tasks=(pkgversion(KernelAbstractions) >= v"0.10" ? false : true))' + run: julia --color=yes -e 'using Pkg; Pkg.add(["Aqua", "MadNLP", "KernelAbstractions"]); Pkg.develop(path="."); using Aqua, MadNLP, KernelAbstractions, SparseDirectSolver; Aqua.test_all(SparseDirectSolver, ambiguities=false, persistent_tasks=(pkgversion(KernelAbstractions) < v"0.10"))'