Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .github/workflows/Aqua.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,9 @@ jobs:
- name: Aqua.jl
# MadNLP is loaded so the SparseDirectSolverMadNLPExt extension assigns the
# exported SDSSolver/SDSSolverOptions globals: the undefined-exports check
# then validates them for real instead of flagging the declared-only state
run: julia --color=yes -e 'using Pkg; Pkg.add(["Aqua", "MadNLP"]); Pkg.develop(path="."); using Aqua, MadNLP, SparseDirectSolver; Aqua.test_all(SparseDirectSolver, ambiguities=false)'
# then validates them for real instead of flagging the declared-only state.
# persistent_tasks is gated off under KernelAbstractions >= 0.10 for the same
# two upstream reasons as in test/test_aqua.jl: KA 0.10's CPU backend keeps
# worker tasks alive at exit, and Aqua's fresh temporary project fails to
# precompile pocl_standalone_jll ("Cannot load SLEEF_jll").
run: julia --color=yes -e 'using Pkg; Pkg.add(["Aqua", "MadNLP", "KernelAbstractions"]); Pkg.develop(path="."); using Aqua, MadNLP, KernelAbstractions, SparseDirectSolver; Aqua.test_all(SparseDirectSolver, ambiguities=false, persistent_tasks=(pkgversion(KernelAbstractions) < v"0.10"))'
7 changes: 4 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,10 @@ Three documents define the work. Read them in this order before touching code:
§1.2–§1.4 and §3.1. They mirror `../CUDSS.jl` so MadNLP can migrate
mechanically.
* New dependencies need a one-line justification in the Report. Never `dev`
unregistered or local checkouts into the project (there is a
`../KernelAbstractions.jl` 0.10-dev checkout on this machine; the package
uses the registered KernelAbstractions 0.9.x).
unregistered or local checkouts into the project (the package admits the
registered KernelAbstractions 0.9.x and 0.10.x; the GPU stacks still resolve
0.9 until CUDA.jl/AMDGPU.jl ship 0.10 support, and the test allocation
budgets carry a per-launch allowance on 0.10 -- see `ka_cpu_alloc_budget`).
* The repository is `github.com/exanauts/SparseDirectSolver.jl` (remote
`origin`, branch `main`). Commit at the end of the task with a message
starting with the task id, e.g. `T05: symbolic pattern, ordering, etree,
Expand Down
2 changes: 1 addition & 1 deletion Project.toml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ CUDACore = "6"
CliqueTrees = "1"
GPUArrays = "11, 12"
GPUArraysCore = "0.2"
KernelAbstractions = "0.9"
KernelAbstractions = "0.9, 0.10"
Krylov = "0.10"
LinearAlgebra = "1.13"
MadNLP = "0.8, 0.9, 0.10"
Expand Down
2 changes: 1 addition & 1 deletion test/Project.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
Adapt = "4"
Aqua = "0.8"
DelimitedFiles = "1"
KernelAbstractions = "0.9"
KernelAbstractions = "0.9, 0.10"
Krylov = "0.10"
LinearAlgebra = "1.13"
ParallelTestRunner = "2"
Expand Down
7 changes: 6 additions & 1 deletion test/test_aqua.jl
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,10 @@
# undefined_exports is excluded HERE only: SDSSolver/SDSSolverOptions are declared
# globals the MadNLP extension assigns, and MadNLP is not in the test environment.
# CI's Aqua workflow loads MadNLP and runs that check with the exports assigned.
# persistent_tasks is skipped under KernelAbstractions >= 0.10, where it reports the
# backend, not this package: KA 0.10's CPU backend keeps worker tasks alive at exit,
# and in Aqua's fresh temporary project its pocl_standalone_jll dependency fails to
# precompile ("Cannot load SLEEF_jll into module pocl_standalone_jll", seen on CI).
# Upstream; re-enable when both are fixed. CI's Aqua workflow applies the same gate.
Aqua.test_all(SparseDirectSolver; ambiguities = false, undefined_exports = false,
persistent_tasks = (tmax = 120,))
persistent_tasks = (pkgversion(KernelAbstractions) >= v"0.10" ? false : (tmax = 120,)))
2 changes: 1 addition & 1 deletion test/test_numeric_ldlt.jl
Original file line number Diff line number Diff line change
Expand Up @@ -304,7 +304,7 @@ end
# no allocation in the numeric phase and the solve (CPU backend)
if backend isa CPU
SDS.factorize!(Nd, Sd, nz; opts)
@test (@allocated SDS.factorize!(Nd, Sd, nz; opts)) <= ldlt_alloc_budget(S)
@test (@allocated SDS.factorize!(Nd, Sd, nz; opts)) <= ldlt_alloc_budget(S, Nd)
x = similar(b)
SDS.sweep_solve!(x, ws, Sd, Nd, b)
@test (@allocated SDS.sweep_solve!(x, ws, Sd, Nd, b)) <= solve_alloc_budget(S, ws; ldlt = true)
Expand Down
2 changes: 1 addition & 1 deletion test/test_numeric_lu.jl
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@ end
# no allocation in the numeric phase and the solve (CPU backend)
if backend isa CPU
SDS.factorize!(Nd, Sd, nz; opts)
@test (@allocated SDS.factorize!(Nd, Sd, nz; opts)) <= ldlt_alloc_budget(S)
@test (@allocated SDS.factorize!(Nd, Sd, nz; opts)) <= ldlt_alloc_budget(S, Nd; lu = true)
x = similar(b)
for transpose in (false, true)
SDS.sweep_solve!(x, ws, Sd, Nd, b; transpose)
Expand Down
59 changes: 48 additions & 11 deletions test/utils.jl
Original file line number Diff line number Diff line change
Expand Up @@ -241,12 +241,20 @@ kernel launches whose `@localmem` buffers total `localmem_bytes`, on top of
the 1024 B allowed to an allocation-free phase (PLAN §3.9). Measured (T09
review round 1): 0 without coverage (the local runs, Julia ≥ 1.13 is required);
with `--code-coverage` (CI's `julia-runtest`) the `@localmem` `MArray`s are
heap-allocated, hence `localmem_bytes` plus 64 B of header per launch. (Julia
heap-allocated, hence `localmem_bytes` plus 64 B of header per launch. Under
KernelAbstractions ≥ 0.10 the CPU backend heap-allocates on every launch
(~230 B measured), hence 320 B per launch on that version only. (Julia
≤ 1.11 also boxed the arguments of every launch behind KA's `__run` inference
barrier; that allowance went with the Julia 1.10 support.)
"""
function ka_cpu_alloc_budget(launches::Integer, localmem_bytes::Integer)
b = 1024
# KernelAbstractions 0.10's CPU backend heap-allocates per launch (~230 B measured
# with a trivial precompiled kernel; the new backend's task machinery), where 0.9
# was allocation-free. 320 B of per-launch headroom keeps the budgets meaningful --
# they still scale with the phase's launch count -- until the upstream allocation
# is fixed; the 0.9 contract stays exact.
pkgversion(KernelAbstractions) >= v"0.10" && (b += 320 * launches)
Comment on lines +252 to +257

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit (non-blocking): the docstring of ka_cpu_alloc_budget (lines 237–247) still states "0 without coverage" as the measured per-launch cost and only lists the coverage and Julia ≤ 1.11 allowances; the KA ≥ 0.10 320 B/launch term lives only in this code comment. Move a one-line mention into the docstring so the budget's three terms are documented in one place (it is the helper the task reports cite).

Base.JLOptions().code_coverage != 0 && (b += localmem_bytes + 64 * launches)
return b
end
Expand Down Expand Up @@ -475,16 +483,45 @@ only on generators without element growth).
d_error(Nh, Nr) = maximum(abs, Nh.d - Nr.d) / maximum(abs, Nr.d)

"""
ldlt_alloc_budget(S) -> Int

[`ka_cpu_alloc_budget`](@ref) for one LDLᵀ/LDLᴴ `factorize!` on the CPU
backend: one launch per launch group, the `max |aᵢⱼ|` and statistics
reductions; `@localmem` of the largest regime-A kernel (48 KiB) per group plus
the two reductions (T15).
"""
function ldlt_alloc_budget(S)
launches = length(S.schedule.groups) + 2
localmem = launches * maximum(SparseDirectSolver.SUBTREE_LOCAL_SIZES) +
ldlt_alloc_budget(S, N; nb = SparseDirectSolver.LDLT_C_NB, lu = false) -> Int

[`ka_cpu_alloc_budget`](@ref) for one LDLᵀ/LDLᴴ `factorize!` of the numeric
storage `N` (its launch plan `N.plan`) on the CPU backend, counting the
launches the driver makes: one per subtree launch and per fused regime-A/B
launch group; for a group on the blocked path (`ldlt_blocked_path`) the three
assembly launches and, per chunk (`ldlt_c_chunk_end`) and batch member, one
`panel_ldlt_kernel!` and up to two GEMMs per front per block of `nb` pivot
columns, the finish kernel and one contribution-block pack per front; plus the
`max |aᵢⱼ|` and statistics reductions. `lu = true` budgets `factorize_lu!`
instead: every launch group is one fused launch, plus the two Schur assembly
launches. `@localmem` (heap-allocated only under `--code-coverage` on KA 0.9):
the largest regime-A kernel (48 KiB) per subtree launch and per launch group,
blocked groups included as before, plus the two reductions (T15).
"""
function ldlt_alloc_budget(S, N; nb::Integer = SparseDirectSolver.LDLT_C_NB, lu::Bool = false)
plan, sc = N.plan, S.schedule
members = N.nbatch
fused = length(plan.sub_first)
blocked = lu ? 2 : 0
nblocked = 0
for k in eachindex(plan.group_first)
a, b = plan.group_first[k], plan.group_last[k]
if !lu && (plan.group_width[k] == 0 || SparseDirectSolver.ldlt_blocked_path(sc, sc.group_nodes[a]))
blocked += 3
nblocked += 1
q = a
while q <= b
e = SparseDirectSolver.ldlt_c_chunk_end(sc, S.layout.cb_len, q, b, S.layout.work_len)
nblocks = maximum(cld(sc.width[sc.group_nodes[r]], nb) for r in q:e)
blocked += members * (nblocks * (1 + 2 * (e - q + 1)) + 1 + (e - q + 1))
q = e + 1
end
else
fused += 1
end
end
launches = fused + blocked + 2
localmem = (fused + nblocked + 2) * maximum(SparseDirectSolver.SUBTREE_LOCAL_SIZES) +
SparseDirectSolver.STATS_WORKGROUP * SparseDirectSolver.FRONT_STATS_FIELDS * sizeof(Int64)
return ka_cpu_alloc_budget(launches, localmem)
end
Expand Down
Loading