diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d1d1c3b5..4fd569fa 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -9,6 +9,19 @@ on: types: [opened, synchronize, reopened] schedule: - cron: '0 0 * * 0' + workflow_dispatch: + inputs: + eltypes: + description: 'SDS_TEST_ELTYPES: comma-separated element types; empty runs all four' + required: false + default: 'Float32,Float64,ComplexF32,ComplexF64' + type: string + +# Element types under test (test/backends.jl, issue #91): pull requests and pushes run one real and one complex +# type, the two that differ most in storage; the weekly schedule runs all four (the empty string), a manual run +# what its input says (default all four). Kernel compilation per element type sets the test time. +env: + SDS_TEST_ELTYPES: ${{ (github.event_name == 'pull_request' || github.event_name == 'push') && 'Float64,ComplexF32' || inputs.eltypes }} jobs: # Runs on the self-hosted kkt machine (more cores than a GitHub runner for the parallel test workers). The diff --git a/AGENTS.md b/AGENTS.md index e969ae1a..ab717317 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -151,6 +151,9 @@ SDS_TEST_CPU=0 julia --project=. -e 'using Pkg; Pkg.test()' SDS_TEST_ONLY="test_symbolic_etree,test_options" julia --project=. -e 'using Pkg; Pkg.test()' SDS_TEST_SKIP="test_aqua" julia --project=. -e 'using Pkg; Pkg.test()' +# fewer element types (default all four), as pull-request CI runs +SDS_TEST_ELTYPES="Float64,ComplexF32" julia --project=. -e 'using Pkg; Pkg.test()' + # the same with ParallelTestRunner arguments (prefix match, `!` excludes), and the number of workers julia --project=. -e 'using Pkg; Pkg.test(; test_args = ["test_symbolic", "!test_symbolic_etree"])' PTR_NUM_JOBS=4 julia --project=. -e 'using Pkg; Pkg.test()' @@ -171,19 +174,27 @@ julia --project=bench/report bench/compare_report.jl `SDS_TEST_GPU` and `SDS_TEST_ONLY` are implemented in `test/runtests.jl` (T01), `SDS_TEST_CPU` (in `test/backends.jl`) and `SDS_TEST_SKIP` were added for CI: the CUDA jobs run with `SDS_TEST_CPU=0 SDS_TEST_SKIP=test_aqua`, since the CPU-only -jobs already cover the CPU backend and Aqua. A task's "passes on CPU and CUDA" -still means the default run (both backends) on the owner's machine. +jobs already cover the CPU backend and Aqua. `SDS_TEST_ELTYPES` (in +`test/backends.jl`, issue #91) selects the element types: pull-request and push CI +runs `Float64,ComplexF32`, the weekly scheduled run (and a manual run, by default) +all four. A task's "passes on CPU and CUDA" still means the default run (both +backends, all four element types) on the owner's machine. The test files run in parallel through ParallelTestRunner.jl: each `test_*.jl` is evaluated in its own module on a pool of worker processes, after the packages, -`test/utils.jl`, `test/matrices.jl`, `test/backends.jl` and `Random.seed!(666)` +`test/backends.jl`, `test/utils.jl`, `test/matrices.jl` and `Random.seed!(666)` (`test/runtests.jl`). A test file therefore cannot use definitions from another test file; shared code belongs in the helpers. A worker's cold compilation, not the tests, sets the wall time, so the long files (`SPLIT_FILES` in `test/runtests.jl`) -run once per element type (`test_api[Float64]`, …): `ELTYPES`, `REAL_ELTYPES` and -`COMPLEX_ELTYPES` are then that part's subset, and a testset that does not loop over -the element types runs in the `Float64` part only (`RUN_SHARED && @testset …`). In -a split file, write every testset either over `ELTYPES` or behind `RUN_SHARED`. +run once per selected element type (`test_api[Float64]`, …): `ELTYPES`, `REAL_ELTYPES` +and `COMPLEX_ELTYPES` are then that part's subset, and a testset that does not loop +over the element types runs in the `Float64` part only (the first selected type's +when `Float64` is not selected; `RUN_SHARED && @testset …`). In a split file, write +every testset either over `ELTYPES` or behind `RUN_SHARED`. A numeric testset for +some element types only loops over `eltypes_among((Float64, ComplexF32))` or +`eltypes_among(Complex)`, never over a hard-coded list or `ELTYPES[k]`, so any +selection with a real type works (host-only symbolic checks of element sizes may +name their types). ## Code conventions @@ -217,7 +228,10 @@ a split file, write every testset either over `ELTYPES` or behind `RUN_SHARED`. `test/utils.jl` (created in T01). Use them; do not define new generators or tolerances inside individual test files. * Every numeric test loops over `BACKENDS` and over `ELTYPES` unless the task - says otherwise. Seed with `Random.seed!(666)`. Tolerance is `tol(T) = + says otherwise. `ELTYPES` is the set selected by `SDS_TEST_ELTYPES` (default + all four; pull-request CI runs `Float64,ComplexF32`, the weekly run all four), + so a test passing on a pull request has seen two element types; a task's + "passes on CPU and CUDA" means the full set on the owner's machine. Seed with `Random.seed!(666)`. Tolerance is `tol(T) = sqrt(eps(real(T)))` on well-conditioned generators. Elementwise device-vs-reference panel checks use `panel_tol(T)` only on well-conditioned generators; on generators with element growth (weak pivots, static pivoting) diff --git a/README.md b/README.md index 9681731a..d337072d 100644 --- a/README.md +++ b/README.md @@ -181,6 +181,7 @@ SDS_TEST_GPU=0 julia --project=. -e 'using Pkg; Pkg.test()' # CPU only SDS_TEST_CPU=0 julia --project=. -e 'using Pkg; Pkg.test()' # GPU only SDS_TEST_ONLY="test_symbolic_etree,test_options" julia --project=. -e 'using Pkg; Pkg.test()' SDS_TEST_SKIP="test_aqua" julia --project=. -e 'using Pkg; Pkg.test()' +SDS_TEST_ELTYPES="Float64,ComplexF32" julia --project=. -e 'using Pkg; Pkg.test()' # element types (default all four) PTR_NUM_JOBS=4 julia --project=. -e 'using Pkg; Pkg.test()' # number of test workers ``` diff --git a/test/backends.jl b/test/backends.jl index 516c37b3..2abb668a 100644 --- a/test/backends.jl +++ b/test/backends.jl @@ -13,6 +13,37 @@ using SparseArrays const TEST_CPU = get(ENV, "SDS_TEST_CPU", "1") != "0" const TEST_GPU = get(ENV, "SDS_TEST_GPU", "1") != "0" +# Element types under test (issue #91): SDS_TEST_ELTYPES is a comma-separated subset of these, default all four. +# Pull-request CI selects two (Float64,ComplexF32), the weekly run and local runs all four. +const ALL_ELTYPES = (Float32, Float64, ComplexF32, ComplexF64) + +""" + selected_eltypes(spec = ENV["SDS_TEST_ELTYPES"]) -> Tuple + +The element types named by the comma-separated `spec` (`"Float64,ComplexF32"`), in the order of +`ALL_ELTYPES`; all four when `spec` is empty. Unknown names and a selection without a real type are errors +(the testsets that do not loop over the element types need a real one). +""" +function selected_eltypes(spec::AbstractString = get(ENV, "SDS_TEST_ELTYPES", "")) + names = [String(strip(s)) for s in split(spec, ',') if !isempty(strip(s))] + isempty(names) && return ALL_ELTYPES + known = Dict(("Float32", "Float64", "ComplexF32", "ComplexF64") .=> ALL_ELTYPES) + unknown = setdiff(names, keys(known)) + isempty(unknown) || + error("SDS_TEST_ELTYPES names unknown element types $(unknown); available: $(join(keys(known), ", "))") + eltypes = Tuple(T for T in ALL_ELTYPES if any(n -> known[n] === T, names)) + any(T -> T <: Real, eltypes) || error("SDS_TEST_ELTYPES = \"$spec\" selects no real element type") + return eltypes +end + +""" + SELECTED_ELTYPES + +The element types of this run (`SDS_TEST_ELTYPES`, see [`selected_eltypes`](@ref)). `runtests.jl` splits the +long test files over them; `ELTYPES` (`utils.jl`) is the subset of the running part. +""" +const SELECTED_ELTYPES = selected_eltypes() + is_package_installed(name::String) = Base.find_package(name) !== nothing const CUDA_LOADED = TEST_GPU && is_package_installed("CUDA") @@ -188,6 +219,8 @@ function print_backends() TEST_CPU || push!(gpus, "CPU backend disabled by SDS_TEST_CPU=0") println("Backends under test: ", join(backend_name.(BACKENDS), ", "), isempty(gpus) ? "" : " [" * join(gpus, "; ") * "]") + println("Element types under test: ", join(SELECTED_ELTYPES, ", "), + SELECTED_ELTYPES == ALL_ELTYPES ? "" : " [selected by SDS_TEST_ELTYPES]") return nothing end diff --git a/test/runtests.jl b/test/runtests.jl index 6c4636b8..17da5af4 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -4,7 +4,8 @@ # # Run a subset with SDS_TEST_ONLY="test_options,test_aqua" or with test arguments (prefix match, `!name` excludes: # `Pkg.test(; test_args = ["test_symbolic", "!test_symbolic_etree"])`), leave files out with SDS_TEST_SKIP="test_aqua", -# skip the GPU backends with SDS_TEST_GPU=0 and the CPU backend with SDS_TEST_CPU=0 (see backends.jl). The number of +# skip the GPU backends with SDS_TEST_GPU=0 and the CPU backend with SDS_TEST_CPU=0, and select the element types with +# SDS_TEST_ELTYPES="Float64,ComplexF32" (default all four; see backends.jl). The number of # workers is ParallelTestRunner's default (CPU threads and free memory); set it with `--jobs=N` or PTR_NUM_JOBS. using ParallelTestRunner @@ -41,20 +42,30 @@ const init_worker_code = quote const SDS_TEST_PART = Ref{Any}(nothing) end -# Long test files run once per element type, "test_api[Float64]" etc., each part in a worker of its own: a -# worker compiles the solver for every element type it tests, and that compilation, not the tests, sets the wall -# time. The testsets that do not loop over the element types run in the Float64 part (`RUN_SHARED`). +# the backends and the element types of the run (`SELECTED_ELTYPES`), read once from the environment in the main +# process: prints the banner and fails early on an empty backend list or a bad SDS_TEST_ELTYPES +module BackendBanner end +Core.eval(BackendBanner, init_code) +Core.eval(BackendBanner, :(include($(joinpath(TEST_DIR, "backends.jl"))))) +BackendBanner.print_backends() + +# Long test files run once per selected element type, "test_api[Float64]" etc., each part in a worker of its own: +# a worker compiles the solver for every element type it tests, and that compilation, not the tests, sets the wall +# time. The testsets that do not loop over the element types run in the Float64 part, or in the first selected +# type's part when SDS_TEST_ELTYPES leaves Float64 out (`RUN_SHARED`). const SPLIT_FILES = ["test_api", "test_dense", "test_fgmres", "test_matching", "test_numeric_cholesky_a", "test_numeric_cholesky_b", "test_numeric_cholesky_c", "test_numeric_ldlt", "test_numeric_lu", "test_ported", "test_refinement", "test_schur", "test_solve", "test_ubatch"] -const SPLIT_ELTYPES = (Float32, Float64, ComplexF32, ComplexF64) +const SPLIT_ELTYPES = BackendBanner.SELECTED_ELTYPES +const SHARED_ELTYPE = Float64 in SPLIT_ELTYPES ? Float64 : first(SPLIT_ELTYPES) -# the seed goes with the test, not into `init_code`: ParallelTestRunner seeds with 1 after `init_code` +# the seed goes with the test, not into `init_code`: ParallelTestRunner seeds with 1 after `init_code`; `backends.jl` +# comes first, `utils.jl` derives `ELTYPES` from its `SELECTED_ELTYPES` test_expr(name, part) = quote Main.SDS_TEST_PART[] = $part + include($(joinpath(TEST_DIR, "backends.jl"))) include($(joinpath(TEST_DIR, "utils.jl"))) include($(joinpath(TEST_DIR, "matrices.jl"))) - include($(joinpath(TEST_DIR, "backends.jl"))) Random.seed!(666) include($(joinpath(TEST_DIR, "$name.jl"))) end @@ -63,7 +74,7 @@ testsuite = Dict{String, Expr}() for name in TEST_FILES if name in SPLIT_FILES for T in SPLIT_ELTYPES - testsuite["$name[$T]"] = test_expr(name, ((T,), T == Float64)) + testsuite["$name[$T]"] = test_expr(name, ((T,), T == SHARED_ELTYPE)) end else testsuite[name] = test_expr(name, nothing) @@ -80,11 +91,5 @@ if filter_tests!(testsuite, args) filter!(t -> !names_test(skip, first(t)), testsuite) end -# the backend banner once, from the main process (also fails early when no backend is left to test) -module BackendBanner end -Core.eval(BackendBanner, init_code) -Core.eval(BackendBanner, :(include($(joinpath(TEST_DIR, "backends.jl"))))) -BackendBanner.print_backends() - runtests(SparseDirectSolver, args; testsuite, init_code, init_worker_code, history_key = get(ENV, "SDS_TEST_CPU", "1") == "0" ? "gpu" : nothing) diff --git a/test/test_fgmres.jl b/test/test_fgmres.jl index 03b79f08..7b8ae3ae 100644 --- a/test/test_fgmres.jl +++ b/test/test_fgmres.jl @@ -74,7 +74,7 @@ end end @testset "FGMRES-IR: uniform batch ($(backend_name(backend)), $T)" for backend in BACKENDS, - T in filter(in((Float64, ComplexF32)), ELTYPES) + T in eltypes_among((Float64, ComplexF32)) Random.seed!(666) n, nb, nrhs = 120, 3, 2 members = batch_members(random_symindef(T, n, 0.03), nb) diff --git a/test/test_helpers.jl b/test/test_helpers.jl index 1eb1384d..11090441 100644 --- a/test/test_helpers.jl +++ b/test/test_helpers.jl @@ -157,3 +157,18 @@ end @test spd_structure(Float32) == "SPD" && spd_structure(ComplexF32) == "HPD" @test sym_structure(Float64) == "S" && sym_structure(ComplexF64) == "H" end + +@testset "element type selection (SDS_TEST_ELTYPES)" begin + @test selected_eltypes("") == ALL_ELTYPES == (Float32, Float64, ComplexF32, ComplexF64) + @test selected_eltypes("ComplexF32, Float64") == (Float64, ComplexF32) # the order of ALL_ELTYPES + @test selected_eltypes("Float32,Float32,") == (Float32,) + @test_throws ErrorException selected_eltypes("Float16") + @test_throws ErrorException selected_eltypes("ComplexF32,ComplexF64") # no real type + @test SELECTED_ELTYPES == selected_eltypes() + @test ELTYPES == SELECTED_ELTYPES # an unsplit file tests every selected type + @test REAL_ELTYPES == filter(T -> T <: Real, SELECTED_ELTYPES) && !isempty(REAL_ELTYPES) + # `@test_logs` keeps the "testset skipped" note of a selection without these types out of the output + @test (@test_logs match_mode = :any eltypes_among(Complex)) == COMPLEX_ELTYPES + @test (@test_logs match_mode = :any eltypes_among((Float64, ComplexF32))) == + filter(in((Float64, ComplexF32)), SELECTED_ELTYPES) +end diff --git a/test/test_numeric_ldlt.jl b/test/test_numeric_ldlt.jl index 9769a072..193776aa 100644 --- a/test/test_numeric_ldlt.jl +++ b/test/test_numeric_ldlt.jl @@ -431,7 +431,7 @@ end end @testset "complex symmetric through the API ($(backend_name(backend)), $T)" for backend in BACKENDS, - T in COMPLEX_ELTYPES + T in eltypes_among(Complex) Random.seed!(666) A = random_symindef(T, 300, 0.02; hermitian = false) solver = DirectSolver(api_matrix(backend, tril(A), Int32), "S", 'L') diff --git a/test/test_reference_ldlt.jl b/test/test_reference_ldlt.jl index 9e89bb2c..4d9c0c47 100644 --- a/test/test_reference_ldlt.jl +++ b/test/test_reference_ldlt.jl @@ -155,7 +155,7 @@ end @test N.factor == Nf.factor && N.d == Nf.d && N.piv == Nf.piv && N.stats == Nf.stats end -@testset "complex symmetric LDLᵀ: $T" for T in COMPLEX_ELTYPES +@testset "complex symmetric LDLᵀ: $T" for T in eltypes_among(Complex) Random.seed!(666) A = random_symindef(T, 300, 0.02; hermitian = false) @test transpose(A) == A && A' != A diff --git a/test/utils.jl b/test/utils.jl index 83ba26ef..9860bcb6 100644 --- a/test/utils.jl +++ b/test/utils.jl @@ -4,11 +4,12 @@ using LinearAlgebra using SparseArrays -# The element types under test. `runtests.jl` runs a long test file once per element type, each part in its own -# worker: the part sets `Main.SDS_TEST_PART[] = (eltypes, shared)` before these helpers are included (`nothing`: -# all element types, one part). +# The element types under test: `SELECTED_ELTYPES` (`backends.jl`, from SDS_TEST_ELTYPES; `backends.jl` is +# included before these helpers). `runtests.jl` runs a long test file once per selected element type, each part in +# its own worker: the part sets `Main.SDS_TEST_PART[] = (eltypes, shared)` before the helpers are included +# (`nothing`: all selected element types, one part). const TEST_PART = isdefined(Main, :SDS_TEST_PART) ? Main.SDS_TEST_PART[] : nothing -const ELTYPES = TEST_PART === nothing ? (Float32, Float64, ComplexF32, ComplexF64) : TEST_PART[1] +const ELTYPES = TEST_PART === nothing ? SELECTED_ELTYPES : TEST_PART[1] const REAL_ELTYPES = filter(T -> T <: Real, ELTYPES) const COMPLEX_ELTYPES = filter(T -> T <: Complex, ELTYPES) @@ -17,9 +18,29 @@ const COMPLEX_ELTYPES = filter(T -> T <: Complex, ELTYPES) `true` when this run of a test file runs the testsets that do not loop over the element types (`RUN_SHARED && @testset …`): in the one part of a test file split -by element type that owns them, and always in an unsplit test file. +by element type that owns them (the `Float64` part, or the first selected type's +when `Float64` is not selected), and always in an unsplit test file. """ const RUN_SHARED = TEST_PART === nothing || TEST_PART[2] + +""" + eltypes_among(types) -> Tuple + eltypes_among(S::Type) -> Tuple + +The element types of this part that are among `types` (or subtypes of `S`), for a +testset that runs for some element types only +(`for T in eltypes_among((Float64, ComplexF32))`, `for T in eltypes_among(Complex)`). +When `SDS_TEST_ELTYPES` selects none of `types`, the part that runs the shared +testsets says that the testset is skipped. +""" +function eltypes_among(types) + if RUN_SHARED && !any(in(types), SELECTED_ELTYPES) + @info "SDS_TEST_ELTYPES selects none of $(join(types, ", ")): testset skipped" + end + return filter(in(types), ELTYPES) +end +eltypes_among(S::Type) = eltypes_among(filter(T -> T <: S, ALL_ELTYPES)) + const INTTYPES = (Int32, Int64) """