CI: select the test element types per run, two on pull requests, all four weekly (#91)

.github/workflows/ci.yml

@@ -9,6 +9,19 @@ on:
     types: [opened, synchronize, reopened]
   schedule:
     - cron: '0 0 * * 0'
+  workflow_dispatch:
+    inputs:
+      eltypes:
+        description: 'SDS_TEST_ELTYPES: comma-separated element types; empty runs all four'
+        required: false
+        default: 'Float32,Float64,ComplexF32,ComplexF64'
+        type: string
+
+# Element types under test (test/backends.jl, issue #91): pull requests and pushes run one real and one complex
+# type, the two that differ most in storage; the weekly schedule runs all four (the empty string), a manual run
+# what its input says (default all four). Kernel compilation per element type sets the test time.
+env:
+  SDS_TEST_ELTYPES: ${{ (github.event_name == 'pull_request' || github.event_name == 'push') && 'Float64,ComplexF32' || inputs.eltypes }}
 
 jobs:
   # Runs on the self-hosted kkt machine (more cores than a GitHub runner for the parallel test workers). The

AGENTS.md

@@ -151,6 +151,9 @@ SDS_TEST_CPU=0 julia --project=. -e 'using Pkg; Pkg.test()'
 SDS_TEST_ONLY="test_symbolic_etree,test_options" julia --project=. -e 'using Pkg; Pkg.test()'
 SDS_TEST_SKIP="test_aqua" julia --project=. -e 'using Pkg; Pkg.test()'
 
+# fewer element types (default all four), as pull-request CI runs
+SDS_TEST_ELTYPES="Float64,ComplexF32" julia --project=. -e 'using Pkg; Pkg.test()'
+
 # the same with ParallelTestRunner arguments (prefix match, `!` excludes), and the number of workers
 julia --project=. -e 'using Pkg; Pkg.test(; test_args = ["test_symbolic", "!test_symbolic_etree"])'
 PTR_NUM_JOBS=4 julia --project=. -e 'using Pkg; Pkg.test()'
@@ -171,19 +174,27 @@ julia --project=bench/report bench/compare_report.jl
 `SDS_TEST_GPU` and `SDS_TEST_ONLY` are implemented in `test/runtests.jl` (T01),
 `SDS_TEST_CPU` (in `test/backends.jl`) and `SDS_TEST_SKIP` were added for CI: the
 CUDA jobs run with `SDS_TEST_CPU=0 SDS_TEST_SKIP=test_aqua`, since the CPU-only
-jobs already cover the CPU backend and Aqua. A task's "passes on CPU and CUDA"
-still means the default run (both backends) on the owner's machine.
+jobs already cover the CPU backend and Aqua. `SDS_TEST_ELTYPES` (in
+`test/backends.jl`, issue #91) selects the element types: pull-request and push CI
+runs `Float64,ComplexF32`, the weekly scheduled run (and a manual run, by default)
+all four. A task's "passes on CPU and CUDA" still means the default run (both
+backends, all four element types) on the owner's machine.
 
 The test files run in parallel through ParallelTestRunner.jl: each `test_*.jl` is
 evaluated in its own module on a pool of worker processes, after the packages,
-`test/utils.jl`, `test/matrices.jl`, `test/backends.jl` and `Random.seed!(666)`
+`test/backends.jl`, `test/utils.jl`, `test/matrices.jl` and `Random.seed!(666)`
 (`test/runtests.jl`). A test file therefore cannot use definitions from another
 test file; shared code belongs in the helpers. A worker's cold compilation, not the
 tests, sets the wall time, so the long files (`SPLIT_FILES` in `test/runtests.jl`)
-run once per element type (`test_api[Float64]`, …): `ELTYPES`, `REAL_ELTYPES` and
-`COMPLEX_ELTYPES` are then that part's subset, and a testset that does not loop over
-the element types runs in the `Float64` part only (`RUN_SHARED && @testset …`). In
-a split file, write every testset either over `ELTYPES` or behind `RUN_SHARED`.
+run once per selected element type (`test_api[Float64]`, …): `ELTYPES`, `REAL_ELTYPES`
+and `COMPLEX_ELTYPES` are then that part's subset, and a testset that does not loop
+over the element types runs in the `Float64` part only (the first selected type's
+when `Float64` is not selected; `RUN_SHARED && @testset …`). In a split file, write
+every testset either over `ELTYPES` or behind `RUN_SHARED`. A numeric testset for
+some element types only loops over `eltypes_among((Float64, ComplexF32))` or
+`eltypes_among(Complex)`, never over a hard-coded list or `ELTYPES[k]`, so any
+selection with a real type works (host-only symbolic checks of element sizes may
+name their types).
 
 ## Code conventions
 
@@ -217,7 +228,10 @@ a split file, write every testset either over `ELTYPES` or behind `RUN_SHARED`.
   `test/utils.jl` (created in T01). Use them; do not define new generators or
   tolerances inside individual test files.
 * Every numeric test loops over `BACKENDS` and over `ELTYPES` unless the task
-  says otherwise. Seed with `Random.seed!(666)`. Tolerance is `tol(T) =
+  says otherwise. `ELTYPES` is the set selected by `SDS_TEST_ELTYPES` (default
+  all four; pull-request CI runs `Float64,ComplexF32`, the weekly run all four),
+  so a test passing on a pull request has seen two element types; a task's
+  "passes on CPU and CUDA" means the full set on the owner's machine. Seed with `Random.seed!(666)`. Tolerance is `tol(T) =
   sqrt(eps(real(T)))` on well-conditioned generators. Elementwise
   device-vs-reference panel checks use `panel_tol(T)` only on well-conditioned
   generators; on generators with element growth (weak pivots, static pivoting)

README.md

@@ -181,6 +181,7 @@ SDS_TEST_GPU=0 julia --project=. -e 'using Pkg; Pkg.test()'        # CPU only
 SDS_TEST_CPU=0 julia --project=. -e 'using Pkg; Pkg.test()'        # GPU only
 SDS_TEST_ONLY="test_symbolic_etree,test_options" julia --project=. -e 'using Pkg; Pkg.test()'
 SDS_TEST_SKIP="test_aqua" julia --project=. -e 'using Pkg; Pkg.test()'
+SDS_TEST_ELTYPES="Float64,ComplexF32" julia --project=. -e 'using Pkg; Pkg.test()'  # element types (default all four)
 PTR_NUM_JOBS=4 julia --project=. -e 'using Pkg; Pkg.test()'                 # number of test workers
 ```
 

test/backends.jl

@@ -13,6 +13,37 @@ using SparseArrays
 const TEST_CPU = get(ENV, "SDS_TEST_CPU", "1") != "0"
 const TEST_GPU = get(ENV, "SDS_TEST_GPU", "1") != "0"
 
+# Element types under test (issue #91): SDS_TEST_ELTYPES is a comma-separated subset of these, default all four.
+# Pull-request CI selects two (Float64,ComplexF32), the weekly run and local runs all four.
+const ALL_ELTYPES = (Float32, Float64, ComplexF32, ComplexF64)
+
+"""
+    selected_eltypes(spec = ENV["SDS_TEST_ELTYPES"]) -> Tuple
+
+The element types named by the comma-separated `spec` (`"Float64,ComplexF32"`), in the order of
+`ALL_ELTYPES`; all four when `spec` is empty. Unknown names and a selection without a real type are errors
+(the testsets that do not loop over the element types need a real one).
+"""
+function selected_eltypes(spec::AbstractString = get(ENV, "SDS_TEST_ELTYPES", ""))
+    names = [String(strip(s)) for s in split(spec, ',') if !isempty(strip(s))]
+    isempty(names) && return ALL_ELTYPES
+    known = Dict(("Float32", "Float64", "ComplexF32", "ComplexF64") .=> ALL_ELTYPES)
+    unknown = setdiff(names, keys(known))
+    isempty(unknown) ||
+        error("SDS_TEST_ELTYPES names unknown element types $(unknown); available: $(join(keys(known), ", "))")
+    eltypes = Tuple(T for T in ALL_ELTYPES if any(n -> known[n] === T, names))
+    any(T -> T <: Real, eltypes) || error("SDS_TEST_ELTYPES = \"$spec\" selects no real element type")
+    return eltypes
+end
+
+"""
+    SELECTED_ELTYPES
+
+The element types of this run (`SDS_TEST_ELTYPES`, see [`selected_eltypes`](@ref)). `runtests.jl` splits the
+long test files over them; `ELTYPES` (`utils.jl`) is the subset of the running part.
+"""
+const SELECTED_ELTYPES = selected_eltypes()
+
 is_package_installed(name::String) = Base.find_package(name) !== nothing
 
 const CUDA_LOADED = TEST_GPU && is_package_installed("CUDA")
@@ -188,6 +219,8 @@ function print_backends()
     TEST_CPU || push!(gpus, "CPU backend disabled by SDS_TEST_CPU=0")
     println("Backends under test: ", join(backend_name.(BACKENDS), ", "),
             isempty(gpus) ? "" : "  [" * join(gpus, "; ") * "]")
+    println("Element types under test: ", join(SELECTED_ELTYPES, ", "),
+            SELECTED_ELTYPES == ALL_ELTYPES ? "" : "  [selected by SDS_TEST_ELTYPES]")
     return nothing
 end
 

test/runtests.jl

@@ -4,7 +4,8 @@
 #
 # Run a subset with SDS_TEST_ONLY="test_options,test_aqua" or with test arguments (prefix match, `!name` excludes:
 # `Pkg.test(; test_args = ["test_symbolic", "!test_symbolic_etree"])`), leave files out with SDS_TEST_SKIP="test_aqua",
-# skip the GPU backends with SDS_TEST_GPU=0 and the CPU backend with SDS_TEST_CPU=0 (see backends.jl). The number of
+# skip the GPU backends with SDS_TEST_GPU=0 and the CPU backend with SDS_TEST_CPU=0, and select the element types with
+# SDS_TEST_ELTYPES="Float64,ComplexF32" (default all four; see backends.jl). The number of
 # workers is ParallelTestRunner's default (CPU threads and free memory); set it with `--jobs=N` or PTR_NUM_JOBS.
 
 using ParallelTestRunner
@@ -41,20 +42,30 @@ const init_worker_code = quote
     const SDS_TEST_PART = Ref{Any}(nothing)
 end
 
-# Long test files run once per element type, "test_api[Float64]" etc., each part in a worker of its own: a
-# worker compiles the solver for every element type it tests, and that compilation, not the tests, sets the wall
-# time. The testsets that do not loop over the element types run in the Float64 part (`RUN_SHARED`).
+# the backends and the element types of the run (`SELECTED_ELTYPES`), read once from the environment in the main
+# process: prints the banner and fails early on an empty backend list or a bad SDS_TEST_ELTYPES
+module BackendBanner end
+Core.eval(BackendBanner, init_code)
+Core.eval(BackendBanner, :(include($(joinpath(TEST_DIR, "backends.jl")))))
+BackendBanner.print_backends()
+
+# Long test files run once per selected element type, "test_api[Float64]" etc., each part in a worker of its own:
+# a worker compiles the solver for every element type it tests, and that compilation, not the tests, sets the wall
+# time. The testsets that do not loop over the element types run in the Float64 part, or in the first selected
+# type's part when SDS_TEST_ELTYPES leaves Float64 out (`RUN_SHARED`).
 const SPLIT_FILES = ["test_api", "test_dense", "test_fgmres", "test_matching", "test_numeric_cholesky_a",
                      "test_numeric_cholesky_b", "test_numeric_cholesky_c", "test_numeric_ldlt", "test_numeric_lu",
                      "test_ported", "test_refinement", "test_schur", "test_solve", "test_ubatch"]
-const SPLIT_ELTYPES = (Float32, Float64, ComplexF32, ComplexF64)
+const SPLIT_ELTYPES = BackendBanner.SELECTED_ELTYPES
+const SHARED_ELTYPE = Float64 in SPLIT_ELTYPES ? Float64 : first(SPLIT_ELTYPES)
 
-# the seed goes with the test, not into `init_code`: ParallelTestRunner seeds with 1 after `init_code`
+# the seed goes with the test, not into `init_code`: ParallelTestRunner seeds with 1 after `init_code`; `backends.jl`
+# comes first, `utils.jl` derives `ELTYPES` from its `SELECTED_ELTYPES`
 test_expr(name, part) = quote
     Main.SDS_TEST_PART[] = $part
+    include($(joinpath(TEST_DIR, "backends.jl")))
     include($(joinpath(TEST_DIR, "utils.jl")))
     include($(joinpath(TEST_DIR, "matrices.jl")))
-    include($(joinpath(TEST_DIR, "backends.jl")))
     Random.seed!(666)
     include($(joinpath(TEST_DIR, "$name.jl")))
 end
@@ -63,7 +74,7 @@ testsuite = Dict{String, Expr}()
 for name in TEST_FILES
     if name in SPLIT_FILES
         for T in SPLIT_ELTYPES
-            testsuite["$name[$T]"] = test_expr(name, ((T,), T == Float64))
+            testsuite["$name[$T]"] = test_expr(name, ((T,), T == SHARED_ELTYPE))
         end
     else
         testsuite[name] = test_expr(name, nothing)
@@ -80,11 +91,5 @@ if filter_tests!(testsuite, args)
     filter!(t -> !names_test(skip, first(t)), testsuite)
 end
 
-# the backend banner once, from the main process (also fails early when no backend is left to test)
-module BackendBanner end
-Core.eval(BackendBanner, init_code)
-Core.eval(BackendBanner, :(include($(joinpath(TEST_DIR, "backends.jl")))))
-BackendBanner.print_backends()
-
 runtests(SparseDirectSolver, args; testsuite, init_code, init_worker_code,
          history_key = get(ENV, "SDS_TEST_CPU", "1") == "0" ? "gpu" : nothing)

test/test_fgmres.jl

@@ -74,7 +74,7 @@ end
 end
 
 @testset "FGMRES-IR: uniform batch ($(backend_name(backend)), $T)" for backend in BACKENDS,
-                                                                       T in filter(in((Float64, ComplexF32)), ELTYPES)
+                                                                       T in eltypes_among((Float64, ComplexF32))
     Random.seed!(666)
     n, nb, nrhs = 120, 3, 2
     members = batch_members(random_symindef(T, n, 0.03), nb)

test/test_helpers.jl

@@ -157,3 +157,18 @@ end
     @test spd_structure(Float32) == "SPD" && spd_structure(ComplexF32) == "HPD"
     @test sym_structure(Float64) == "S" && sym_structure(ComplexF64) == "H"
 end
+
+@testset "element type selection (SDS_TEST_ELTYPES)" begin
+    @test selected_eltypes("") == ALL_ELTYPES == (Float32, Float64, ComplexF32, ComplexF64)
+    @test selected_eltypes("ComplexF32, Float64") == (Float64, ComplexF32)   # the order of ALL_ELTYPES
+    @test selected_eltypes("Float32,Float32,") == (Float32,)
+    @test_throws ErrorException selected_eltypes("Float16")
+    @test_throws ErrorException selected_eltypes("ComplexF32,ComplexF64")   # no real type
+    @test SELECTED_ELTYPES == selected_eltypes()
+    @test ELTYPES == SELECTED_ELTYPES   # an unsplit file tests every selected type
+    @test REAL_ELTYPES == filter(T -> T <: Real, SELECTED_ELTYPES) && !isempty(REAL_ELTYPES)
+    # `@test_logs` keeps the "testset skipped" note of a selection without these types out of the output
+    @test (@test_logs match_mode = :any eltypes_among(Complex)) == COMPLEX_ELTYPES
+    @test (@test_logs match_mode = :any eltypes_among((Float64, ComplexF32))) ==
+          filter(in((Float64, ComplexF32)), SELECTED_ELTYPES)
+end

test/test_numeric_ldlt.jl

@@ -431,7 +431,7 @@ end
 end
 
 @testset "complex symmetric through the API ($(backend_name(backend)), $T)" for backend in BACKENDS,
-                                                                              T in COMPLEX_ELTYPES
+                                                                              T in eltypes_among(Complex)
     Random.seed!(666)
     A = random_symindef(T, 300, 0.02; hermitian = false)
     solver = DirectSolver(api_matrix(backend, tril(A), Int32), "S", 'L')

test/test_reference_ldlt.jl

@@ -155,7 +155,7 @@ end
     @test N.factor == Nf.factor && N.d == Nf.d && N.piv == Nf.piv && N.stats == Nf.stats
 end
 
-@testset "complex symmetric LDLᵀ: $T" for T in COMPLEX_ELTYPES
+@testset "complex symmetric LDLᵀ: $T" for T in eltypes_among(Complex)
     Random.seed!(666)
     A = random_symindef(T, 300, 0.02; hermitian = false)
     @test transpose(A) == A && A' != A

test/utils.jl

@@ -4,11 +4,12 @@
 using LinearAlgebra
 using SparseArrays
 
-# The element types under test. `runtests.jl` runs a long test file once per element type, each part in its own
-# worker: the part sets `Main.SDS_TEST_PART[] = (eltypes, shared)` before these helpers are included (`nothing`:
-# all element types, one part).
+# The element types under test: `SELECTED_ELTYPES` (`backends.jl`, from SDS_TEST_ELTYPES; `backends.jl` is
+# included before these helpers). `runtests.jl` runs a long test file once per selected element type, each part in
+# its own worker: the part sets `Main.SDS_TEST_PART[] = (eltypes, shared)` before the helpers are included
+# (`nothing`: all selected element types, one part).
 const TEST_PART = isdefined(Main, :SDS_TEST_PART) ? Main.SDS_TEST_PART[] : nothing
-const ELTYPES = TEST_PART === nothing ? (Float32, Float64, ComplexF32, ComplexF64) : TEST_PART[1]
+const ELTYPES = TEST_PART === nothing ? SELECTED_ELTYPES : TEST_PART[1]
 const REAL_ELTYPES = filter(T -> T <: Real, ELTYPES)
 const COMPLEX_ELTYPES = filter(T -> T <: Complex, ELTYPES)
 
@@ -17,9 +18,29 @@ const COMPLEX_ELTYPES = filter(T -> T <: Complex, ELTYPES)
 
 `true` when this run of a test file runs the testsets that do not loop over the
 element types (`RUN_SHARED && @testset …`): in the one part of a test file split
-by element type that owns them, and always in an unsplit test file.
+by element type that owns them (the `Float64` part, or the first selected type's
+when `Float64` is not selected), and always in an unsplit test file.
 """
 const RUN_SHARED = TEST_PART === nothing || TEST_PART[2]
+
+"""
+    eltypes_among(types) -> Tuple
+    eltypes_among(S::Type) -> Tuple
+
+The element types of this part that are among `types` (or subtypes of `S`), for a
+testset that runs for some element types only
+(`for T in eltypes_among((Float64, ComplexF32))`, `for T in eltypes_among(Complex)`).
+When `SDS_TEST_ELTYPES` selects none of `types`, the part that runs the shared
+testsets says that the testset is skipped.
+"""
+function eltypes_among(types)
+    if RUN_SHARED && !any(in(types), SELECTED_ELTYPES)
+        @info "SDS_TEST_ELTYPES selects none of $(join(types, ", ")): testset skipped"
+    end
+    return filter(in(types), ELTYPES)
+end
+eltypes_among(S::Type) = eltypes_among(filter(T -> T <: S, ALL_ELTYPES))
+
 const INTTYPES = (Int32, Int64)
 
 """