I use PkgBenchmark.jl to run my benchmarks in CI. I recently decided to decompose one of my benchmarks into a few benchmarks. So, I originally had an entry in a BenchmarkGroup that was a single benchmark, but the new version instead has a nested BenchmarkGroup for that entry. This caused MethodErrors to be thrown in the BenchmarkTools.jl code, both when loading the tuning parameters and when judging.
ERROR: MethodError: no method matching loadparams!(::BenchmarkTools.Benchmark, ::BenchmarkGroup, ::Symbol, ::Symbol)
Closest candidates are:
loadparams!(::BenchmarkTools.Benchmark, !Matched::BenchmarkTools.Parameters, ::Any...)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/execution.jl:25
loadparams!(!Matched::BenchmarkGroup, ::BenchmarkGroup, ::Any...)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:172
loadparams!(!Matched::BenchmarkTools.Parameters, !Matched::BenchmarkTools.Parameters, ::Any...)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/parameters.jl:96
Stacktrace:
[1] loadparams!(::BenchmarkGroup, ::BenchmarkGroup, ::Symbol, ::Vararg{Symbol}) (repeats 3 times)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:174
[2] __runbenchmark_local(file::String, output::String, tunefile::String, retune::Bool, runoptions::@NamedTuple{verbose::Bool})
@ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/runbenchmark.jl:279
[3] (::PkgBenchmark.var"#30#31"{String, String, String, Bool, @NamedTuple{verbose::Bool}})()
@ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/runbenchmark.jl:264
[4] with_logstate(f::Function, logstate::Any)
@ Base.CoreLogging ./logging.jl:515
[5] with_logger
@ ./logging.jl:627 [inlined]
[6] _runbenchmark_local(file::String, output::String, tunefile::String, retune::Bool, runoptions::@NamedTuple{verbose::Bool}, logger_factory_path::Tuple{String, String, Symbol})
@ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/runbenchmark.jl:263
[7] top-level scope
@ none:2ERROR: LoadError: MethodError: no method matching judge(::BenchmarkTools.BenchmarkGroup, ::BenchmarkTools.TrialEstimate)
Closest candidates are:
judge(::BenchmarkTools.BenchmarkGroup, !Matched::BenchmarkTools.BenchmarkGroup...; kwargs...)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:128
judge(!Matched::BenchmarkTools.TrialEstimate, ::BenchmarkTools.TrialEstimate; kwargs...)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/trials.jl:231
Stacktrace:
[1] (::BenchmarkTools.var"#25#26"{@Kwargs{}})(::BenchmarkTools.BenchmarkGroup, ::Vararg{Any})
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129
[2] mapvals!(f::Function, dest::BenchmarkTools.BenchmarkGroup, src::BenchmarkTools.BenchmarkGroup, srcs::BenchmarkTools.BenchmarkGroup)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:88
[3] mapvals
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:96 [inlined]
[4] #judge#24
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129 [inlined]
[5] judge(group::BenchmarkTools.BenchmarkGroup, groups::BenchmarkTools.BenchmarkGroup)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:128
[6] (::BenchmarkTools.var"#25#26"{@Kwargs{}})(::BenchmarkTools.BenchmarkGroup, ::Vararg{BenchmarkTools.BenchmarkGroup})
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129
[7] mapvals!(f::Function, dest::BenchmarkTools.BenchmarkGroup, src::BenchmarkTools.BenchmarkGroup, srcs::BenchmarkTools.BenchmarkGroup)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:88
[8] mapvals
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:96 [inlined]
[9] #judge#24
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129 [inlined]
[10] judge(group::BenchmarkTools.BenchmarkGroup, groups::BenchmarkTools.BenchmarkGroup)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:128
[11] (::BenchmarkTools.var"#25#26"{@Kwargs{}})(::BenchmarkTools.BenchmarkGroup, ::Vararg{BenchmarkTools.BenchmarkGroup})
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129
[12] mapvals!(f::Function, dest::BenchmarkTools.BenchmarkGroup, src::BenchmarkTools.BenchmarkGroup, srcs::BenchmarkTools.BenchmarkGroup)
@ BenchmarkTools /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:88
[13] mapvals
@ /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:96 [inlined]
[14] #judge#24
@ /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:129 [inlined]
[15] judge
@ /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/BenchmarkTools/1i1mY/src/groups.jl:128 [inlined]
[16] judge(target::BenchmarkResults, baseline::BenchmarkResults, f::typeof(minimum); judgekwargs::Dict{Any, Any})
@ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/judge.jl:50
[17] judge
@ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/judge.jl:49 [inlined]
[18] judge(target::BenchmarkResults, baseline::BenchmarkResults)
@ PkgBenchmark /nobackup2/gitlab-runner/zDyhf_w4/0/Genesis/Genesis.jl/.julia/packages/PkgBenchmark/15w1x/src/judge.jl:49
[19] top-level scope
@ /nobackup2/gitlab_runner/zDyhf_w4/0/Genesis/Genesis.jl/benchmark/run_ci.jl:24Could the logic be made more robust to handle a restructuring of benchmarks, similar to what happens if a benchmark is added or removed?
I use PkgBenchmark.jl to run my benchmarks in CI. I recently decided to decompose one of my benchmarks into a few benchmarks. So, I originally had an entry in a
BenchmarkGroupthat was a single benchmark, but the new version instead has a nestedBenchmarkGroupfor that entry. This causedMethodErrors to be thrown in the BenchmarkTools.jl code, both when loading the tuning parameters and when judging.Could the logic be made more robust to handle a restructuring of benchmarks, similar to what happens if a benchmark is added or removed?