[DRAFT] Reorganize unit tests

[xeren]

Changes multiple parts in the testing package of Dartagnan.

• All solver test suites provide both testAssume and testRefinement. Those who had just one enabled before fail org.junit.Assume.assumeTrue to skip the other method. (This can be undone.)
• Remove most usages of the Provider class, only leaving Provider<ShutdownManager>.

Changes to the test suite hierarchy

• AbstractComparisonTest
• AbstractCompilationTest
• AbstractVerificationTaskSolverTest (new)
  • AbstractAsmTest (new, 6 existing subclasses)
  • AbstractCTest
  • AbstractLitmusTest
    • edit: ArrayValidTest, BranchTest (+ existing subclasses)
  • AbstractSpirvTest (new)
    • AbstractSpirvOpenclTest (new, 11 existing subclasses)
    • AbstractSpirvVulkanTest (new, 12 existing subclasses)

Changes to the test utilities

• edit: Now located in package com.dat3m.dartagnan.test
• Program TestHelper.parseProgram(Path)
• Wmm TestHelper.parseWmm(Path)
• Path ResourceHelper.getCatPath(Arch,String)

Ideas for further changes

• AbstractComparisonTest and AbstractCompilationTest could extend AbstractVerificationTaskSolverTest. The source task solver would provide an optional expected value for the target task solver.

[ThomasHaas]

+900/-2800 looks good :)

[ThomasHaas]

I think you can also delete the unused ShutdownRule and MethodSpecificProvider classes.

Btw. in TestHelper.runDartagnanAsApplication there is a missing placeholder {} in the call to the logger:

if (exitCode != 0) {
    String error = new String(process.getErrorStream().readAllBytes());
    logger.warn("Dartagnan finished with exit code {}. Error:", exitCode, error); // Missing "{}" at the end.
}

Also, why is the IndexedSetTest in utils? Shouldn't it be in others? Maybe you can fix this as part of this PR as well.

[xeren]

I would rather do the opposite. As I see it, the *.others package mirrors the source package structure. So I suggest moving its contents one layer up.

IndexedSetTest is in the same package as IndexedSet and could potentially test package-private subroutines (but it currently does not). Same goes for ModifierTraitTest, CatParserTest, ProgramBuilderTest etc.
AnalysisTest, AggregateTypeTest, EventGraphTest etc. are in other packages than the classes they test. Thus they can only access the public interface and cannot test e.g. subroutines in isolation.

[ThomasHaas]

I didn't know that the unit tests use the same packages as the main code...
Anyways, my point is that test.utils (as a directory, not as a package) was meant to contain utility to write tests but should not contain such tests itself. Seeing a single stray unit test in that directory feels very wrong to me.

[xeren]

How about moving the test utilities into a new package com.dat3m.dartagnan.test?

[ThomasHaas]

That's fine with me.

[github-actions]

Performance comparison

Linux x64

Benchmark details

Memory model: vmm

Benchmark	Base branch	PR branch	Improvement (99% CI)	Result
benchmarks/locks/cna.c	12.166 ± 1.354 s	11.476 ± 0.605 s	➖ +5.1% [-48.8%, +58.9%]	UNKNOWN
benchmarks/locks/mutex_musl.c	23.117 ± 4.153 s	25.624 ± 2.364 s	➖ -12.1% [-80.7%, +56.5%]	UNKNOWN
benchmarks/lfds/dglm.c	19.325 ± 2.782 s	18.751 ± 1.412 s	➖ +1.1% [-115.7%, +118.0%]	UNKNOWN
benchmarks/lfds/ms.c	29.494 ± 2.499 s	26.205 ± 5.907 s	➖ +11.7% [-65.9%, +89.3%]	UNKNOWN
benchmarks/lfds/treiber.c	5.894 ± 0.181 s	5.535 ± 0.077 s	➖ +6.1% [-5.7%, +17.8%]	UNKNOWN
benchmarks/challenging/cna.c	27.905 ± 2.401 s	27.380 ± 1.544 s	➖ +1.7% [-16.4%, +19.9%]	UNKNOWN

Memory model: power

Benchmark	Base branch	PR branch	Improvement (99% CI)	Result
benchmarks/locks/linuxrwlock.c	11.714 ± 0.418 s	11.497 ± 0.235 s	➖ +1.7% [-27.9%, +31.4%]	UNKNOWN
benchmarks/locks/mutex_musl.c	15.477 ± 0.545 s	15.156 ± 1.133 s	➖ +2.0% [-43.2%, +47.2%]	UNKNOWN
benchmarks/lfds/ms.c	18.816 ± 0.998 s	20.451 ± 3.575 s	➖ -8.2% [-87.5%, +71.1%]	UNKNOWN
benchmarks/lfds/treiber.c	9.682 ± 0.077 s	9.600 ± 0.268 s	➖ +0.8% [-19.6%, +21.3%]	UNKNOWN

Memory model: aarch64

Benchmark	Base branch	PR branch	Improvement (99% CI)	Result
benchmarks/challenging/cna.c	8.052 ± 1.403 s	8.246 ± 0.054 s	➖ -4.3% [-99.5%, +90.9%]	UNKNOWN

Total

Benchmarks	Base branch	PR branch	Improvement (99% CI)
All reported benchmarks	181.644 ± 5.229 s	179.920 ± 8.595 s	➖ +0.9% [-31.1%, +32.9%]

7 benchmark(s) omitted because both averages were below 5 seconds.

macOS ARM64

Benchmark details

Memory model: vmm

Benchmark	Base branch	PR branch	Improvement (99% CI)	Result
benchmarks/locks/cna.c	17.998 ± 0.515 s	18.915 ± 0.754 s	➖ -5.2% [-46.9%, +36.4%]	UNKNOWN
benchmarks/locks/mutex_musl.c	37.794 ± 5.141 s	34.496 ± 3.454 s	➖ +7.3% [-90.7%, +105.3%]	UNKNOWN
benchmarks/lfds/dglm.c	52.816 ± 3.117 s	54.536 ± 1.724 s	➖ -3.6% [-53.0%, +45.9%]	UNKNOWN
benchmarks/lfds/ms.c	74.667 ± 4.726 s	71.333 ± 7.024 s	➖ +4.6% [-21.8%, +31.0%]	UNKNOWN
benchmarks/lfds/treiber.c	17.785 ± 0.317 s	18.666 ± 0.468 s	➖ -5.0% [-19.4%, +9.5%]	UNKNOWN
benchmarks/lfds/safe_stack.c	12.612 ± 0.654 s	11.793 ± 0.519 s	➖ +6.2% [-46.0%, +58.3%]	UNKNOWN
benchmarks/challenging/cna.c	42.657 ± 6.503 s	53.634 ± 8.833 s	❌ -25.6% [-50.1%, -1.0%]	UNKNOWN

Memory model: aarch64

Benchmark	Base branch	PR branch	Improvement (99% CI)	Result
benchmarks/locks/linuxrwlock.c	20.201 ± 2.966 s	22.638 ± 2.679 s	➖ -12.4% [-32.3%, +7.5%]	UNKNOWN
benchmarks/locks/mutex_musl.c	16.781 ± 1.588 s	16.273 ± 3.749 s	➖ +2.4% [-139.2%, +143.9%]	UNKNOWN
benchmarks/lfds/dglm.c	13.987 ± 1.039 s	10.922 ± 0.975 s	✅ +22.0% [+9.0%, +35.0%]	PASS
benchmarks/lfds/ms.c	14.610 ± 0.417 s	14.479 ± 0.450 s	➖ +0.9% [-19.0%, +20.7%]	UNKNOWN
benchmarks/challenging/cna.c	34.342 ± 1.246 s	36.921 ± 1.722 s	❌ -7.5% [-14.5%, -0.5%]	UNKNOWN
benchmarks/challenging/wsq.c	16.990 ± 1.274 s	14.919 ± 0.842 s	➖ +11.8% [-37.2%, +60.9%]	UNKNOWN

Memory model: power

Benchmark	Base branch	PR branch	Improvement (99% CI)	Result
benchmarks/locks/linuxrwlock.c	58.839 ± 1.928 s	55.894 ± 2.334 s	➖ +5.0% [-20.0%, +29.9%]	UNKNOWN
benchmarks/locks/mutex_musl.c	34.658 ± 4.372 s	41.660 ± 5.191 s	➖ -20.2% [-42.2%, +1.7%]	UNKNOWN
benchmarks/lfds/dglm.c	9.187 ± 0.520 s	10.049 ± 0.422 s	➖ -9.4% [-19.8%, +1.0%]	UNKNOWN
benchmarks/lfds/ms.c	53.752 ± 5.556 s	52.548 ± 5.765 s	➖ +2.2% [-25.2%, +29.7%]	UNKNOWN
benchmarks/lfds/treiber.c	28.070 ± 2.876 s	27.398 ± 0.714 s	➖ +1.6% [-66.8%, +69.9%]	UNKNOWN

Total

Benchmarks	Base branch	PR branch	Improvement (99% CI)
All reported benchmarks	557.744 ± 16.545 s	567.074 ± 29.539 s	➖ -1.6% [-17.1%, +13.8%]