Run native tests without running tests on JVM first

[dnestoro]

Problem: At the moment, every time we run native tests (with both Maven or Gradle), we must run tests on JVM first, in order to collect test-ids required for native run. With more complex tests, this additional step increases time required for tests execution.

Idea: We should use JUnit's dry-run mode to just collect test-ids without executing tests on JVM.

Solution: we can set junit.platform.execution.dryRun.enabled system property to true when executing tests on JVM (maven change, gradle change).

NOTE: Since this approach changes the default behavior, we should bump major version for the next release.

[dnestoro]

Changes in this file are not related to the original purpose of this PR. This is just a small clean-up triggered by some changes in Utils.java class.

[dnestoro]

Currently, the tests are failing because there are different number of discovered tests with and without junit.platform.execution.dryRun.enabled while we expect the same number of tests in both cases.

This only happens if the tests are annotated with annotations like @ParameterizedTest or @RepeatedTest.
Test ids discovered without dry-run:

[engine:junit-vintage]/[runner:tests.VintageTests]/[test:testEvery(tests.VintageTests)]
[engine:junit-vintage]/[runner:tests.VintageTests]/[test:testExpectedExceptionCause(tests.VintageTests)]
[engine:junit-vintage]/[runner:tests.VintageTests]/[test:testExpectedException(tests.VintageTests)]
[engine:junit-jupiter]/[class:tests.ComplexTest]/[method:accessMethodReflectively()]
[engine:junit-jupiter]/[class:tests.ComplexTest]/[method:resourceTest()]
[engine:junit-jupiter]/[class:tests.ComplexTest]/[method:accessFiledReflectively()]
[engine:junit-jupiter]/[class:tests.ComplexTest]/[method:callMethodFromOtherClass()]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:repeatedTest(org.junit.jupiter.api.RepetitionInfo)]/[test-template-invocation:#1]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:repeatedTest(org.junit.jupiter.api.RepetitionInfo)]/[test-template-invocation:#2]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:repeatedTest(org.junit.jupiter.api.RepetitionInfo)]/[test-template-invocation:#3]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:testWithExternalFieldSource(java.lang.String)]/[test-template-invocation:#1]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:testWithExternalFieldSource(java.lang.String)]/[test-template-invocation:#2]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:testWithExternalFieldSource(java.lang.String)]/[test-template-invocation:#3]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:test(java.lang.String)]/[test-template-invocation:#1]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:test(java.lang.String)]/[test-template-invocation:#2]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[method:skipThisTest()]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:singleFieldSource(java.lang.String)]/[test-template-invocation:#1]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:singleFieldSource(java.lang.String)]/[test-template-invocation:#2]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[method:beforeAndAfterEachTest1()]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[method:beforeAndAfterEachTest2()]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[method:beforeAndAfterEachTest3()]
[engine:junit-jupiter]/[class:tests.OrderTests]/[method:firstTest()]
[engine:junit-jupiter]/[class:tests.OrderTests]/[method:secondTest()]
[engine:junit-jupiter]/[class:tests.OrderTests]/[method:thirdTest()]

Test ids discovered with dry-run:

[engine:junit-vintage]/[runner:tests.VintageTests]/[test:testEvery(tests.VintageTests)]
[engine:junit-vintage]/[runner:tests.VintageTests]/[test:testExpectedExceptionCause(tests.VintageTests)]
[engine:junit-vintage]/[runner:tests.VintageTests]/[test:testExpectedException(tests.VintageTests)]
[engine:junit-jupiter]/[class:tests.ComplexTest]/[method:accessMethodReflectively()]
[engine:junit-jupiter]/[class:tests.ComplexTest]/[method:resourceTest()]
[engine:junit-jupiter]/[class:tests.ComplexTest]/[method:accessFiledReflectively()]
[engine:junit-jupiter]/[class:tests.ComplexTest]/[method:callMethodFromOtherClass()]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[method:skipThisTest()]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[method:beforeAndAfterEachTest1()]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[method:beforeAndAfterEachTest2()]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[method:beforeAndAfterEachTest3()]
[engine:junit-jupiter]/[class:tests.OrderTests]/[method:firstTest()]
[engine:junit-jupiter]/[class:tests.OrderTests]/[method:secondTest()]
[engine:junit-jupiter]/[class:tests.OrderTests]/[method:thirdTest()] 

[sbrannen]

This only happens if the tests are annotated with annotations like @ParameterizedTest or @RepeatedTest.

See junit-team/junit-framework#3164 (comment).

[dnestoro]

See junit-team/junit-framework#3164 (comment).

@sbrannen can we somehow extend dryRun mode to collect ids for all tests? If we are going to break some existing support, can we add a new mode (like dryRunAll) to collect all test ids?

I see that dryRun mode is completely separated, but can we follow standard execution pipeline and do something like:

// do all test preparations as in the normal execution
if (!dryRunAll()) {
    executeTest()
}
// post processing where we register test-id

If that idea is not possible to achieve for some reason, do you have some other proposal?

cc @marcphilipp @vjovanov @sdeleuze @melix

[vjovanov]

@jormundur00 will take this ticket over.

[oracle-contributor-agreement]

Thank you for your pull request and welcome to our community! To contribute, please sign the Oracle Contributor Agreement (OCA).
The following contributors of this PR have not signed the OCA:

• PR author: dnestoro

To sign the OCA, please create an Oracle account and sign the OCA in Oracle's Contributor Agreement Application.

When signing the OCA, please provide your GitHub username. After signing the OCA and getting an OCA approval from Oracle, this PR will be automatically updated.

If you are an Oracle employee, please make sure that you are a member of the main Oracle GitHub organization, and your membership in this organization is public.