Skip to content

Fix the site test report and count Android in the coverage #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
garydgregory merged 3 commits into from
Sep 2, 2026
+169 −18
Merged

Fix the site test report and count Android in the coverage #82

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fbd1211
Report on the site the tests the build already ran
ppkarwasz Sep 1, 2026
3bfdf6c
Count the Android instrumented run in the coverage report
ppkarwasz Sep 1, 2026
502501b
Set coverage minimums a single JDK run can meet
ppkarwasz Sep 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 73 additions & 11 deletions android-tests/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,35 +26,97 @@ module; it is a standalone Gradle build kept separate so the default `mvn` goal
- JDK 17 on `PATH` (AGP 8.x requires it).
- Android SDK with `platforms/android-34` and `build-tools/34.0.0` installed;
export `ANDROID_HOME` (or `ANDROID_SDK_ROOT`) to point at it.
- Either an attached emulator/device (`adb devices` shows it) or the AGP managed device
bundled into this build (`api31`, AOSP system image).
- The library JAR built by the parent Maven build:

```shell
export ANDROID_HOME=/path/to/android
```
cd .. && mvn -DskipTests package
- Either an attached emulator/device (`adb devices` shows it) or the AGP managed device bundled into this build (`api33`, AOSP system image).
- The library JAR built by the parent Maven build:

```shell
mvn -f.. -DskipTests package
```

## Running

Against an attached emulator/device:

```
```shell
./gradlew connectedAndroidTest
```

Against the bundled AGP managed device (downloads the AOSP API 31 system image on first
run, then provisions and tears down a headless emulator for each invocation):
Against the bundled AGP managed device
(downloads the AOSP API 33 system image on first run,
then provisions and tears down a headless emulator for each invocation):

```shell
./gradlew api33DebugAndroidTest
```
./gradlew api31DebugAndroidTest

## Coverage

The run is instrumented with JaCoCo,
pinned to the version the Maven build reports with,
and leaves one execution data file per device under `build/outputs/code_coverage`.
The full round trip:

1. Build the JAR the instrumented tests run against:

```shell
mvn -f.. -DskipTests package
```

2. Run the tests on a device, either an attached one or the bundled managed device:

```shell
./gradlew connectedDebugAndroidTest
./gradlew api33DebugAndroidTest
```

3. Fold the device data into the project's coverage:

```shell
mvn -f.. -Pjacoco verify site
```

The JVM suite runs under the JaCoCo agent in this step,
so both halves end up in `target/jacoco.exec`:
the `jacoco` profile merges whatever this module produced into it in the site lifecycle,
just before the report is written.
The coverage check runs earlier, on the JVM data alone,
so its minimums mean the same thing whether a device run is lying around;
the device data widens the report, not the bar.

Repeat step 2 whenever the library changes.
The execution data identifies each class by its bytecode,
and JaCoCo drops data that no longer matches the compiled class,
so a device run left over from an older JAR quietly lowers the numbers instead of inflating them.

### Maximum coverage

This library is written to work from JDK 8 through 25 and on Android.
Since new JAXP methods were introduced in JDK 9, 13, and 18,
maximum coverage is obtained by the following recipe:

```shell
export JDK8=/path/to/jdk8
export JDK21=/path/to/jdk21
export ANDROID_HOME=/path/to/android
JAVA_HOME=$JDK21 mvn -f.. -Pjacoco clean package
./gradlew api33DebugAndroidTest
JAVA_HOME=$JDK8 mvn -f.. -Pjacoco test
JAVA_HOME=$JDK21 mvn -f.. -Pjacoco verify site
```

## Excluded test groups

The build excludes JUnit 5 tags for JAXP types Android does not ship:
The build runs the `dom`, `sax`, `schema`, and `trax` tags and excludes the rest:

- `stax`: there is no `XMLInputFactory` on Android.
- `schema`: there is no `SchemaFactory` on Android.
- `xpath`: Android ships an XPath implementation,
but it is currently untested.
- `xpath3`: relies on Saxon, which is not on the Android classpath.

DOM, SAX, TrAX and XPath 1.0 paths are exercised in full.
DOM, SAX, TrAX, and schema paths are exercised in full;
the schema tests run against the Apache Xerces the `androidTest` classpath brings in,
since Android ships `javax.xml.validation` without a `SchemaFactory` implementation.
11 changes: 11 additions & 0 deletions android-tests/build.gradle.kts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,17 @@ android {
targetCompatibility = JavaVersion.VERSION_1_8
}

// Match the JaCoCo the Maven build reports with, so the on-device agent understands the same execution data format.
testCoverage {
jacocoVersion = "0.8.15"
}

buildTypes {
getByName("debug") {
enableAndroidTestCoverage = true
}
}

sourceSets {
getByName("androidTest") {
java.srcDirs("../src/test/java")
Expand Down
92 changes: 85 additions & 7 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,14 +75,19 @@ limitations under the License.
<commons.xerces.version>2.12.2</commons.xerces.version>
<!-- Test-only: computes each secure class' shade closure, mirroring maven-shade minimizeJar, for ShadingFootprintTest. -->
<commons.jdependency.version>2.16</commons.jdependency.version>
<!-- jacoco-maven-plugin: Should only get better -->
<!--
jacoco-maven-plugin: Should only get better.

A single `mvn -Pjacoco clean verify` run covers less than the project as a whole does.
These minima need to pass on all JDK versions, in particular JDK 8 and JDK 21.
-->
<commons.jacoco.haltOnFailure>true</commons.jacoco.haltOnFailure>
<commons.jacoco.classRatio>0.97</commons.jacoco.classRatio>
<commons.jacoco.instructionRatio>0.97</commons.jacoco.instructionRatio>
<commons.jacoco.methodRatio>0.99</commons.jacoco.methodRatio>
<commons.jacoco.branchRatio>0.91</commons.jacoco.branchRatio>
<commons.jacoco.lineRatio>0.96</commons.jacoco.lineRatio>
<commons.jacoco.complexityRatio>0.96</commons.jacoco.complexityRatio>
<commons.jacoco.classRatio>0.96</commons.jacoco.classRatio>
<commons.jacoco.instructionRatio>0.93</commons.jacoco.instructionRatio>
<commons.jacoco.methodRatio>0.95</commons.jacoco.methodRatio>
<commons.jacoco.branchRatio>0.87</commons.jacoco.branchRatio>
<commons.jacoco.lineRatio>0.93</commons.jacoco.lineRatio>
<commons.jacoco.complexityRatio>0.92</commons.jacoco.complexityRatio>
Comment on lines +85 to +90

@ppkarwasz ppkarwasz Sep 1, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is it that running more tests lowers the test coverage percentages?

Sorry, now I understand the question.

The values were lowered, so that a single build on any JDK passes them. This way if you run:

mvn -Pjacoco clean verify

the build passes and you don't need to run multiple -Pjacoco test on multiple JDKs, before you can run verify.

</properties>
<dependencies>
<!--
Expand Down Expand Up @@ -453,9 +458,82 @@ limitations under the License.
</rulesets>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-report-plugin</artifactId>
<reportSets>
<!-- The `report` and `failsafe-report` goals fork the test phase, running the suite a second time; the `*-only` goals read what the build already produced. -->
<reportSet>
<id>surefire</id>
<reports>
<report>report-only</report>
</reports>
<configuration>
<!-- The suite runs once per JAXP combination, each execution into its own directory. -->
<reportsDirectories>
<reportsDirectory>${project.build.directory}/surefire-reports/jdk-xerces</reportsDirectory>
<reportsDirectory>${project.build.directory}/surefire-reports/saxon</reportsDirectory>
<reportsDirectory>${project.build.directory}/surefire-reports/saxon-xerces</reportsDirectory>
<reportsDirectory>${project.build.directory}/surefire-reports/stockjdk</reportsDirectory>
<reportsDirectory>${project.build.directory}/surefire-reports/woodstox</reportsDirectory>
<reportsDirectory>${project.build.directory}/surefire-reports/xalan</reportsDirectory>
<reportsDirectory>${project.build.directory}/surefire-reports/xalan-xerces</reportsDirectory>
<reportsDirectory>${project.build.directory}/surefire-reports/xerces</reportsDirectory>
</reportsDirectories>
</configuration>
</reportSet>
<reportSet>
<id>failsafe</id>
<!-- The integration tests keep the plugin's own default, ${project.build.directory}/failsafe-reports. -->
<reports>
<report>failsafe-report-only</report>
</reports>
</reportSet>
</reportSets>
</plugin>
</plugins>
</reporting>
<profiles>
<!--
Fold the Android instrumented run into the coverage data.
-->
<profile>
<id>jacoco</id>
<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<executions>
<execution>
<id>merge-android</id>
<phase>pre-site</phase>
<goals>
<goal>merge</goal>
</goals>
<configuration>
<fileSets>
<fileSet>
<directory>${project.build.directory}</directory>
<includes>
<include>jacoco.exec</include>
</includes>
</fileSet>
<fileSet>
<directory>${project.basedir}/android-tests/build/outputs/code_coverage</directory>
<includes>
<include>**/*.ec</include>
</includes>
</fileSet>
</fileSets>
<destFile>${project.build.directory}/jacoco.exec</destFile>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<!--
Regression guard for the generated OSGi and JPMS descriptors.
The JPMS test requires JDK 9+
Expand Down
Loading