Skip to content

ci(api): the integration suite runs in six shards, and the floors read all six - #1322

Merged
ExtraToast merged 2 commits into
mainfrom
ci/shard-api-tests
Sep 18, 2026
Merged

ExtraToast merged 2 commits into
mainfrom
ci/shard-api-tests

Conversation

@ExtraToast

@ExtraToast ExtraToast commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Closes #1318. Part of #1321.

api-tests was 8m20s and, once #1309 lands, the longest thing left in validate.yml. Measured on run 35332899202:

unit tests 1m02s, 1006 tests
jacoco report 36s
integrationTest 5m16s, 1303 tests over 121 classes

The integration suite ran serially in one JVM — a single Test worker, no maxParallelForks, no JUnit parallelism.

What this does

Splits the job in two. Unit tests need no MariaDB service at all, so they get their own job and stop sitting in front of the integration suite.

Shards the integration suite six ways, on the same stable FQCN hash :tests:system and the frontend e2e suite use. Each shard gets its own MariaDB service — the fixtures collide on unique fields, so maxParallelForks against one shared database was never an option.

Moves the coverage floors to a fan-in. jacocoIntegrationTestCoverageVerification reads a single integrationTest.exec, and a shard's file covers only its slice, so no floor holds against one alone. Shards upload their execution data; api-coverage merges the six. -PjacocoExecDir points the report and both floors at that directory, and lifts the dependsOn that would otherwise re-run the whole suite inside the fan-in job. The floors from #1282 keep their exact numbers.

Moves ktlint and detekt out of api-static. Source-only, nothing waits on them, 46s off the path every test job sits behind.

The cache trap this had to avoid

The shard filter is applied in doFirst, which Gradle does not hash. Left alone, all six shards present identical task inputs, and a cacheable Test task under --build-cache with cache-read-only: false can serve one shard's result to the next as FROM-CACHE — six green shards where one suite ran. So the shard number is declared:

inputs.property("shardTotal", shardTotal)
inputs.property("shardIndex", shardIndex)

:tests:system has the same shape and does not declare them. Not touched here; filed as #1320.

Expected shard times

Partitioning the 121 classes by the hash, against their measured durations:

shard classes test time
1 24 51.1s
2 21 31.9s
3 15 30.1s
4 24 114.2s
5 20 57.6s
6 17 26.3s

Bounded by shard 4, of which RecoveryControllerEmailIT alone is 68.5s for 7 tests. That class is the floor on what sharding can reach here and is filed separately as #1319.

Review notes

  • Validate complete gains api-lint, api-unit-tests, api-integration-tests and api-coverage, and stays the only required check, so no branch-protection edit is needed.
  • Running integrationTest without SHARD_TOTAL / SHARD_INDEX set behaves exactly as before, so a local run is unchanged.
  • -PjacocoExecDir is the only thing that lifts the dependsOn. A developer running jacocoIntegrationTestCoverageVerification locally still gets the suite run for them.
  • This adds six concurrent jobs. api-tests started 3m18s after its dependencies finished on the run measured above, so the queue is worth watching — the per-shard times and the queue delay on this PR's own run are the measurement, and the shard count comes down if the parallelism is not being served.

Diff breakdown added removed, scaled to the largest row.

ci                                                +214    -38    2
  build & config     ██████████████████████░░░░   +214    -38    2

──────────────────────────────────────────────────────────────────
total (hand-written)                              +214    -38  2 files

…d all six

api-tests was 8m20s, of which the integration suite was 5m16s run serially in
one JVM over 121 classes. Split it: unit tests get their own job with no
database at all, and the integration suite shards six ways on the same stable
FQCN hash :tests:system and the frontend e2e suite already use.

Each shard gets its own MariaDB service. The fixtures collide on unique fields,
so the suite cannot share one database across parallel writers.

The coverage floors read a single integrationTest.exec, and a shard's file
covers only its own slice, so no floor holds against one alone. Shards upload
their execution data and api-coverage merges the six: -PjacocoExecDir points the
report and both floors at that directory and lifts the dependency that would
otherwise re-run the suite. The floors stay as strict as they were.

The shard filter is applied in doFirst, which Gradle does not hash, so the shard
number is declared as a task input. Without it every shard has identical inputs
and one shard's result is served to the next FROM-CACHE.

ktlint and detekt move out of api-static into their own job. They are
source-only and nothing waits on them, so 46s leaves the critical path that
every test job sits behind.

Refs #1318
…y one refuses

`fileTree(dir)` inside :services:api resolves against services/api, so
-PjacocoExecDir=services/api/build/jacoco/shards looked under
services/api/services/api/ and found nothing. Resolve against the root instead.

Finding nothing was not loud. JacocoReport with empty execution data goes
NO-SOURCE and JacocoCoverageVerification passes with nothing to measure, so a
wrong path reads as a green coverage gate — only the report upload failing on
if-no-files-found made it visible. The merged mode now refuses an empty set
rather than measuring it.

Verified against the six .exec files the first run produced: merged instruction
coverage 0.6851 against a floor of 0.40, and the report is written.
@ExtraToast

Copy link
Copy Markdown
Contributor Author

First run: every shard green, the fan-in caught a real defect in my own wiring.

job time
api-static 4m24s (was 5m40s, lint lifted out)
api-lint 1m26s
api-unit-tests 3m57s
integration shard 3 2m59s
integration shard 5 3m15s
integration shard 6 3m16s
integration shard 1 3m49s
integration shard 2 4m55s
integration shard 4 5m46s

The unsharded api-tests job ran at the same moment on #1309 and took 10m38s, so the comparison is 10m38s against a 5m46s bound.

The defect

api-coverage failed, but not where it looked. The Gradle step passed in 26s and the artifact upload failed on if-no-files-found: error.

fileTree(dir) inside :services:api resolves relative to services/api, so -PjacocoExecDir=services/api/build/jacoco/shards looked under services/api/services/api/build/jacoco/shards and found nothing. That part is an ordinary path bug. What makes it worth writing down is how it failed: JacocoReport with empty execution data goes NO-SOURCE, and JacocoCoverageVerification with nothing to read passes. A mistyped path reads as a green coverage gate. The upload step is the only reason it was visible at all.

Fixed in 061c20b by resolving against the root, and by refusing an empty set rather than measuring it:

No .exec files under …/build/jacoco/nonexistent — the merged coverage would measure nothing.

Verified locally against the six .exec artifacts this run produced: merged instruction coverage 0.6851 against the 0.40 floor, class 0.7972, report written. So the fan-in was correct; it was reading the wrong directory.

Shard count

Per-shard overhead is 2–4 minutes and varies more than the test content does — shard 2 took 4m55s to run 31.9s of tests. Six shards may be more than this suite is worth once #1319 removes the 68.5s class that shard 4 is pinned by. Worth revisiting with a second data point rather than now.

@ExtraToast
ExtraToast merged commit dd1755f into main Sep 18, 2026
35 checks passed
@ExtraToast
ExtraToast deleted the ci/shard-api-tests branch September 18, 2026 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

the API integration suite runs in six shards, and the floors read all six

1 participant