Introduce a Gradle S3 build cache

[bric3]

What Does This Do

Enables S3 build cache for Gradle task outputs in GitLab CI. Eligible jobs can reuse matching outputs, while release pipelines use isolated, pipeline-specific namespaces. Dependency caches, plugin caches and job artifacts remain in GitLab. This should significantly improve Gradle job duration.

This effectively retire the old Gitlab based build cache.

Motivation

The existing pipeline-scoped GitLab cache limits reuse and transfers whole cache archives. In the warm build cache pipeline run, the S3 build cache reduced the parent pipeline duration from 42m23s to 26m22s and the median duration of 197 matching Gradle jobs by 36.9%.

Additional Notes

Scope and release isolation

• Enabled only in GitLab CI, with S3 build cache reads and writes using the runner's default AWS credentials.
• Bucket and region supplied by .gitlab-ci.yml: S3_BUILD_CACHE_BUCKET, S3_BUILD_CACHE_REGION.
• Local development and other CI providers unchanged.
• GitLab build-output cache removed (&build_cache, BUILD_CACHE_POLICY). Dependency/plugin caches and job artifacts retained.

Release tags use a pipeline-specific namespace. Jobs and retries share entries within that pipeline; a new pipeline for the same tag starts cold. No fallback to the non-release cache. CI_PIPELINE_ID provides the isolation; the tag name is not needed. Release branches without a tag follow the non-tag policy.

Release configuration is implemented locally; tagged-pipeline validation is pending. Performance measurements below cover non-tag pipelines only.

Cache namespace

Current object paths:

trial/v1/<CI_COMMIT_REF_SLUG>/<architecture>/<Gradle cache key>
release/v1/<CI_PIPELINE_ID>/<architecture>/<Gradle cache key>

Note

trial/ with CI_COMMIT_REF_SLUG is tailored for this experiment, its role is to facilitate rerun of pipeline. The final S3 build cache key path for non-release pipeline needs some decision; in particular depending on the choice that will be made a shared prefix would enable reuse across branches and nightly builds.

What needs to appear in the cache key:

• The architecture, the tests results in particular need to distinguish the AMD64/ARM64 test runs.
• The OS is currently omitted as CI runners are Linux-only. However, it might be useful to add when enabling Windows test runners.
• The actual Gradle key is privy to how Gradle fingerprint task inputs and what dd-trace-java's build is declaring inputs, see Cache keys.

Warning

A prefix change means the pipeline starts on a cold cache.

Configuring the cache key prefix

There's a S3_BUILD_CACHE_PREFIX in .gitlab-ci.yml that sets a "namespace" for all pipelines but release pipelines. To benefit the most from the S3 build cache it should be shared across pipelines, putting release pipeline aside for now. This could look like shared/v1, nightly job could help feed the cache, but essentially all pipeline participate in this cache ; note this isn't an issue given all cache entry are fingerprinted by entries and as such do not mess with other entries.

Fore release in particular, S3_BUILD_CACHE_PREFIX could take this value release/v1/<CI_PIPELINE_ID>, to isolate release pipelines.

Missing or blank prefixes should fail the pipeline.

Nightly cache warming

I'm a bit undecided if we need to use the nightly job to fill in this cache if all pipeline participate in maintaining this cache.

Currently, the POPULATE_CACHE populates dependency/plugin caches. And as such build and build_tests can also populate S3 build cache but this won't fill in the blanks for the test results, which is a significant part of this experiment, and where we can achieve significant time saving.

Results

Snapshot captured on 2026-09-17 for two runs on 2026-09-16:

• Gitlab based cached pipeline 139108386
• initial S3 build cache pipeline 137893496
• warm S3 build cache pipeline 137919730

Both S3 build cache runs used the same commit 17b8990cdb and cache namespace, with S3 build cache plugin 1.9.9 and Gradle 9.7.1.

S3 build cache statistics were collected from 198 Gradle jobs in the parent pipeline's tests and tests-arm64 stages; build jobs and the downstream system tests pipeline were excluded. Duration comparisons use the 197 jobs that completed successfully in both S3 build cache runs (one muzzle job fail due to maven central 429).

Measure	Scope / calculation	Initial S3 build cache run	Warm S3 build cache rerun
GitLab reported pipeline duration	dd-trace-java parent pipeline, including non-Gradle jobs	42m23s	26m22s
Median per-job duration change	197 matched successful Gradle jobs; median of each job's percentage change	Reference	-36.9%
System-test elapsed duration	Parent system_tests trigger job, including its downstream wait	15m19s	15m03s
S3 build cache lookup hit rate	Total hits / total lookups across the 198-job cache sample	84.6%	96.7%
Gradle tasks FROM-CACHE	Total restored tasks / total actionable tasks across the same 198 jobs	44.4%	52.9%
S3 build cache write operations	Total writes across the same 198 jobs; not unique cache keys	18,024	4,000

Note

GitLab's pipeline duration merges job-running intervals and excludes queue time and retries. Job durations are GitLab job runtimes, not just Gradle invocations. This comparison does not measure the full elapsed time for all CI checks, and it excludes the separate GitHub Run system tests workflow. Duration calculation.

Measured improvement: shorter Gradle job runtimes and a 37.8% reduction in the pipeline duration. This change doesn't affect system-tests.

Comparison limits:

• The initial run failed in muzzle: [1/8] on OpenSearch dependency resolution (429 status); the warm run passed. That job is excluded from the 197-job duration sample.

• Against the parent commit's GitLab-cache pipeline 137888666, the initial S3 build cache run's median per-job duration change was +0.3% across the 197 same Gradle jobs:

  => no overall per-job improvement, but no degradation either

  Please note that the reported 49m versus 42m23s pipeline durations are not directly comparable because master and branch job selection differs.

• Both S3 build cache runs used on purpose the same commit and cache key namespace; it is assumed that ci runner load were not controlled. It is also assumed tha the ci runner are behaving equivalently, but a repeatable speedup over GitLab caching needs comparable runs across more commits.

Where the time goes

Instrumentation tests dominate today's pipeline run and the initial S3 build cache run (cold cache). In the warm rerun, system tests finished last, with benchmark and configuration validation close behind. Note that system tests run in parallel with Gradle tests.

In the charts below the job matrices and phases are collapsed into categories. Each bar spans the category's first start to last finish, including gaps. Obviously, parallel rows are not additive. Minor jobs are omitted.

Time zero is the first recorded parent job, excluding the initial pipeline queue. These are elapsed timelines, not GitLab's duration metric: the GitLab-cache chart ends at 45m03s, versus 45m01s reported by GitLab, and the warm chart ends at 26m26s, versus 26m22s reported by GitLab. The S3 build cache charts cover the original runs (excluding retries, and muzzle failure).

Today's GitLab-cache based PR pipeline

=> pipeline 139108386

gantt
    title GitLab-cache PR run — pipeline 139108386
    dateFormat YYYY-MM-DD HH:mm:ss
    axisFormat %M:%S
    tickInterval 5minute
    todayMarker off

    section Parent pipeline
    Build and test compilation : 2000-01-01 00:00:00, 2000-01-01 00:09:47
    Instrumentation tests [48] : 2000-01-01 00:06:08, 2000-01-01 00:43:29
    Latest-dependency tests [30] : 2000-01-01 00:07:06, 2000-01-01 00:32:32
    Smoke tests [48] : 2000-01-01 00:09:49, 2000-01-01 00:31:46
    Other Gradle tests and checks : 2000-01-01 00:00:02, 2000-01-01 00:22:23
    Exploration tests [7] : 2000-01-01 00:06:01, 2000-01-01 00:20:21
    Packaging and system-test setup : 2000-01-01 00:05:21, 2000-01-01 00:12:29
    Startup benchmarks [2] : 2000-01-01 00:06:17, 2000-01-01 00:29:13
    Benchmark reporting and configuration validation : 2000-01-01 00:29:15, 2000-01-01 00:36:18
    Test-result aggregation and pipeline finalization : 2000-01-01 00:43:30, 2000-01-01 00:45:03

    section Downstream pipelines
    System tests : 2000-01-01 00:12:30, 2000-01-01 00:25:20
    Test-matrix generation : 2000-01-01 00:12:43, 2000-01-01 00:15:08
    SSI system tests : 2000-01-01 00:15:09, 2000-01-01 00:25:19

    section Comparison scale
    50-minute boundary :milestone, 2000-01-01 00:50:00, 0s

Loading

Initial S3 build cache pipeline (cold)

=> pipeline 137893496

gantt
    title Initial S3 build cache run — pipeline 137893496
    dateFormat YYYY-MM-DD HH:mm:ss
    axisFormat %M:%S
    tickInterval 5minute
    todayMarker off

    section Parent pipeline
    Build and test compilation : 2000-01-01 00:00:00, 2000-01-01 00:10:30
    Instrumentation tests [48] : 2000-01-01 00:06:40, 2000-01-01 00:42:23
    Latest-dependency tests [30] : 2000-01-01 00:05:40, 2000-01-01 00:33:08
    Smoke tests [48] : 2000-01-01 00:08:56, 2000-01-01 00:29:09
    Other Gradle tests and checks : 2000-01-01 00:00:05, 2000-01-01 00:19:47
    Packaging and system-test setup : 2000-01-01 00:05:29, 2000-01-01 00:14:51
    Startup benchmarks [2] : 2000-01-01 00:07:14, 2000-01-01 00:22:28
    Benchmark reporting and configuration validation : 2000-01-01 00:22:29, 2000-01-01 00:30:31

    section Downstream pipelines
    System tests : 2000-01-01 00:14:53, 2000-01-01 00:30:12
    Test-matrix generation : 2000-01-01 00:14:57, 2000-01-01 00:16:22
    SSI system tests : 2000-01-01 00:16:27, 2000-01-01 00:30:11

    section Comparison scale
    50-minute boundary :milestone, 2000-01-01 00:50:00, 0s

Loading

Warm S3 build cache pipeline (hot, pipeline rerun)

=> pipeline 137919730

gantt
    title Warm S3 build cache rerun — pipeline 137919730
    dateFormat YYYY-MM-DD HH:mm:ss
    axisFormat %M:%S
    tickInterval 5minute
    todayMarker off

    section Parent pipeline
    Build and test compilation : 2000-01-01 00:00:00, 2000-01-01 00:06:32
    Instrumentation tests [48] : 2000-01-01 00:04:39, 2000-01-01 00:16:19
    Latest-dependency tests [30] : 2000-01-01 00:04:47, 2000-01-01 00:16:44
    Smoke tests [48] : 2000-01-01 00:06:36, 2000-01-01 00:21:23
    Other Gradle tests and checks : 2000-01-01 00:00:06, 2000-01-01 00:15:31
    Packaging and system-test setup : 2000-01-01 00:04:14, 2000-01-01 00:10:00
    Startup benchmarks [2] : 2000-01-01 00:05:13, 2000-01-01 00:21:15
    Benchmark reporting and configuration validation : 2000-01-01 00:21:17, 2000-01-01 00:24:57
    Test-result aggregation and pipeline finalization : 2000-01-01 00:25:05, 2000-01-01 00:26:26

    section Downstream pipelines
    System tests : 2000-01-01 00:10:01, 2000-01-01 00:25:04
    Test-matrix generation : 2000-01-01 00:10:02, 2000-01-01 00:12:33
    SSI system tests : 2000-01-01 00:12:34, 2000-01-01 00:25:04

    section Comparison scale
    50-minute boundary :milestone, 2000-01-01 00:50:00, 0s

Loading

• GitLab-cache PR run: system tests finished at 25m20s, while instrumentation tests finished at 43m29s.
• Initial S3 build cache run (Cold cache): the last job instrumentation tests test_inst: [8, 5/8] job, ran for 35m38s. System tests had finished 12m11s earlier.
• Rerun (warm cache): the last Gradle job, smoke tests test_smoke: [8, 8/8], ran for 14m44s and finished at 21m23s. System tests finished at 25m04s.

Further instrumentation/smoke-test reductions would save runner time, but would not directly shorten this warm build cache run.

This improvement expose at least system-test and benchmarks as the next limit (benchmarks in particular within the merge queue).

Remaining questions

Why did the rerun (i.e. on a warm build cache) still write entries?

• cold build cache: 18,024 writes

• warm build cache: 4,000 write operations (22.2% of writes)

• warm build cache: 3,909 misses

• warm build cache: 117,710 lookups

• 3.3% is the actual lookup miss rate.

• The write counter records operations rather than distinct cache keys, so it does not show how many unique entries were written.

• Spotless: 1,074 writes.

• Muzzle's eight shards: 2,433 writes, including 983 in muzzle: [1/8].

• All other measured jobs: 493 writes.

Spotless and Muzzle account for 87.7% of writes. The initial muzzle: [1/8] failure produced no entries so the next pipeline fills in the missing ones. Other misses remain unexplained: the retained logs lack task-key/input hashes.

Next check: compare Spotless/Muzzle keys with --info -Dorg.gradle.caching.debug=true, including resolved dependencies, formatter state, toolchains and path/ref-dependent inputs. Same commit does not guarantee identical inputs. Cache-miss diagnostics.

Correctness checks before rollout

• Verify cache invalidation for dd-trace-java specifics: changes to source files, dependencies, test options, JDK or builder tags, architecture, environment, native tools, services and nested builds. Disable caching when an input cannot be declared. Integration-test caching.
  • Make dd-trace-java.muzzle reports additional JVM data in test task inputs #12631
  • Make dd-trace-java.test-jvm-constraints reports additional JVM data in test task inputs #12630
  • Make dd-trace-java.test-jvm-constraints report architecture in test task inputs #12635
• Ensure build and test jobs resolve the same locked dependencies. The S3 build cache does not refresh dependencies, and --refresh-dependencies does not update locks. Dependency caching, dependency locking.
  • That's expected, refresh locks update locks locally in a job, if that changes then the entry is out of date.
  • Same for test, and latest dependency tests, it's intentional to have moving dependencies, if there's a new version, then no previous build cache entry will exist.
• CI builder image, should they be accounted, in particular if JVM are fingerprinted in testJvm and muzzle ?
• In test-containers track the actual version or hash of icr.io/appcafe/websphere-traditional:latest, or run websphere-jmx tests uncached after a fresh pull. alwaysPull() does not run when Gradle restores test results. Image pull policy.
  • Track Testcontainers images as Gradle test inputs #12633
• Compare with --no-build-cache --rerun-tasks; check artifacts, test results and executed-test counts. Keep validate_build independent of the tested cache.
• Allow tweaking behavior on demand.
• Do we need to check cached outputs for secrets?

Test JVM

[!CAUTION]
Possibly need to track the JVM fingerprint, not only the JVM major version.

Container images

Tests having a moving version tag are a problem as they are not accounted in task's fingerprints. The following is a proposed solution to account for them.

Proposed build-logic DSL:

dependencies {
  testContainerImage(image("cassandra:4", "test.cassandra.image"))
}

The factory method image declares the reference and explicit system-property name; testContainerImage associates it with the consuming test suite. The plugin would:

1. Apply Testcontainers image substitution, including TESTCONTAINERS_HUB_IMAGE_NAME_PREFIX and JDBC's SQL Server mapping.
2. Resolve the effective registry's digest before test fingerprinting; revalidate moving tags each build, including with configuration-cache reuse.
3. Set test.cassandra.image to the full registry/repository@sha256:... reference on the consuming test tasks.

Tests then use System.getProperty("test.cassandra.image"), retaining any Testcontainers compatibility mapping. Both property name and value are Gradle task inputs. A changed digest changes the key. Resolution in doFirst would be too late.

This syntax can be supplied through DependencyHandler extensions; the plugin must provide the DSL and OCI resolver. No native Gradle OCI resolution is implied.

Moving test references found: WebSphere latest, Google Cloud CLI emulators, Cassandra 3/4, MySQL 8.0, PostgreSQL 16-alpine, implicit SQL Server latest, Oracle 23.5-slim-faststart, and Testcontainers' Alpine 3.17 helper. Apply digest tracking to release tags too: they can be republished. Resolve the effective CI mirror, whose contents can differ from upstream.

CI builders

Builder ci-* tags can move. Track the actual builder digest as an input to affected cacheable Gradle tasks, and retain architecture separation.

Test counts

The initial and warm aggregates match for all 172 counted jobs: 400,054 reported tests each. These include restored results by the build cache.

[!NOTE]
.gitlab/collect-result/* still runs after Gradle and processes restored XML normally. Existing counter limitations remain: retried failures are retained, and text matching can mistake log contents for XML errors.

Muzzle

The planner discovers dependency versions; the plugin registers per-version checks with stable names derived from pass/fail, dependency coordinates and the optional directive name. Discovery and planning are not cached by the S3 build cache. Artifact downloads use Gradle's separate dependency cache.

Each MuzzleTask is already cacheable. Reuse depends on its implementation, directive and instrumentation/tooling/dependency classpaths; a stable task name alone does not guarantee a hit. Each planned check has two build-cache outcomes:

• Matching entry: restore reports/<task>.txt and skip the compatibility assertion.
• No matching entry: run the assertion and cache its output on success. This can happen for a newly published or newly sampled dependency version, changed inputs, or an expired entry.

A warm run can therefore mix restored and executed checks: the planner is free to select versions whose results have not been cached. Random version sampling can change that selection between runs.

muzzle-end is not cacheable; it regenerates reports from both restored and newly produced result files.

[!CAUTION]
The actual validation JDK identity still needs tracking beyond the requested Java major.

Latest dependencies and scheduled locks

• Same resolved classpath and other inputs: reuse is expected.
• Changed resolved classpath: execute, unless that exact input combination is already cached.

The scheduled lock refresh does not change test dependency selection in this PR. It runs in build and validate_build, not in build_tests or test jobs, and the rewritten locks are not shared. build also uses -PskipTests, so it produces no cached test results against those refreshed locks.

Before and after the migration, test jobs resolve dependencies using their checked-in locks. The S3 build cache changes output reuse: matching results from earlier test jobs can now be restored across pipelines. Outputs built with refreshed dependencies are reusable only when the consuming task's fingerprints match. No lock-refresh or lock-handoff change is needed for this migration.

Remaining input checks

Standard sources, classpaths and test options are tracked. Check full JDK/native-tool versions, relevant environment variables, service state and dependencies resolved inside nested builds. Paths and moving labels do not identify their contents. The S3 build cache adds no inputs automatically.

Compare representative results with --no-build-cache --rerun-tasks. For untrackable external state, disable caching and up-to-date reuse on affected tasks. Keep release isolation and independent validate_build checks.

Success measures and remaining work

Primary metrics: median/tail job duration, pipeline critical path, S3 build cache lookup hit rate and Gradle FROM-CACHE, split by job family and architecture. Also track writes/misses, transfer time/bytes, cache errors and actual test executions. The plugin's cumulative cache-time estimate is not pipeline wall time.

• Validate release isolation, artifact handoffs and input invalidation in CI.
• Choose the shared non-release prefix and trusted-writer policy; CI_COMMIT_REF_SLUG remains experiment-only.
• Enable nightly warming on master and check architecture/task coverage; keep fresh dependency/container validation separate.
• Diagnose remaining Spotless/Muzzle misses and compare more commits against the GitLab-cache baseline.

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors
• Once approved, use merge queue to merge the PR

Jira ticket: [PROJ-IDENT]