Skip to content

CLDSRV-955: Parallelize docker builds into a matrix#6228

Open
tcarmet wants to merge 1 commit into
development/9.3from
improvement/CLDSRV-955/docker-build-matrix
Open

CLDSRV-955: Parallelize docker builds into a matrix#6228
tcarmet wants to merge 1 commit into
development/9.3from
improvement/CLDSRV-955/docker-build-matrix

Conversation

@tcarmet

@tcarmet tcarmet commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Speeds up CI by building the container images in parallel instead of one after another, so the test jobs that depend on them start sooner. The release-only images that depend on the freshly built server image and are not needed by any test are moved off the critical path to run alongside the tests. Image names and tags are unchanged. This is the second of two changes tracked under CLDSRV-955 to reduce the tests workflow wall time; the first caches the installed dependency tree.

Convert the build job into a matrix that builds the cloudserver,
cloudserver-testcoverage, pykmip and ci-mongodb images in parallel
instead of sequentially, so downstream test jobs no longer wait on a
single serial build. All image tags are preserved exactly.

The testcoverage image writes to its own cache scope to avoid racing the
production build on the shared scope while still reading it for reuse.

Federation and the dashboards push are moved to a separate
build-federation job (needs: build): they depend on the pushed
production image but are not consumed by any test job, so they now run
in parallel with the tests rather than blocking them.
@bert-e

bert-e commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Hello tcarmet,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval
/bypass_build_status Bypass the build and test status
/bypass_commit_size Bypass the check on the size of the changeset TBA
/bypass_incompatible_branch Bypass the check on the source branch prefix
/bypass_jira_check Bypass the Jira issue check
/bypass_peer_approval Bypass the pull request peers' approval
/bypass_leader_approval Bypass the pull request leaders' approval
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request TBA
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e

bert-e commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue CLDSRV-955 contains:

  • None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 9.3.14

  • 9.4.0

Please check the Fix Version/s of CLDSRV-955, or the target
branch of this pull request.

@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
8489 1 8488 0
View the top 2 failed test(s) by shortest run time
should log correct bucketGetReplication operation with all required fields::Server Access Logs - File Output With v4 signature should log correct bucketGetReplication operation with all required fields
Stack Traces | 0.039s run time
Expected 4 log entries, got 5

5 !== 4
should complete a multipart upload on real AWS location with bucketMatch set to false::Multipart Upload API with AWS Backend should complete a multipart upload on real AWS location with bucketMatch set to false
Stack Traces | 60s run time
Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (.../tests/multipleBackend/multipartUpload.js)
View the full list of 1 ❄️ flaky test(s)
"after each" hook for "should fail if trying to overwrite a delete marker"::MPU with x-scal-s3-version-id header With v4 signature "after each" hook for "should fail if trying to overwrite a delete marker"

Flake rate in main: 100.00% (Passed 0 times, Failed 228 times)

Stack Traces | 0.018s run time
We encountered an internal error. Please try again.

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@tcarmet

tcarmet commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Docker build matrix - CI timing impact

Measured on a warm run of this branch (run 29056334730, warm attempt) vs a recent successful development/9.3 run (28062248343).

Build phase

development/9.3 this PR delta
Build phase wall time 623s (one sequential build job) 462s (matrix, max of 4 parallel entries) −161s (~26%)
Time from run start → first test job starts 630s 470s −160s

Matrix entry breakdown (warm):

entry duration
cloudserver (production) 208s
cloudserver-testcoverage 462s ← matrix bottleneck
pykmip 14s
ci-mongodb 13s
build-federation (separate job) 113s - now runs in parallel with the tests, no longer blocking them

Honest caveats

  • The win is real (~160s off the pre-test critical path) but smaller than it could be. cloudserver-testcoverage is FROM production; splitting it into its own parallel entry with a separate cache scope means it re-builds the production layers instead of reusing them (in the old sequential job it was only ~14s, right after production). A follow-up that couples production+testcoverage in a single matrix entry would drop the matrix wall to roughly max(~220s, pykmip, mongodb)~220s, i.e. a ~400s build-phase reduction rather than ~160s.
  • The development/9.3 GHA docker layer cache warmth varies (those runs are ~weekly and the cache evicts after 7 days), so the 623s baseline is not guaranteed fully-warm; treat these as directional.

Note: the functional suite is independently flaky on the shared MPU abort-race teardown (CLDSRV-938), unrelated to this change; this run needed reruns of a couple of single flaky tests to go green. All build/matrix jobs passed on every attempt.

@tcarmet tcarmet marked this pull request as ready for review July 10, 2026 00:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants