ci(e2e): shard scenarios across a retrying matrix - #405
Merged
Merged
Conversation
Run the 76 multistep scenarios as a fail-fast:false matrix of 5 serial shards instead of one serial 45-minute job, so a flake is an isolated ~9-minute single-shard rerun. gotestsum --rerun-fails=2 gives each leg 3 attempts, rerunning only failed scenarios. workflow_dispatch shard and scenario inputs collapse the matrix to one leg or one scenario for targeted runs. The Integration Gate aggregates all legs; the workflow name and gate identity are unchanged. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
joshua-temple
enabled auto-merge (squash)
June 27, 2026 22:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The e2e suite runs 76 multistep scenarios serially on ONE runner (E2E_PARALLEL=1). Any single per-scenario environmental flake (act startup, Docker exec timing) reds the whole ~45-min job, forcing admin-overrides to merge unrelated PRs.
Fix
Shard the scenarios across a fail-fast:false matrix of 5 serial legs (~15 each, ~9 min parallel), each with 3 attempts (gotestsum --rerun-fails=2, rerunning only the failed scenarios). A flake now self-heals within a leg, or is an isolated ~9-min single-shard rerun instead of a 45-min full-suite gamble.
e2e/harness/shard.go:ShardFromEnv(E2E_SHARD_INDEX/TOTAL, default 0-of-1 so localgo testis unchanged), round-robinSelectShard(complete + disjoint partition), FNVOwnsfor standalone heavy tests (each runs once across the fleet). Unit-tested -race.e2e.yaml: aplanjob computes the shard matrix;gateneeds all legs (green only if every shard passes). Workflowname:and gate identity unchanged (fleet-e2e's workflow_run depends on them).shard=Ninput runs only that leg;scenario=<regex>input collapses the matrix and runs only matching scenarios (Go-side E2E_SCENARIO filter, since gotestsum does not honor passthrough -run).Proof (local Docker)
total=1 -> 76 scenarios; shard 0/5 ran its subset (PASS) and the same scenario was absent from shard 1; E2E_SCENARIO selected exactly 1 and ran only it (PASS); gotestsum drove the retries. go build/vet/lint/actionlint, shard unit tests -race, and main
go test(2021 pass) all clean.This PR's own CI is the first live run of the sharded matrix. Addresses the e2e flakiness that forced recent admin-merges.