fix(e2e): cap scenario concurrency and narrow the transient classifier - #162
Merged
Merged
Conversation
Each scenario stands up its own docker network + gitea + act with nested act job containers. Run the heavy hotfix-family scenarios concurrently and they oversubscribe the docker address pool, host RAM, or gitea; act then exits non-zero with no parsed job failure and no cascade crash frame, lands in the classifier catch-all transient bucket, and burns the whole retry budget against the same ceiling. Add a host-sized scenarioConcurrency semaphore (independent of go test -parallel, E2E_MAX_CONCURRENT override) so heavy stacks serialize and stop self-inflicting the contention. Narrow the transient classifier so a non-zero act exit is retryable ONLY when a named infra-saturation signature is present (address-pool exhaustion, no space left, cannot allocate memory, gitea connection reset/refused, docker daemon unreachable); every other job-less, crash-free non-zero exit is now deterministic. Cut the retry budget from 5 to 2. Justified retries (gitea-405, network removal, convergence polls, product push) are untouched. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
joshua-temple
added a commit
that referenced
this pull request
Jun 14, 2026
#162 (#163) * test(e2e): seed callback workflows before generation for validate and failing-rollback scenarios PR #161 removed the inline run callback emission path and converted two scenarios to reusable workflows, but staged the referenced workflow bodies only via a step commit.files, which lands after generation runs. The harness seeds callback stubs and runs generation once at setup, so validate.yaml was missing at generation time and the failing rollback deploy used the generic non-failing stub. PR #162 narrowed the transient classifier and surfaced both. Add a setup_workflows map on MultiStepScenario seeded into the setup commit before generation, and a validate workflow_call stub clause keyed on config.validate.workflow. Move the failing deploy-app.yaml into setup_workflows so the rollback re-deploy fails under the Rollback caller, and rely on the seeded validate stub so orchestrate.yaml generation emits the gate. Test-infra only; no product behavior change. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com> * test(e2e): fail rollback re-deploy via dispatch env not caller workflow name Inside a reusable workflow_call callback, $GITHUB_WORKFLOW is the callee's own name, not the caller's, on act and on real GitHub. The converted rollback scenario keyed its failing deploy on the caller name (Rollback*), which never matches, so the re-deploy always succeeded and the state-unchanged assertion broke. Both promote and rollback also dispatch via workflow_dispatch, so the event name cannot disambiguate either. Set CASCADE_E2E_ROLLBACK on the rollback dispatch only and key the deploy callback on it; act passes top-level env into the reusable callee. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com> --------- Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The recurring hotfix-family e2e "flake" is self-inflicted contention, not an unavoidable external transient. Each scenario stands up its own docker network + gitea + act, and act spawns nested job containers per workflow run. Under parallelism the heaviest (hotfix-family) scenarios run their full stacks concurrently and oversubscribe the docker address pool (#125), host RAM, or gitea (#121). act then exits non-zero with no parsed job failure and no cascade crash frame, lands in the classifier's catch-all
default -> transientbucket, and burns the entire retry budget hitting the same ceiling.Fix
scenarioConcurrencysemaphore in the scenario runner, acquired before each scenario stands up its docker/gitea/act stack and released after teardown - independent ofgo test -parallel. Sized from host CPUs and total memory (min of floor(NumCPU/2) and floor(totalGiB/3), floored at 1), with anE2E_MAX_CONCURRENToverride. On a 4-core/8GB runner this caps at 2, removing the oversubscription at its root.defer h.Cleanup()on every path (including setup failure), soremoveNetworkalways runs and networks do not leak to re-cause fix(e2e): eliminate docker network address-pool exhaustion #125.Untouched (justified) retries
gitea-405
gitea_retry.go, network-removalremoveNetwork, gitea convergence polls, and the productCommitAndPushWithRetry/generated push-retry loops are unchanged - those guard genuine external transients.Verification
go build ./...; e2ego build/vet;golangci-lint run ./...clean on both modules.--- PASS: TestMultiStepScenarios/Two_Environment_Happy_Path (53.17s).Note: the address-pool ceiling itself is best mitigated by a daemon
default-address-poolsheadroom change on the runner, which the repo code cannot set alone; the cap + narrowed classifier make any residual saturation deterministic and named rather than silently retried.