test(e2e): assert runtime outcomes instead of emitted script text - #552
Merged
Merged
Conversation
…nd state-write scenarios Replace source-grep e2e assertions with runtime-outcome checks for the breaking-change gate (37/38), release-trigger dispatch-only (40), and the state-write retry loop (08). Add an expect_log step surface plus orchestrate event and expect_no_run knobs so a scenario can assert a log line the running job emitted and a suppressed trigger. Relabel the rollout-strategy scenario (43) generation-only, since act cannot observe fail-fast matrix cancellation. Codify the assert-a-runtime-outcome standard in CONTRIBUTING. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
…gate signal The multi-env breaking-gate crossing (prerelease to release) coincides with the is_final_env publish, so the allow_breaking promote reaches the Trigger Release Build step. That step shells out to gh, which act does not have, so a correct run failed under act on exit 127. Scope the runtime assertion to the gate outcome the act path can observe: Publish Release cuts the v1.0.0 tag and reaps the RC before the gh dispatch, so the published release is the gate-bypass signal and flips against the blocked promote purely on the gate. The allow step is expect_failure with the published release and tags asserted; the state leaf is not asserted because Finalize runs after the dispatch. The release-build dispatch shape stays a labeled generation-only check, with its execution proven on the fleet. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
…dispatch Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Several e2e scenarios asserted a behavior by grepping the EMITTED workflow script (for example
can_proceed=true, the state-write retry loop text, apush:trigger line). Those assertions stay green even if the runtime behavior regresses, because they only check what the generator wrote, not what the workflow does when it runs.Fix
De-vacuum the load-bearing ones to assert on RUNTIME outcomes the running act+gitea workflow produces:
state.devadvances, plus a newexpect_logfor thecascade-state-write: ok attempt=1line the running finalize job emits.feat!, then assert a breaking publish/promote is BLOCKED (fails, release/state unchanged) and proceeds only withallow_breaking(release present).push(newexpect_no_run) but a run advances state onworkflow_dispatch.Harness: adds
StepExpect.ExpectLog(mirrorsRollbackStep.ExpectLog),OrchestrateStep.Event/ExpectNoRun, with a round-trip parse test. Scenario 43 (deploy fail_fast matrix) is relabeled generation-only with a documented residual, since act does not reliably cancel a fail-fast sibling matrix leg.Adds a CONTRIBUTING standard: assert a runtime outcome that differs under a regression, never a marker matchable in the emitted script.
Verification
go build ./...,go test ./...(2711 pass),golangci-lint run ./...clean; e2e module builds and vets; scenario configs validate against the schema. The e2e run on this PR exercises the new runtime assertions on correct behavior.