test(e2e): assert runtime outcomes instead of emitted script text

CONTRIBUTING.md

@@ -55,6 +55,7 @@ cascade holds to a few conventions in its own codebase and in the workflows it g
 - **Path fields reach every path sink**: a manifest field that widens which files a component reacts to must thread through all three places a path is consumed, or it is a silent bug. The emitted `on: push` paths filter fires the workflow, per-callback change detection decides which builds and deploys run, and the version commit range decides the bump. A field that reaches only some of these triggers a run that then no-ops, or bumps a version whose builds skip as unchanged. When you add such a field, add a test that asserts the shared path reaches each sink.
 - **A breaking generator or validation change moves with the fleet, in the same change**: a change that makes a previously valid manifest invalid, such as rejecting a field `parse-config` used to accept, is breaking even when it ships as a `fix:`. The fleet repin re-stamps every example repository onto the release-candidate binary and regenerates its workflows before any suite runs, so a manifest still carrying the now-rejected shape fails that repin, not the intended test. Before landing a validation change that can reject something that used to pass, scan the fleet example repos for that shape and migrate any that use it in the same pull request, alongside the doc's migration note. This generalizes the existing rule that a fleet suite and the eligibility logic it exercises are one coupled unit (see [Making a change](#making-a-change)); it applies to validation, not only to eligibility.
 - **Every generated workflow kind carries executing coverage**: each workflow the generator emits (`orchestrate`, `promote`, `external-update`, and the `cascade-` lanes) is mapped to the e2e scenarios and fleet lanes that run it in `internal/coverage/registry.yaml`. The coverage gate derives the emitted kinds straight from the generator source and fails when an emitted kind has no registry entry, so a new generated workflow cannot ship without a scenario or lane that exercises it. When you add a generated workflow kind, add its entry pointing at the scenario or fleet lane that runs it; a referenced scenario or lane that does not exist also fails the gate.
+- **Assert a runtime outcome, never the script that produces it**: an `e2e/` or fleet-suite assertion for a load-bearing behavior must compare against a runtime artifact that differs when the behavior regresses: a state leaf (`state.<env>` sha/version/ref), a job conclusion (`success`/`skipped`/`failure`), a preflight output, a tag, a release, a branch, a pull request, or a line the running job actually logged (`expect_log`). A passing assertion must be reachable only by the behavior working at run time. Never assert a behavior by grepping emitted script source for a marker that also appears in that source: a `workflow_files.contains`/`not_contains` check over a generated `.yaml` proves a string was rendered, not that the logic ran, and stays green when the runtime behavior is deleted because the marker text is still literally present in the file. As a cautionary example, the state-write retry loop was once "covered" by grepping the emitted `orchestrate.yaml` for `cascade-state-write: exhausted attempts=10`. That branch never runs on the happy path, yet the string is always present in the script, so the check was unconditionally green and a regression that replaced the whole loop with a single `git push` would have shipped green; the fix asserts the marker the running job emits (`cascade-state-write: ok attempt=1`) and proves it red-able by breaking the emission. Restrict `workflow_files` checks to behaviors whose entire effect is the generated shape (a `concurrency:` block, a `timeout-minutes:` value, a real-GitHub-only step act cannot execute) and label those scenarios generation-only in the header. When a behavior is genuinely un-runnable in act, its executing proof lives on the fleet, and a generation-only e2e cell is a ceiling that must be labeled as such, never credited as runtime coverage. The bar: a generator or behavior change adds or updates an assertion a reviewer can turn red by reverting the behavior alone, leaving the emitted string in place.
 - **Callback isolation**: generated workflows call your workflows via `workflow_call`, and cascade never reaches into your callback logic.
 - **Metadata courier**: cascade passes artifact identifiers and versions between stages. It never touches your container registry, package registry, or the systems you deploy to directly.
 

e2e/harness/multistep.go

@@ -224,6 +224,18 @@ type CommitStep struct {
 // repo-wide orchestrate.yaml, byte-identical to an orchestrate step with no config.
 type OrchestrateStep struct {
 	Component string `yaml:"component,omitempty"`
+	// Event overrides the GitHub event the orchestrate workflow runs under
+	// (default "push"). release_trigger: dispatch drops the push: trigger, so a
+	// scenario runs the same workflow under "push" (paired with ExpectNoRun to
+	// prove no job fires) and under "workflow_dispatch" (proving the dispatch path
+	// still advances state).
+	Event string `yaml:"event,omitempty"`
+	// ExpectNoRun asserts the orchestrate workflow produced no job run at all: act
+	// scheduled zero jobs because the event does not match any of the workflow's
+	// triggers. It is the runtime signal that a trigger was correctly suppressed.
+	// A bare source grep for the absent "push:" string cannot distinguish a
+	// suppressed trigger from a malformed on: block that would still fire.
+	ExpectNoRun bool `yaml:"expect_no_run,omitempty"`
 }
 
 // PromoteStep defines a promote action
@@ -384,6 +396,15 @@ type StepExpect struct {
 	// assert a config field survives a routine state write rather than being
 	// dropped on finalize.
 	Manifest *ManifestExpect `yaml:"manifest,omitempty"`
+	// ExpectLog asserts the last workflow run's logs contain this substring, so a
+	// scenario can prove a load-bearing runtime marker the running job actually
+	// emitted (for example the state-write loop's "cascade-state-write: ok
+	// attempt=1") instead of grepping the emitted script source, which stays green
+	// even when the loop is deleted because the marker text is still literally
+	// present in the file. Mirrors RollbackStep.ExpectLog and is evaluated against
+	// the same run result the Jobs assertion reads, so it applies to any step that
+	// ran a workflow (orchestrate, promote).
+	ExpectLog string `yaml:"expect_log,omitempty"`
 }
 
 // ManifestExpect asserts substrings against the live manifest read from Gitea.

e2e/harness/multistep_test.go

@@ -67,6 +67,53 @@ steps:
 	assert.Equal(t, "skipped", scenario.Steps[1].Expect.Jobs["build-worker"])
 }
 
+// TestParseMultiStepScenario_RuntimeAssertionFields round-trips the runtime
+// de-vacuum surfaces (expect_log on a step expectation; event and expect_no_run
+// on an orchestrate step) so a scenario that asserts a runtime log marker or a
+// suppressed trigger parses into the fields the runner consumes.
+func TestParseMultiStepScenario_RuntimeAssertionFields(t *testing.T) {
+	yaml := `
+name: "Runtime assertion fields"
+config:
+  environments: [dev]
+steps:
+  - name: "Orchestrate on push; assert state-write marker"
+    action: orchestrate
+    expect:
+      state:
+        dev:
+          version: "v0.1.0-rc.0"
+      expect_log: "cascade-state-write: ok attempt=1"
+  - name: "Push does not trigger a dispatch-only orchestrate"
+    action: orchestrate
+    orchestrate:
+      event: push
+      expect_no_run: true
+  - name: "Dispatch triggers the orchestrate"
+    action: orchestrate
+    orchestrate:
+      event: workflow_dispatch
+    expect:
+      state:
+        dev:
+          version: "v0.1.0-rc.0"
+`
+
+	scenario, err := ParseMultiStepScenario([]byte(yaml))
+	require.NoError(t, err)
+	require.Len(t, scenario.Steps, 3)
+
+	assert.Equal(t, "cascade-state-write: ok attempt=1", scenario.Steps[0].Expect.ExpectLog)
+
+	require.NotNil(t, scenario.Steps[1].Orchestrate)
+	assert.Equal(t, "push", scenario.Steps[1].Orchestrate.Event)
+	assert.True(t, scenario.Steps[1].Orchestrate.ExpectNoRun)
+
+	require.NotNil(t, scenario.Steps[2].Orchestrate)
+	assert.Equal(t, "workflow_dispatch", scenario.Steps[2].Orchestrate.Event)
+	assert.False(t, scenario.Steps[2].Orchestrate.ExpectNoRun)
+}
+
 func TestDiscoverMultiStepScenarios(t *testing.T) {
 	// Create temp directory with test scenarios
 	dir := t.TempDir()

e2e/harness/runner.go

@@ -1020,10 +1020,19 @@ func (r *Runner) executeOrchestrate(ctx context.Context, config Config, expectFa
 		branch = "main"
 	}
 
+	// The event defaults to push (a trunk merge). A dispatch-only orchestrate
+	// (release_trigger: dispatch) is run under "push" with ExpectNoRun to prove
+	// no job fires, and under "workflow_dispatch" to prove the dispatch path
+	// still advances state.
+	event := "push"
+	if orch != nil && orch.Event != "" {
+		event = orch.Event
+	}
+
 	// Run the actual orchestrate workflow via ActRunner
 	result, err := r.harness.act.RunWorkflowFromRepo(ctx, RunOpts{
 		WorkflowPath: workflowPath,
-		Event:        "push",
+		Event:        event,
 		Env: map[string]string{
 			"GITHUB_SHA":        sha,
 			"GITHUB_REF":        fmt.Sprintf("refs/heads/%s", branch),
@@ -1037,6 +1046,18 @@ func (r *Runner) executeOrchestrate(ctx context.Context, config Config, expectFa
 	// Store workflow result for assertions
 	r.lastWorkflowResult = result
 
+	// Handle the suppressed-trigger case: the workflow scheduled no jobs because
+	// the event matches none of its triggers. act marks a zero-job targeted run
+	// as a "failure" (missing/unloadable workflow); ExpectNoRun reinterprets that
+	// specific outcome as the success path, proving the trigger was dropped.
+	if orch != nil && orch.ExpectNoRun {
+		if len(result.Jobs) == 0 {
+			r.t.Logf("  Orchestrate: no job ran under event %q, as expected", event)
+			return nil
+		}
+		return fmt.Errorf("expected no orchestrate run under event %q but %d job(s) ran", event, len(result.Jobs))
+	}
+
 	// Handle expected failures (mirrors executePromote's ExpectFailure path).
 	if expectFailure {
 		if result.Conclusion == "failure" {
@@ -1540,6 +1561,16 @@ func (r *Runner) assertStep(ctx context.Context, step *Step, preState *Execution
 		allErrs = append(allErrs, errs...)
 	}
 
+	// Assert a runtime log marker the last workflow run actually emitted. This
+	// reads r.lastWorkflowResult (the same result the Jobs assertion consumes),
+	// so it proves a behavior ran rather than that its marker text was rendered
+	// into the workflow file. Skipped in unit-test mode where no workflow ran.
+	if expect.ExpectLog != "" && r.lastWorkflowResult != nil {
+		if !strings.Contains(r.lastWorkflowResult.Logs, expect.ExpectLog) {
+			allErrs = append(allErrs, fmt.Errorf("expected workflow logs to contain %q but did not", expect.ExpectLog))
+		}
+	}
+
 	return allErrs
 }
 

e2e/scenarios/08-state-push-retry.yaml

@@ -1,38 +1,48 @@
 name: "State Push Retry Loop"
 description: |
-  Verifies the generated orchestrate.yaml's Update Manifest step uses a
-  fetch+reset+reapply+push retry loop (#101) instead of a single git push.
+  Verifies the generated orchestrate.yaml's Update Manifest step runs its
+  fetch+reset+reapply+push retry loop (#101) at runtime instead of a single git
+  push.
 
-  Generator-output verification. Reproducing the actual concurrent-push race
-  needs real GHA and is out of scope for the act+gitea harness.
+  Runtime assertion. An orchestrate run advances dev state and the executed
+  finalize job logs the loop's success marker "cascade-state-write: ok
+  attempt=1". That marker is emitted only by the running loop, so it disappears
+  if the loop is replaced by a bare push, whereas a grep of the emitted script
+  source would stay green because the marker text is still literally present in
+  the file. Reproducing the actual concurrent-push 409 race needs real GHA and
+  stays a fleet concern.
 
 config:
   trunk_branch: main
-  environments: []
+  environments: [dev]
   builds:
     - name: app
       workflow: build.yaml
       triggers: ["src/**"]
   deploys: []
 
 steps:
-  - name: "Initial commit; assert retry loop in Update Manifest step"
+  - name: "Initial commit"
     action: commit
     commit:
       message: "feat: add app"
       files:
         src/app.go: |
           package main
           func main() {}
+
+  - name: "Orchestrate; assert the retry loop ran and wrote state"
+    action: orchestrate
     expect:
-      workflow_files:
-        - path: ".github/workflows/orchestrate.yaml"
-          contains:
-            - "Update Manifest"
-            - "for attempt in 1 2 3 4 5 6 7 8 9 10"
-            - "cascade-state-write: attempt=$attempt/10"
-            - "cascade-state-write: ok attempt=$attempt"
-            - "cascade-state-write: exhausted attempts=10"
-            - "git fetch origin"
-            - "git reset --hard"
-            - "apply_state_edits"
+      # State advanced: the Update Manifest loop reached its push and committed
+      # the new leaf. A regressed loop that never pushes leaves this unchanged.
+      state:
+        dev:
+          sha: commit1
+          version: "v0.1.0-rc.0"
+      jobs:
+        build-app: success
+      # The running loop's success marker. attempt=1 is the shell-expanded value
+      # the executed job logs on the first successful push, not the unexpanded
+      # "$attempt" that appears in the emitted script source.
+      expect_log: "cascade-state-write: ok attempt=1"

e2e/scenarios/37-release-breaking-gate.yaml

@@ -1,53 +1,105 @@
 name: "Release workflow breaking-change gate, dry-run, concurrency"
 description: |
-  Verifies that the single-environment release workflow emits its
-  breaking-change gate, the dry_run skip wiring, and the serialized
-  non-cancelling concurrency block (#322).
+  Verifies at runtime that the single-environment Release workflow's
+  breaking-change gate blocks publishing a breaking (major) release and lets it
+  through once allow_breaking_changes is set (#322).
 
-  A single-environment manifest generates a Release workflow into
-  promote.yaml (see 09-single-env-repo.yaml). This asserts the generated
-  shape of that workflow.
+  A single-environment manifest generates a Release workflow into promote.yaml
+  (see 09-single-env-repo.yaml). From a published v0.1.0 baseline, a feat!
+  commit is orchestrated to a v1.0.0-rc.0 draft. Publishing it without the flag
+  fails and leaves the published pointer at v0.1.0; the same publish with
+  allow_breaking succeeds and cuts v1.0.0. The blocked/allowed pair is observed
+  through the run conclusion and the published release, both of which flip when
+  the gate is inverted, rather than by grepping the emitted can_proceed script.
 
-  Covers:
-  - allow_breaking_changes workflow_dispatch input
-  - the Check Breaking Changes preflight step and its can_proceed outputs
-  - the dry_run workflow_dispatch input and the release-job skip guard
-  - the top-level concurrency block with cancel-in-progress: false
-
-  Generator-output verification only.
+  The dry_run skip wiring and the serialized non-cancelling concurrency block
+  are pure generated shape, asserted generation-only.
 
 config:
   trunk_branch: main
   environments: [prod]
-  deploys:
+  builds:
     - name: app
-      workflow: .github/workflows/deploy.yaml
+      workflow: build.yaml
       triggers: ["src/**"]
+  deploys: []
+
+# Published v0.1.0 baseline so a subsequent major bump is gated.
+setup:
+  state:
+    prod:
+      version: "v0.1.0"
+  tags:
+    - "v0.1.0"
+  releases:
+    - tag: "v0.1.0"
+      prerelease: false
 
 steps:
-  - name: "Initial commit generates release-flavored promote.yaml; assert the breaking gate"
+  - name: "Breaking change commit"
     action: commit
     commit:
-      message: "feat: add app source"
+      message: "feat!: major app refactor"
       files:
         src/app.go: |
           package main
-          func main() {}
+          func mainV2() {}
+
+  - name: "Orchestrate; assert draft RC and the emitted gate/concurrency shape"
+    action: orchestrate
     expect:
+      state:
+        prod:
+          sha: commit1
+          version: "v1.0.0-rc.0"
+      jobs:
+        build-app: success
+      releases:
+        - tag: "v1.0.0-rc.0"
+          prerelease: true
+          draft: true
+      # Generation-only: the dry_run skip guard and the serialized concurrency
+      # block are pure generated shape act does not execute here. The gate's
+      # RUNTIME behavior is proven by the blocked/allowed publish pair below.
       workflow_files:
         - path: ".github/workflows/promote.yaml"
           contains:
-            # Breaking-change gate input and detection step.
             - "      allow_breaking_changes:\n"
             - "      - name: Check Breaking Changes\n"
-            - "          ALLOW_BREAKING: ${{ github.event.inputs.allow_breaking_changes }}\n"
-            - "          echo \"has_breaking=$HAS_BREAKING\" >> \"$GITHUB_OUTPUT\"\n"
-            # The gate proceeds only when allowed, and blocks otherwise.
-            - "              echo \"can_proceed=true\" >> \"$GITHUB_OUTPUT\"\n"
-            - "              echo \"can_proceed=false\" >> \"$GITHUB_OUTPUT\"\n"
-            # dry_run input and the release-job skip guard.
             - "      dry_run:\n"
             - "    if: ${{ github.event.inputs.dry_run != 'true' }}\n"
-            # Serialized, non-cancelling concurrency block.
             - "concurrency:\n"
             - "  cancel-in-progress: false\n"
+
+  - name: "Publish without the flag is blocked"
+    action: promote
+    promote:
+      mode: default
+      expect_failure: true
+    expect:
+      # The published pointer never advanced to v1.0.0; the draft RC is intact.
+      state:
+        release:
+          unchanged: true
+      releases:
+        - tag: "v1.0.0"
+          deleted: true
+
+  - name: "Publish with allow_breaking proceeds"
+    action: promote
+    promote:
+      mode: default
+      allow_breaking: true
+    expect:
+      state:
+        release:
+          sha: commit1
+          version: "v1.0.0"
+      releases:
+        - tag: "v1.0.0"
+          prerelease: false
+          draft: false
+          latest: true
+      tags:
+        exist: ["v1.0.0"]
+        deleted: ["v1.0.0-rc.0"]