fix: detect runtime crashes in e2e harness and preserve retry evidence - #146
Merged
Merged
Conversation
A non-zero act exit with no parseable job failure was classified transient and retried up to five times. A real runtime crash (cascade CLI or act) emits a goroutine dump that corrupts act's --json stream, so the job-failure parse misses and a genuine crash is misclassified as a flake, burning the retry budget while the CI log expires before the stack trace can be read. Detect Go-runtime crash signatures (panic:, fatal error:, a goroutine dump header, a fatal signal) in act's raw output and treat them as definitive, non-transient failures so they surface immediately with their reason. Anchor on real runtime signatures so a benign log line mentioning the word panic does not misfire. On retry-budget exhaustion, persist the last attempt's full act stdout/stderr to a per scenario+attempt file under e2e/_artifacts and reference its path in the returned error, so the next genuine failure keeps its stack-trace origin. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
This was referenced Jun 13, 2026
joshua-temple
added a commit
that referenced
this pull request
Jun 13, 2026
The crash classifier introduced in #146 flagged any act exit carrying a Go crash signature as a non-transient hard failure. On CI run 27480827218 this hard-failed Hotfix_Refusal_Guards on a bare goroutine dump from act's own monitorJobCancellation teardown watchdog (under cancel-in-progress): no panic/fatal/signal trigger and zero cascade frames, just nektos/act frames. That scenario passes locally with no retries, so it was an act-infra teardown flake that retry should absorb, not a real bug. Refine detectCrashSignature to classify a crash as non-transient only when BOTH a real runtime trigger (panic:/fatal error: at line start, or a signal SIG report) AND a real cascade stack frame are present. Drop the standalone goroutine-dump-header trigger. A non-zero act exit lacking either stays transient and retryable, preserving #146's benign-transient and real-job-failure paths. A genuine cascade-CLI panic still carries both a trigger and cmd/cascade or internal/... frames, so it is still caught and fast-failed. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A non-zero
actexit with no parseableJob failedevent was classified as a transient flake and retried up to five times. A real runtime crash (the cascade CLI or act itself) emits a Go goroutine dump that corrupts act's--jsonstream, so the job-failure parse misses entirely. A genuine crash was therefore misclassified as a transient flake: it burned the whole retry budget and the CI log expired before anyone could read the stack trace.Fix
Crash-signature detection beats the transient label.
detectCrashSignature(parser.go) scans act's raw stdout/stderr for anchored Go-runtime signatures:panic:/fatal error:at line start, a goroutine-dump header (^goroutine \d+ \[), and a fatal signal (signal SIGSEGV/SIGABRT/...).ParseActOutput(and the parse-failure fallback in act.go) setCrashed/CrashReasonon the result.normalizeWorkflowResultnow classifies a crashing non-zero exit as a definitive, non-transient failure (ExecError=false) so the scenario runner does not retry it and the reason surfaces. The existinghasJobFailurebehavior and the benign no-jobs transient path are preserved; this is an additional non-transient branch. Anchoring avoids misfiring on a log line that merely contains the word panic.Evidence on exhaustion. Failing workflow steps now carry the run's raw act logs. On retry-budget exhaustion,
runScenarioWithRetrywrites the last attempt's full act stdout/stderr to a per scenario+attempt file undere2e/_artifacts/and references the path in the returned error. Filenames are sanitized and unique per scenario+attempt so parallel scenarios never collide. The artifact dir is gitignored.Verification
go build ./...+go test ./...-> 1213 passed.go build ./...,go vet ./...clean;go test ./harness/... -count=1-> 228 passed (new crash + preserved-transient + artifact tests included).golangci-lint run ./...clean for both modules.--- PASS: TestMultiStepScenarios/Hotfix_Promote_Guards (37.99s)on the first attempt (no retry logged).