fix(hotfix): self-heal a stale orphan env branch instead of fail-closing - #408
Merged
Merged
Conversation
When an env/<env> integration branch was left behind by an interrupted hotfix (apply ran and opened a PR, but finalize never ran so state[env].IsDiverged()==false), a subsequent hotfix plan would fail closed on the stale env tip even though the environment was not actually diverged. Introduce a safe self-heal path: - reconcileBranch and verifyRemoteEnvTip accept a singleFlightChecked bool that is only true when a real PRChecker ran and found no open cascade-hotfix PR, proving the branch is an abandoned orphan. - The self-heal fires exclusively when !IsDiverged() AND singleFlightChecked==true; every other mismatch continues to fail closed with the existing envTipDivergenceError. - ResetBranch is added to the gitRunner interface; it force-pushes the remote branch to the recorded base SHA then aligns the local ref. Replace ghPRChecker (gh CLI, absent from the pinned act runner image) with restPRChecker: a stdlib net/http client that reads GITHUB_API_URL, GH_TOKEN, and GITHUB_TOKEN from the environment and supports both bearer token and basic-auth credentials. It follows Link rel=next pagination and fails closed on non-200 responses. The generated plan job gains --repo so the single-flight gate is active in the workflow, job-level contents:write for the force-push, and GH_TOKEN so the REST checker can authenticate. Add unit tests (5 cases) covering the self-heal path, the fail-closed-when-diverged path, the fail-closed-when-not-checked path, the untouched-when-tip-matches path, and the chain mirror. Add REST checker tests (5 cases + 3 nextPageURL unit tests). Add e2e scenario hotfix-orphan-selfheal that creates the orphan condition, advances the recorded base past the stale tip, and proves the plan self-heals instead of aborting. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
joshua-temple
force-pushed
the
fix/hotfix-orphan-selfheal
branch
from
June 28, 2026 16:04
e6a0f3a to
a766085
Compare
joshua-temple
enabled auto-merge (squash)
June 28, 2026 16:10
This was referenced Jun 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
env/<env>is an ephemeral hotfix-integration branch. The preflightreconcileBranchonly checked branch-exists + tip == state.SHA and fail-closed on any mismatch ("interrupted hotfix or manual edits: replay the hotfix"). So an abandoned or interrupted hotfix (closed PR, cancelled workflow, dead runner) leaves a standing env branch; a later normal promote advances state.SHA; the next hotfix then hits the wall and needs manualgit push --deletesurgery. Hotfixes must be bulletproof - a leftover branch silently armed a future outage response.Fix
reconcileBranch(and the chain mirrorverifyRemoteEnvTip) now self-heal a STALE orphan: when the target env is NOT diverged AND the single-flight open-PR gate actually ran and found no open cascade-hotfix PR, it force-resetsenv/<env>to state.SHA and proceeds. Every other mismatch stays fail-closed with the existing error.Safety (the load-bearing part)
Divergence is only recorded at hotfix finalize, so a LIVE in-flight hotfix is still not-diverged - a naive reset would destroy in-progress work. So self-heal requires
!diverged && singleFlightChecked, wheresingleFlightCheckedis true ONLY when a real PR-checker ran (not the no-op). Two prerequisites this PR also fixes: the generated plan job now passes--repo(it didn't, so the gate was inert in production), and the PR-checker is now a portable REST call (GITHUB_API_URL+/pulls?state=open+cascade-hotfixlabel) that works on real GitHub and the gitea harness, with basic-auth support for the harnessuser:passcredential. No--repo/API/token -> no self-heal (fail-closed). REST errors fail-close.Tests
Unit (selfheal_test.go): orphan+checked -> reset+proceed; not-checked (no-op) -> fail-closed (proves no naive reset); diverged+mismatch -> fail-closed; tip-matches -> untouched; chain mirror. REST checker + pagination (pr_checker_test.go, 8 cases). e2e
hotfix-orphan-selfheal(apply-without-finalize -> promote past -> new hotfix self-heals) andhotfix-rejoinregression both PASS in Docker. 2043 unit tests, golangci-lint, actionlint all green.First PR of the hotfix-orphan goal; status consistency --fix and the fleet coverage follow.