Repository navigation
ci: roll a failed deploy back to the commit that ran before (D3) - #82
Merged
Merged
Conversation
The deploy steps move into .github/actions/deploy (deploy.sh), used by the automatic deploy in ci.yml and by a new manual workflow, deploy.yml, which deploys a chosen commit on main (to go back to an earlier release or redeploy one) under the same environment and lock. deploy.sh: - reads the commit that runs now from the cluster before changing anything; - refuses to start unless all seven images of the commit are published; - deploys the commit from its own deployments/k8s (git archive into a work directory, the checkout stays untouched): server-side dry run, apply, rollout status per workload, then a smoke test from outside (/health reports the commit and /api/v1/leaderboard answers 200); - if any of that fails, deploys the earlier commit the same way, with its own manifests and images, and exits non-zero either way. Not kubectl rollout undo: a bad release can be a manifest change, and undo leaves the last-applied annotation stale. Every step checks its own result: set -e does not apply inside functions called from if or ||. Tested in an Ubuntu container (git, curl, jq, kubectl 1.35.9): images of the running commit found, an unbuilt commit reported missing for all seven, a commit's overlay rendered with seven images at its SHA while the checkout kept its placeholders, an unknown commit refused, and the smoke test passing against production for the commit it runs. actionlint and shellcheck clean. Plan: D3 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.github/actions/deploy/deploy.sh, used by the automatic deploy (ci.yml) and a new manualdeploy.yml(deploy a chosen commit on main: go back to an earlier release, or redeploy one; same environment anddeploy-prodlock).deploy.sh: reads what runs now from the cluster → refuses unless all 7 images of the commit are published → deploys the commit from its owndeployments/k8s(dry run, apply, rollout status, smoke test:/healthreports the commit and/api/v1/leaderboardanswers 200) → on any failure redeploys the previous commit with its own manifests and images and exits non-zero.kubectl rollout undo: a bad release can be a manifest change, and undo leaveslast-appliedstale.Tested
In an Ubuntu container (git, curl, jq, kubectl 1.35.9): images found for the running commit; an unbuilt commit reported missing for all 7 (nothing touched); a commit's overlay rendered with 7 images at its SHA while the checkout kept its placeholders; an unknown commit refused; smoke test passing against production. actionlint and shellcheck clean.
Next: this PR's merge is the first deploy through the script (happy path). Then a deliberately broken change (gateway
JWT_SECRETtoo short →config.Validaterefuses to start) to show the job turns red, rolls back, and production keeps serving.🤖 Generated with Claude Code