Skip to content

fix(iac/aws): bind rollback.yml jobs to dev/staging/prod, not <cloud>-<env>-rollback - #367

Merged
cristim merged 2 commits into
mainfrom
fix/139-rollback-environment-bindings
Sep 28, 2026
Merged

cristim merged 2 commits into
mainfrom
fix/139-rollback-environment-bindings

Conversation

@cristim

@cristim cristim commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Summary

Rebinds rollback.yml's four rollback jobs to the same environment name
each job's corresponding deploy workflow already uses, instead of a
compound <cloud>-<env>-rollback name; deletes the AWS trust entries that
compound naming needed and this rebind makes unused.

Root cause (corrected after independent review)

The original version of this PR incorrectly claimed no cloud trusted the
compound <cloud>-<env>-rollback subjects.
That was wrong for AWS.
Re-verified against terraform/environments/aws/ci-cd-permissions/role.tf:

  • AWS already trusted all six aws-{lambda,fargate}-<env>-rollback
    subjects (role.tf:139-144, added for #1648). rollback-aws-lambda and
    rollback-aws-fargate could already authenticate before this PR.
  • Azure never trusted any compound rollback subject. Its federated
    credentials (terraform/environments/azure/ci-cd-permissions/sp.tf) only
    cover ref:refs/heads/main plus plain environment:{dev,staging,prod}
    (via the github_environment for_each). rollback-azure is the only
    job that actually failed OIDC auth (AADSTS70021) on every attempt.
  • GCP was unaffected either way. Its trust condition
    (github_oidc.tf) checks repository/ref, never sub, so the
    environment name was never load-bearing there.

So the real, narrower finding is: rollback-azure had a genuine
pre-existing auth failure; rollback-aws-lambda/rollback-aws-fargate
did not.

Fix

Rebind every rollback job to the same environment its own deploy job
uses
, rather than a -rollback-suffixed name:

  • rollback-aws-lambda -> ${{ inputs.environment }} (plain
    dev/staging/prod, matching deploy-aws-lambda.yml)
  • rollback-gcp -> ${{ inputs.environment }} (matching deploy-gcp.yml)
  • rollback-azure -> ${{ inputs.environment }} (matching
    deploy-azure.yml) -- this is the actual authentication fix
  • rollback-aws-fargate -> aws-fargate-${{ inputs.environment }}
    (matching deploy-aws-fargate.yml, not plain dev/staging/prod
    -- fixed after review; the first version of this PR bound it to the
    Lambda environment name by mistake)

Deleted the six now-dead AWS trust entries (role.tf:139-144) rather
than leaving them as unused trust surface: once every rollback job binds
its deploy's own environment name, nothing presents
environment:aws-{lambda,fargate}-<env>-rollback any more. Updated
role.tf's README table and environment count (9, not 15) to match.
Also dropped the now-stale rollback.yml -> azure-{dev,staging,prod}- rollback line from Azure's github_environments doc comment
(variables.tf) -- rollback-azure binds plain dev/staging/prod
now, already covered by that variable's default.

Option 1 from the issue (give rollback its own environment family
everywhere) would still need 12 more GitHub Environments and, for Azure
specifically, more federated credential resources (Azure allows no
wildcards). This PR stays with option 2 -- reuse what already exists and
is already trusted -- documented as a deliberate tradeoff in the
workflow-level comment: rollback-aws-lambda, rollback-gcp and
rollback-azure now share their approval gate with the corresponding
deploy environment; rollback-aws-fargate already shared it with
deploy-aws-fargate even before this PR.

Regression test

No automated test applies: this is an OIDC trust-boundary binding with no
local harness that exercises a real configure-aws-credentials/
azure/login exchange, and per WORKER_BRIEF.md this sandbox cannot
dispatch a real rollback against cloud infrastructure. Verified the
corrected root cause structurally instead:

$ grep -n '"repo:\${var.github_repo}' terraform/environments/aws/ci-cd-permissions/role.tf
# BEFORE this PR: ref:refs/heads/main, environment:{dev,staging,prod},
# environment:aws-fargate-{dev,staging}, environment:aws-db-*, PLUS the six
# environment:aws-{lambda,fargate}-{dev,staging,prod}-rollback entries --
# i.e. AWS already trusted the compound rollback subjects.

$ grep -n 'subject' terraform/environments/azure/ci-cd-permissions/sp.tf
# github_main (ref:refs/heads/main) and github_environment (for_each over
# var.github_environments = ["dev","staging","prod"]) -- no compound
# *-rollback subject anywhere, before or after this PR.

Confirmed via git show <parent>:... that both the AWS trust entries and
the Azure gap predate this PR and this whole issue family.

Verification

$ actionlint .github/workflows/rollback.yml
(native actionlint v1.7.12, exit 0, no findings)

$ zizmor --offline --persona=pedantic --min-severity=medium .github/workflows/rollback.yml
No findings to report. Good job! (5 ignored, 1 suppressed)

$ cd terraform/environments/aws/ci-cd-permissions && terraform fmt -check -diff . && terraform init -backend=false && terraform validate && tflint
$ cd terraform/environments/azure/ci-cd-permissions && terraform fmt -check -diff . && terraform init -backend=false && terraform validate && tflint
(no diff; Success! The configuration is valid; no tflint findings, both modules)

$ markdownlint terraform/environments/aws/ci-cd-permissions/README.md
(passed via pre-commit's markdownlint hook)

Pre-commit's actionlint hook (language: docker_image) was skipped
locally with SKIP=actionlint on both commits because this host's Docker
daemon is hung (per repo CLAUDE.md); native actionlint was run instead
(above). CI's own (non-Docker) actionlint/pre-commit jobs run for real on
this PR.

Related / not fixed here

Closes #139

Summary by CodeRabbit

  • Deployment and Rollback
    • Rollbacks for AWS Lambda, AWS Fargate, GCP, and Azure now use the shared dev, staging, or prod approval gate, aligning rollback approvals with deployments.

@coderabbitai

coderabbitai Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: LeanerCloud/cloud-commitments-platform/.coderabbit.yaml

Review profile: CHILL

Plan: Essentials

Run ID: f7c46d0b-b6ef-4c3b-be9d-9c7577a21139

📥 Commits

Reviewing files that changed from the base of the PR and between b9b0283 and 1a6cd00.

📒 Files selected for processing (1)
  • .github/workflows/rollback.yml

Included review availability: This review used your included allowance. 1 included review remains after this review. Your included PR review attempts over the past 7 days set your current allowance at 5 reviews per hour.


📝 Walkthrough

Walkthrough

The rollback workflow now binds its AWS Lambda, AWS Fargate, GCP, and Azure jobs to the plain dev, staging, or prod environment. Its comments document that rollback and deployment share the approval gate.

Changes

Rollback environments

Layer / File(s) Summary
Use shared deployment environments
.github/workflows/rollback.yml
The workflow documents required reviewers on the shared dev, staging, and prod environments. All four rollback jobs use the selected environment name instead of a cloud-specific rollback name.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: 🟡 Moderate · up to 1a6cd

Until the separately applied AWS and Azure trust policies accept the new environment identities, rollback authentication can fail. Confirm the bootstrap re-apply before relying on the changed rollback workflow.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed PR #367 implements issue #139 option 2. All four rollback jobs in .github/workflows/rollback.yml now use the existing dev, staging, or prod environments. These names match the AWS and Azure tr…
Out of Scope Changes check ✅ Passed The changed file is .github/workflows/rollback.yml. The changes update rollback environment bindings and add comments that explain reviewer requirements and the shared deploy approval gate. These ch…
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the primary change: rollback jobs now use dev, staging, or prod instead of cloud-specific rollback environments. The iac/aws scope is narrower than the full multi-cloud cha…
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Comment @coderabbitai help to get the list of available commands.

@cristim cristim added urgency/this-sprint Within the current sprint impact/internal Team-internal only priority/p1 Next up; this sprint severity/high Significant harm effort/s Hours type/bug Defect triaged Item has been triaged labels Sep 28, 2026
cristim and others added 2 commits September 28, 2026 05:03
…>-rollback

The four rollback jobs bound compound environment names
(aws-lambda-<env>-rollback, aws-fargate-<env>-rollback,
gcp-<env>-rollback, azure-<env>-rollback), producing the OIDC subject
repo:<org/repo>:environment:<name>. None of those compound names is in
any cloud's trust allowlist: AWS's role.tf sub list only has
ref:refs/heads/main plus plain environment:{dev,staging,prod}, and
Azure's federated credentials (sp.tf) only cover plain
dev/staging/prod. So configure-aws-credentials and azure/login failed
outright on every rollback attempt (AssumeRoleWithWebIdentity denied /
AADSTS70021 respectively) -- the rollback path has never been able to
authenticate, independent of anything this PR's sibling fixes touch.
GCP was unaffected: its trust condition checks repository and ref, not
sub, so the environment name was never load-bearing there.

Rebind all four jobs to the plain environment name
(${{ inputs.environment }}), reusing the dev/staging/prod
environments every deploy workflow already binds to and that AWS/Azure
already trust, per option 2 in the issue (the smaller of the two fixes
the issue lays out; option 1 would need 12 new environments and, for
Azure, 12 new federated credential resources, since Azure allows no
wildcards).

Tradeoff, called out in the workflow-level comment: a rollback now
shares its approval gate with the corresponding deploy environment
rather than having a distinct rollback-only reviewer set. This is a
policy call the issue leaves open; if rollback ever needs a separate
reviewer list, that is option 1, done later as its own change.

Not fixed here (separate issues): no environment in this repo has
protection rules yet (#141), and the live trust policies still need the
bootstrap re-apply from #2 before any of this authenticates against a
real cloud account.

Verified: actionlint (native v1.7.12, exit 0) and zizmor
(--offline --persona=pedantic --min-severity=medium, 0 findings) on
rollback.yml.

Closes #139

Co-Authored-By: claude-flow <ruv@ruv.net>
… fargate binding

Independent review caught three problems in the original rebind:

1. The workflow comment and PR body claimed AWS never trusted the
   compound <cloud>-<env>-rollback subjects. False: role.tf:139-144
   already allowlisted all six aws-{lambda,fargate}-<env>-rollback
   subjects (added for #1648) -- only Azure's federated credentials
   (sp.tf) lacked any compound-rollback entry, so rollback-azure was
   the only job that actually failed OIDC auth (AADSTS70021) before
   this fix. GCP was unaffected either way, since its trust condition
   never checks sub. Rewrote the workflow-level comment in rollback.yml
   to state this accurately instead of the blanket "no cloud trusted
   these" claim.

2. Once every rollback job binds its deploy's own environment name
   instead of a *-rollback suffix, nothing presents the six AWS
   compound subjects any more. Deleted them from role.tf (and the
   matching README table rows / count), rather than leaving unused
   trust surface that reads as intentional.

3. rollback-aws-fargate was rebound to plain dev/staging/prod, but
   deploy-aws-fargate.yml binds aws-fargate-<env> (already trusted,
   role.tf) -- a different, already-existing environment. Rebound
   rollback-aws-fargate to match its own deploy job instead of
   Lambda's plain-name environment.

Also dropped the now-stale "rollback.yml -> azure-{dev,staging,prod}-
rollback" line from azure/ci-cd-permissions/variables.tf's
github_environments doc comment -- rollback-azure binds plain
dev/staging/prod now, which that variable's default already covers.

Verified: actionlint (native v1.7.12) and zizmor (0 findings) on
rollback.yml; terraform fmt/init -backend=false/validate/tflint clean
on both terraform/environments/{aws,azure}/ci-cd-permissions.

Co-Authored-By: claude-flow <ruv@ruv.net>
@cristim
cristim force-pushed the fix/139-rollback-environment-bindings branch from 1a6cd00 to dae3ced Compare September 28, 2026 03:04
cristim added a commit that referenced this pull request Sep 28, 2026
… review

Independent review caught two problems left over from #367's final
shape:

1. The dev/staging/prod bullet had a garbled mid-sentence insertion
   ("deploy-aws-fargate's sibling compute platforms use their own
   names (below)") and, now that #367 rebinds rollback-aws-fargate to
   aws-fargate-<env> instead of plain dev/staging/prod, the "all four
   rollback.yml jobs" claim was also wrong -- only three of the four
   do. Rewrote the bullet to name the three rollback jobs that share
   this group and call out that rollback-aws-fargate pairs with
   deploy-aws-fargate.yml's own aws-fargate-<env> group instead.

2. The trust-allowlist note implied all three clouds gate environment
   names the same way via a "github_environments variable". Only Azure
   has one; AWS's trust list is a literal sub array inlined in
   role.tf, not a variable; GCP doesn't gate by environment/sub at all
   (ref-based only, per #141). Replaced the generic note with the
   three distinct mechanisms and where each one lives.

Verified: markdownlint clean.

Co-Authored-By: claude-flow <ruv@ruv.net>
@cristim

cristim commented Sep 28, 2026

Copy link
Copy Markdown
Member Author

Independent adversarial review (two rounds) + local verification: MERGE at dae3ced. Rollback jobs now bind the same environments their deploys use (dev/staging/prod; aws-fargate- for Fargate), which every cloud's trust already covers; the six orphaned aws-*-rollback subjects are removed from role.tf. Reconciled every role-to-assume job against role.tf: all presented subjects trusted, none of the removed ones presented. Bootstrap-only change, effective on the #2 re-apply. terraform/actionlint/zizmor clean; policy guard 40 passed; 25/25 checks.

@cristim
cristim merged commit b51ce64 into main Sep 28, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/s Hours impact/internal Team-internal only priority/p1 Next up; this sprint severity/high Significant harm triaged Item has been triaged type/bug Defect urgency/this-sprint Within the current sprint

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(iac/aws): rollback environments are absent from the OIDC trust allowlist, so AWS/Azure rollback cannot authenticate

1 participant