Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/cleanup-staging.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ on:
# deployment environment. `guard` authenticates to nothing and must not hold it.
#
# IMPORTANT — what the `environment:` binding does and does not buy. It scopes
# secrets/vars and sets the OIDC subject to `repo:<org/repo>:environment:staging`.
# secrets/vars and sets the OIDC subject to `<repo-ids>:environment:staging`.
# What each cloud does with that subject differs; see the per-job comments.
#
# It is NOT a reviewer gate. GitHub only blocks a job once required-reviewer
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/deploy-aws-lambda.yml
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,7 @@ jobs:
needs: prepare
permissions:
# Assumes the AWS deploy role. The `environment:` binding below is what
# scopes this job's OIDC `sub` to `repo:<org/repo>:environment:<name>`,
# scopes this job's OIDC `sub` to `<repo-ids>:environment:<name>`,
# which is the subject the trust policy matches. NOTE it is not by itself
# a reviewer gate: an Environment only blocks a job once required-reviewer
# protection rules are configured on it in repo settings, and as of this
Expand Down Expand Up @@ -368,7 +368,7 @@ jobs:
if: always() && needs.build-and-deploy.result == 'success'
permissions:
# Assumes the AWS deploy role. The `environment:` binding below is what
# scopes this job's OIDC `sub` to `repo:<org/repo>:environment:<name>`,
# scopes this job's OIDC `sub` to `<repo-ids>:environment:<name>`,
# which is the subject the trust policy matches. NOTE it is not by itself
# a reviewer gate: an Environment only blocks a job once required-reviewer
# protection rules are configured on it in repo settings, and as of this
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/deploy-azure.yml
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ jobs:
permissions:
# Assumes the Azure deploy SP via federated identity. The `environment:`
# binding below scopes this job's OIDC `sub` to
# `repo:<org/repo>:environment:<name>`, which must have a matching
# `<repo-ids>:environment:<name>`, which must have a matching
# `azuread_application_federated_identity_credential` (see
# terraform/environments/azure/ci-cd-permissions/sp.tf) or `azure/login`
# fails with AADSTS70021. NOTE the binding is not by itself a reviewer
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/deploy-gcp.yml
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ jobs:
permissions:
# Assumes the GCP deploy identity via Workload Identity Federation. The
# `environment:` binding below scopes this job's OIDC `sub` to
# `repo:<org/repo>:environment:<name>`. NOTE it is not by itself a
# `<repo-ids>:environment:<name>`. NOTE it is not by itself a
# reviewer gate: an Environment only blocks a job once required-reviewer
# protection rules are configured on it in repo settings. See #141.
id-token: write
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/destroy-fargate-dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ on:
# environment. `guard` authenticates to nothing and must not hold it.
#
# IMPORTANT — what the `environment:` binding does and does not buy. It scopes
# secrets/vars and sets the OIDC subject to `repo:<org/repo>:environment:dev`,
# secrets/vars and sets the OIDC subject to `<repo-ids>:environment:dev`,
# which the AWS trust policy's sub allowlist accepts (role.tf:30).
#
# It is NOT a reviewer gate. GitHub only blocks a job once required-reviewer
Expand Down Expand Up @@ -87,7 +87,7 @@ jobs:
permissions:
id-token: write
contents: read
# Binds the OIDC subject to repo:<org/repo>:environment:dev, which the AWS
# Binds the OIDC subject to <repo-ids>:environment:dev, which the AWS
# trust policy matches on. Not a reviewer gate until protection rules exist
# on this environment -- see the note at the top of this file.
environment: dev
Expand Down
142 changes: 142 additions & 0 deletions docs/deploy-trust-bootstrap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
# Deploy trust bootstrap runbook

The CI/CD deploy identities in `terraform/environments/{aws,azure,gcp}/ci-cd-permissions/` trust
GitHub Actions tokens from this repository, matched on its immutable IDs rather than its
`owner/name`. A rename, or a new repository that reuses a freed name, neither gains nor loses
deploy rights. Only `main` and the defined deployment environments can deploy.

This is a bootstrap step (see "CI/CD IAM" in `CLAUDE.md`). A privileged human runs it by hand.
The deploy workflows never run it.

## What each cloud accepts

The repository's OIDC `sub` claim template is set to
`["repository_owner_id", "repository_id", "context"]`, so GitHub mints subjects of the form:

```text
repository_owner_id:<owner_id>:repository_id:<repo_id>:ref:refs/heads/main
repository_owner_id:<owner_id>:repository_id:<repo_id>:environment:<name>
```

| Cloud | Trust | Matches |
| --- | --- | --- |
| AWS | `cudly-terraform-deploy` role, `StringEquals` on `token.actions.githubusercontent.com:sub` | the main-branch subject and one environment subject per name listed in `role.tf` |
| Azure | `cudly-terraform-deploy` app, one federated credential per subject | the main-branch subject and one environment subject per `var.github_environments` entry |
| GCP | WIF provider `github-actions` | `attribute_condition`: `assertion.repository_owner_id == '<owner_id>' && assertion.repository_id == '<repo_id>' && assertion.ref == '<deploy_ref>'`; the deploy SA grants `roles/iam.workloadIdentityUser` to `attribute.repository_id/<repo_id>` |

An environment subject does not say which branch triggered the run. Each environment's
deployment branch policy, limited to `main`, is what keeps other branches out. The bootstrap
script sets that policy.

For `LeanerCloud/cloud-commitments-platform`, `gh api repos/LeanerCloud/cloud-commitments-platform
--jq '.id,.owner.id'` returns `1391259743` and `86753534`.

## Prerequisites

- `main` is protected (#154). Protection controls who can reach the `ref:refs/heads/main` subject.
- A `gh` login with admin rights on the repository, plus `jq`.
- For each cloud, the credentials listed under "Prerequisites" in that module's `README.md`.

## Apply order

### 0. Record the current state

The rollback steps below restore from these files.

```bash
gh api repos/LeanerCloud/cloud-commitments-platform/actions/oidc/customization/sub > oidc-sub.before.json
gh api repos/LeanerCloud/cloud-commitments-platform/environments > environments.before.json
```

### 1. Configure GitHub

```bash
scripts/bootstrap-github-deploy-config.sh # dry run: prints each WOULD: line
scripts/bootstrap-github-deploy-config.sh --apply
scripts/bootstrap-github-deploy-config.sh # expect only OK: lines
```

The script:

- creates every environment the workflows bind to (`dev`, `staging`, `prod`,
`aws-fargate-*`, `aws-db-*`, `gcp-db-*` and `azure-db-*`, each for `dev`/`staging`/`prod`), with
custom deployment branch policies that admit only `main`. It keeps existing required reviewers
and wait timers;
- stops with an error if an environment admits any other branch or tag. Remove the extra policy in
**Settings > Environments**, then re-run;
- sets the OIDC `sub` claim template.

The old name-based subjects stop matching from this step on. They already fail for this repository
after the split, so this step causes no new outage.

### 2. Re-apply each cloud's `ci-cd-permissions`

Run this for `aws`, `azure` and `gcp`:

```bash
cd terraform/environments/<cloud>/ci-cd-permissions
# In terraform.tfvars: delete the old github_repo line and set
# github_repository_id = "1391259743"
# github_repository_owner_id = "86753534"
terraform init -backend-config=backend.hcl
terraform plan
terraform apply
```

Check that the plan contains only these changes. If it shows anything else, stop.

- **AWS**: `aws_iam_openid_connect_provider.github[0]` has moved to
`aws_iam_openid_connect_provider.github`. `aws_iam_role.cudly_deploy` is updated in place, and
its `sub` list changes from `repo:...` to `repository_owner_id:...` subjects.
- **Azure**: `github_main[0]` has moved to `github_main`. Each federated credential's `subject`
and `description` change.
- **GCP**: the pool and provider move from `[0]` to unindexed addresses. The provider's
`attribute_mapping` and `attribute_condition` are updated in place.
`google_service_account_iam_member.github_actions` is replaced, because its `member` changes
from `attribute.repository/<name>` to `attribute.repository_id/<id>`.

After the apply, no trust names a repository, so the entries for the old monorepo name are gone.

### 3. Test a deploy from `main`

```bash
gh workflow run deploy-aws-lambda.yml -R LeanerCloud/cloud-commitments-platform --ref main -f environment=dev
gh workflow run deploy-azure.yml -R LeanerCloud/cloud-commitments-platform --ref main -f environment=dev
gh workflow run deploy-gcp.yml -R LeanerCloud/cloud-commitments-platform --ref main -f environment=dev
```

Each run must get past its cloud login step. Then dispatch one of the workflows from any other
branch. GitHub must refuse the environment-bound job ("not allowed to deploy to dev due to
environment protection rules").

## Rollback

Restore the GitHub template and the Terraform together. Either one alone leaves every deploy
failing at the cloud login step.

1. Restore the default `sub` template:

```bash
gh api -X PUT repos/LeanerCloud/cloud-commitments-platform/actions/oidc/customization/sub -F use_default=true
```

2. Before you apply the previous module code with `github_repo` set, move each resource back to
its indexed address. If you skip this, Terraform destroys and recreates the resources. A
deleted GCP pool keeps its ID reserved for 30 days, so the recreate fails.

```bash
# aws
terraform state mv aws_iam_openid_connect_provider.github 'aws_iam_openid_connect_provider.github[0]'
# azure
terraform state mv azuread_application_federated_identity_credential.github_main \
'azuread_application_federated_identity_credential.github_main[0]'
# gcp
terraform state mv google_iam_workload_identity_pool.github 'google_iam_workload_identity_pool.github[0]'
terraform state mv google_iam_workload_identity_pool_provider.github 'google_iam_workload_identity_pool_provider.github[0]'
terraform state mv google_service_account_iam_member.github_actions 'google_service_account_iam_member.github_actions[0]'
```

Then run `terraform plan` and `terraform apply` from the previous commit.

3. You can leave the environments and their `main`-only branch policies in place, because they
only restrict. `environments.before.json` records what existed before step 1.
129 changes: 129 additions & 0 deletions scripts/bootstrap-github-deploy-config.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
#!/usr/bin/env bash
# Configures the GitHub side of the CI/CD deploy trust:
# 1. every deployment environment the workflows bind to, with a deployment
# branch policy that admits only main;
# 2. the repository's OIDC sub claim template, which decides the subject the
# terraform/environments/*/ci-cd-permissions trusts match.
#
# Dry-run by default; --apply performs the changes. Idempotent: settings that
# already match are left alone. Needs a gh login with admin on the repository.
# Apply order and rollback: docs/deploy-trust-bootstrap.md.
set -euo pipefail

REPO="LeanerCloud/cloud-commitments-platform"
DEPLOY_BRANCH="main"

# Must stay in lockstep with local.github_oidc_sub_prefix in
# terraform/environments/{aws,azure}/ci-cd-permissions; the Go test
# terraform/environments/deploy_trust_guard_test.go enforces it.
SUB_CLAIM_KEYS='["repository_owner_id","repository_id","context"]'

# Every environment a job in .github/workflows binds to, with the
# ${{ ... }} expressions resolved to their dev/staging/prod choices.
ENVIRONMENTS=(
dev staging prod
aws-fargate-dev aws-fargate-staging aws-fargate-prod
aws-db-dev aws-db-staging aws-db-prod
gcp-db-dev gcp-db-staging gcp-db-prod
azure-db-dev azure-db-staging azure-db-prod
)

usage() {
echo "Usage: $0 [--dry-run | --apply]"
echo " --dry-run print what would change (default)"
echo " --apply make the changes on github.com/$REPO"
}

APPLY=0
while [ $# -gt 0 ]; do
case "$1" in
--apply) APPLY=1 ;;
--dry-run) APPLY=0 ;;
-h | --help)
usage
exit 0
;;
*)
usage >&2
exit 2
;;
esac
shift
done

die() {
echo "ERROR: $*" >&2
exit 1
}

# change DESCRIPTION CMD...: runs CMD under --apply, only describes it otherwise.
change() {
local desc="$1"
shift
if [ "$APPLY" -eq 1 ]; then
echo "APPLY: $desc"
"$@" >/dev/null
else
echo "WOULD: $desc"
fi
}

command -v gh >/dev/null || die "gh CLI not found"
command -v jq >/dev/null || die "jq not found"

repo_ids=$(gh api "repos/$REPO" --jq '"repository_id=\(.id) repository_owner_id=\(.owner.id)"')
echo "Repository: $REPO ($repo_ids)"
echo "Check these IDs against terraform.tfvars in each ci-cd-permissions module."
[ "$APPLY" -eq 1 ] || echo "Dry run: nothing will be changed. Re-run with --apply to make the changes."

existing_envs=$(gh api --paginate "repos/$REPO/environments" --jq '.environments[]?.name')

for env in "${ENVIRONMENTS[@]}"; do
current='{}'
if printf '%s\n' "$existing_envs" | grep -qxF "$env"; then
current=$(gh api "repos/$REPO/environments/$env")
fi

policies=''
if jq -e '.deployment_branch_policy == {"protected_branches": false, "custom_branch_policies": true}' \
<<<"$current" >/dev/null; then
echo "OK: environment $env uses custom branch policies"
# This endpoint only answers once custom branch policies are enabled.
policies=$(gh api --paginate "repos/$REPO/environments/$env/deployment-branch-policies" \
--jq '.branch_policies[] | "\(.type // "branch"):\(.name)"')
else
# PUT replaces the environment's settings, so carry over any wait timer
# and required reviewers it already has.
payload=$(jq -c '
(.protection_rules // []) as $r
| {deployment_branch_policy: {protected_branches: false, custom_branch_policies: true}}
+ ([$r[] | select(.type == "wait_timer") | {wait_timer}] | add // {})
+ ([$r[] | select(.type == "required_reviewers")
| {prevent_self_review, reviewers: [.reviewers[] | {type, id: .reviewer.id}]}] | add // {})
' <<<"$current")
change "create/update environment $env with $payload" \
gh api -X PUT "repos/$REPO/environments/$env" --input - <<<"$payload"
fi

others=$(printf '%s\n' "$policies" | grep -vxF "branch:$DEPLOY_BRANCH" | grep -v '^$' || true)
[ -z "$others" ] || die "environment $env admits refs other than $DEPLOY_BRANCH:
$others
Remove them in Settings -> Environments -> $env, then re-run."

if printf '%s\n' "$policies" | grep -qxF "branch:$DEPLOY_BRANCH"; then
echo "OK: environment $env admits branch $DEPLOY_BRANCH only"
else
change "limit environment $env to branch $DEPLOY_BRANCH" \
gh api -X POST "repos/$REPO/environments/$env/deployment-branch-policies" \
-f name="$DEPLOY_BRANCH" -f type=branch
fi
done

sub_payload=$(jq -cn --argjson keys "$SUB_CLAIM_KEYS" '{use_default: false, include_claim_keys: $keys}')
current_sub=$(gh api "repos/$REPO/actions/oidc/customization/sub")
if jq -e --argjson want "$sub_payload" '. == $want' <<<"$current_sub" >/dev/null; then
echo "OK: OIDC sub claim template is $SUB_CLAIM_KEYS"
else
change "set OIDC sub claim template to $sub_payload (currently $current_sub)" \
gh api -X PUT "repos/$REPO/actions/oidc/customization/sub" --input - <<<"$sub_payload"
fi
Loading
Loading