Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -415,7 +415,10 @@ gh workflow run rollback.yml \

### Safety Features

- **Image Verification** - Confirms image exists before deploying
- **Image Verification** - Confirms image exists before deploying. For AWS,
the check parses the account, region and repository out of the exact image
URI Terraform deploys, refuses to run if that account is not the account of
`AWS_ROLE_TO_ASSUME`, and pins the lookup to that registry
- **Audit Trail** - Records all rollbacks (365 day retention)
- **Reason Tracking** - Requires reason for accountability
- **Manual Only** - Cannot be triggered automatically
Expand Down
23 changes: 23 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1658,6 +1658,28 @@ jobs:
- name: Run state namespace self-tests
run: bash scripts/test-aws-tfstate-platform-key.sh

# Run the AWS rollback jobs' "Verify image exists" step, lifted out of
# rollback.yml, against a stubbed aws CLI (#142). IMAGE_URI names the
# AWS_ACCOUNT_ID registry while the step runs as AWS_ROLE_TO_ASSUME, so the
# step must parse the registry out of the IMAGE_URI Terraform deploys, refuse
# when its account differs from the role's, and look the tag up there. A
# rollback run cannot show this without real credentials, so it is asserted
# here. Fast (shell only), so it always runs.
rollback-ecr-registry:
name: Rollback ECR registry check
runs-on: ubuntu-latest
permissions:
contents: read

steps:
- name: Checkout code
uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5.0.1
with:
persist-credentials: false

- name: Run rollback verify-step self-tests
run: bash scripts/test-rollback-ecr-registry.sh

# Assert that no Terraform file declares an azurerm_key_vault_access_policy
# resource. This project's only Key Vault sets enable_rbac_authorization =
# true, and an RBAC-enabled vault ignores access policies entirely, so such a
Expand Down Expand Up @@ -1729,6 +1751,7 @@ jobs:
- rds-deletion-protection-scope
- cloud-sql-delete-scope
- aws-tfstate-platform-key
- rollback-ecr-registry
- azure-kv-access-policy
- migration-down-gate
if: always()
Expand Down
88 changes: 72 additions & 16 deletions .github/workflows/rollback.yml
Original file line number Diff line number Diff line change
Expand Up @@ -237,24 +237,52 @@ jobs:
aws-region: ${{ vars.AWS_REGION || 'us-east-1' }}

- name: Verify image exists
# Checks the exact IMAGE_URI the Terraform step below deploys (#142).
# The account, region and repository are parsed out of that string
# rather than re-read from vars: this job binds `environment:`, so
# vars.* resolve environment-first here but repo/org-level in the
# validate job that built IMAGE_URI, and the two can differ.
env:
IMAGE_TAG: ${{ inputs.image_tag }}
ECR_REPOSITORY: ${{ vars.ECR_REPOSITORY }}
AWS_REGION: ${{ vars.AWS_REGION || 'us-east-1' }}
IMAGE_URI: ${{ needs.validate.outputs.image_uri }}
run: |
set -euo pipefail

if [ -z "${ECR_REPOSITORY:-}" ]; then
echo "::error::ECR_REPOSITORY repository variable is not set"
# <account>.dkr.ecr.<region>.amazonaws.com/<repository>:<tag>, as the
# validate job builds it. Anything else fails closed.
IMAGE_URI="${IMAGE_URI:-}"
ECR_URI_RE='^([0-9]{12})\.dkr\.ecr\.([a-z0-9-]+)\.amazonaws\.com/([a-z0-9][a-z0-9._/-]*):([A-Za-z0-9._-]+)$'
if [[ ! "$IMAGE_URI" =~ $ECR_URI_RE ]]; then
echo "::error::Image URI from the validate job is not an ECR image URI (<account>.dkr.ecr.<region>.amazonaws.com/<repository>:<tag>)"
exit 1
fi
REGISTRY_ACCOUNT="${BASH_REMATCH[1]}"
REGISTRY_REGION="${BASH_REMATCH[2]}"
REPOSITORY="${BASH_REMATCH[3]}"
IMAGE_TAG="${BASH_REMATCH[4]}"

# Terraform applies with the assumed role, and the deploy workflows
# push to the ECR repository in that role's account. Refuse to roll
# back when the registry IMAGE_URI names is in a different account,
# rather than deploy an image from a registry this job never checked.
if ! CALLER_ACCOUNT="$(aws sts get-caller-identity --query Account --output text)"; then
echo "::error::Could not determine the assumed role's AWS account"
exit 1
fi
if [ "$CALLER_ACCOUNT" != "$REGISTRY_ACCOUNT" ]; then
echo "::error::Image URI registry account ($REGISTRY_ACCOUNT) does not match the assumed role's account ($CALLER_ACCOUNT); refusing to roll back"
exit 1
fi

echo "Checking if image exists: $ECR_REPOSITORY:$IMAGE_TAG"
echo "Checking if image exists: $IMAGE_URI"

# --registry-id and --region pin the lookup to the registry IMAGE_URI
# names instead of the CLI's defaults (the caller's own account and
# the configured region).
if aws ecr describe-images \
--repository-name "$ECR_REPOSITORY" \
--registry-id "$REGISTRY_ACCOUNT" \
--repository-name "$REPOSITORY" \
--image-ids "imageTag=$IMAGE_TAG" \
--region "$AWS_REGION" > /dev/null 2>&1; then
--region "$REGISTRY_REGION" > /dev/null 2>&1; then
echo "✅ Image exists in ECR"
else
echo "::error::Image not found in ECR"
Expand Down Expand Up @@ -344,24 +372,52 @@ jobs:
aws-region: ${{ vars.AWS_REGION || 'us-east-1' }}

- name: Verify image exists
# Checks the exact IMAGE_URI the Terraform step below deploys (#142).
# The account, region and repository are parsed out of that string
# rather than re-read from vars: this job binds `environment:`, so
# vars.* resolve environment-first here but repo/org-level in the
# validate job that built IMAGE_URI, and the two can differ.
env:
IMAGE_TAG: ${{ inputs.image_tag }}
ECR_REPOSITORY: ${{ vars.ECR_REPOSITORY }}
AWS_REGION: ${{ vars.AWS_REGION || 'us-east-1' }}
IMAGE_URI: ${{ needs.validate.outputs.image_uri }}
run: |
set -euo pipefail

if [ -z "${ECR_REPOSITORY:-}" ]; then
echo "::error::ECR_REPOSITORY repository variable is not set"
# <account>.dkr.ecr.<region>.amazonaws.com/<repository>:<tag>, as the
# validate job builds it. Anything else fails closed.
IMAGE_URI="${IMAGE_URI:-}"
ECR_URI_RE='^([0-9]{12})\.dkr\.ecr\.([a-z0-9-]+)\.amazonaws\.com/([a-z0-9][a-z0-9._/-]*):([A-Za-z0-9._-]+)$'
if [[ ! "$IMAGE_URI" =~ $ECR_URI_RE ]]; then
echo "::error::Image URI from the validate job is not an ECR image URI (<account>.dkr.ecr.<region>.amazonaws.com/<repository>:<tag>)"
exit 1
fi
REGISTRY_ACCOUNT="${BASH_REMATCH[1]}"
REGISTRY_REGION="${BASH_REMATCH[2]}"
REPOSITORY="${BASH_REMATCH[3]}"
IMAGE_TAG="${BASH_REMATCH[4]}"

# Terraform applies with the assumed role, and the deploy workflows
# push to the ECR repository in that role's account. Refuse to roll
# back when the registry IMAGE_URI names is in a different account,
# rather than deploy an image from a registry this job never checked.
if ! CALLER_ACCOUNT="$(aws sts get-caller-identity --query Account --output text)"; then
echo "::error::Could not determine the assumed role's AWS account"
exit 1
fi
if [ "$CALLER_ACCOUNT" != "$REGISTRY_ACCOUNT" ]; then
echo "::error::Image URI registry account ($REGISTRY_ACCOUNT) does not match the assumed role's account ($CALLER_ACCOUNT); refusing to roll back"
exit 1
fi

echo "Checking if image exists: $ECR_REPOSITORY:$IMAGE_TAG"
echo "Checking if image exists: $IMAGE_URI"

# --registry-id and --region pin the lookup to the registry IMAGE_URI
# names instead of the CLI's defaults (the caller's own account and
# the configured region).
if aws ecr describe-images \
--repository-name "$ECR_REPOSITORY" \
--registry-id "$REGISTRY_ACCOUNT" \
--repository-name "$REPOSITORY" \
--image-ids "imageTag=$IMAGE_TAG" \
--region "$AWS_REGION" > /dev/null 2>&1; then
--region "$REGISTRY_REGION" > /dev/null 2>&1; then
echo "✅ Image exists in ECR"
else
echo "::error::Image not found in ECR"
Expand Down
Loading
Loading