Skip to content

fix: clone cluster failure with IRSA due to wrong AWS_REGION and overwritten AWS_ACCESS_KEY_ID - #1213

Merged
hughcapet merged 8 commits into
triggerfrom
fix/irsa-access-key-id-clobbered-by-s3-prefix
Sep 8, 2026
Merged

hughcapet merged 8 commits into
triggerfrom
fix/irsa-access-key-id-clobbered-by-s3-prefix

Conversation

@tcondeixa

@tcondeixa tcondeixa commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Fix three bugs in configure_spilo.py that cause clone clusters to fail when using IRSA.

Bug 1 — AWS_ACCESS_KEY_ID overwritten with S3 prefix
A stale "future compatibility" block introduced during the wal-e → wal-g migration copies WALG_S3_PREFIX into AWS_ACCESS_KEY_ID when no static credentials are set. With IRSA, AWS_ACCESS_KEY_ID is legitimately absent, so wal-g ends up with a garbage S3 path as a credential. Regular clusters survive this because wal-g's credential chain picks up AWS_ROLE_ARN/AWS_WEB_IDENTITY_TOKEN_FILE first — but it remains a latent issue.

Bug 2 — AWS_REGION incorrectly extracted as loca
The regex used to derive the AWS region was applied to the full S3 path. When PGVERSION is not yet known at clone configure time, the path ends with /wal/ (no version), causing the greedy regex to match kube-1 in the bucket name and extract the wrong preceding segment (loca) as the region. Fixed by extracting the bucket name before running the regex. Also renamed the temporary variable from prefix to bucket_or_prefix to avoid shadowing the function parameter, which would have incorrectly suppressed BACKUP_NUM_TO_RETAIN for the main envdir when S3 is configured.

Bug 3 — AWS_REGION not falling back to bare env var for clone and standby envdir
The IRSA fallback that copies bare env vars when prefixed versions are absent covered AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE, and AWS_STS_REGIONAL_ENDPOINTS — but not AWS_REGION. Even with AWS_REGION=eu-central-1 correctly set in the pod env, both the clone and standby envdir would still fall through to the broken regex. The fallback is shared code (irsa_names list in update_and_write_walg_configuration), so adding AWS_REGION to it fixes both in one change.

NOTE: PR created with the help of AI agent.

@tcondeixa tcondeixa added the bugfix Bug fixes and patches, e.g. fixing of a production issue that is affecting the customer experience. label Sep 2, 2026
@tcondeixa
tcondeixa marked this pull request as draft September 2, 2026 13:39
@tcondeixa tcondeixa changed the title fix: AWS_ACCESS_KEY_ID being overwritten with S3 prefix when using IRSA fix: clone cluster failure with IRSA due to wrong AWS_REGION and overwritten AWS_ACCESS_KEY_ID Sep 2, 2026
@tcondeixa
tcondeixa marked this pull request as ready for review September 2, 2026 14:23
@tcondeixa

Copy link
Copy Markdown
Collaborator Author

The main issue we are facing is the bug3, but AI reported the others while I was investigating.

@tcondeixa
tcondeixa requested a review from jopadi September 3, 2026 07:05
Copilot AI lite review requested due to automatic review settings September 4, 2026 10:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new AWS region fallback logic accidentally shadows the prefix function parameter, which can change later behavior (e.g., skipping BACKUP_NUM_TO_RETAIN in the unprefixed/main envdir).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR targets wal-g/IRSA clone reliability in configure_spilo.py by correcting how AWS region is derived, ensuring IRSA env var fallbacks include AWS_REGION, and removing a stale “future compatibility” block that could corrupt wal-g credentials.

Changes:

  • Extract S3 bucket name before applying the region regex to avoid false matches from path suffixes like /wal/.
  • Extend IRSA bare-env fallback propagation to include AWS_REGION alongside other IRSA variables.
  • Remove a block that could overwrite AWS_ACCESS_KEY_ID with a wal-g S3 prefix template.
File summaries
File Description
postgres-appliance/scripts/configure_spilo.py Fixes IRSA/clone-related wal-g env generation (region derivation, IRSA fallbacks, and removal of stale credential-overwrite logic).
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread postgres-appliance/scripts/configure_spilo.py Outdated
…environment

Signed-off-by: tcondeixa <tscondeixa@gmail.com>
…b-tcondeixa:zalando/spilo into fix/irsa-access-key-id-clobbered-by-s3-prefix
@tcondeixa

Copy link
Copy Markdown
Collaborator Author

👍

@hughcapet
hughcapet merged commit e46761e into trigger Sep 8, 2026
10 checks passed
@hughcapet
hughcapet deleted the fix/irsa-access-key-id-clobbered-by-s3-prefix branch September 8, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Bug fixes and patches, e.g. fixing of a production issue that is affecting the customer experience.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants