CI e2e jobs failing with S3 403 InvalidAccessKeyId since Nov 2: re-trigger after CI AWS key rotation #2532

Summary

The CI AWS access key used by the Prow e2e jobs for the S3 BackupStorageLocation (BSL) buckets was disabled on 2026-11-02 at about 17:20 UTC (its last successful use was about 07:37 UTC the same day). Since then, every e2e job that creates a BSL fails, regardless of what the PR changes. A new key is being rotated in; until then, these failures are not caused by the code under test.

This issue catalogs the affected PRs and periodic jobs so they can be re-triggered once the credentials are fixed.

Signature

Velero (artifacts/**/velero.log) logs:

level=error msg="fail to validate backup store" backup-storage-location=openshift-adp/<bsl-name> controller=backup-storage-location error="rpc error: code = Unknown desc = operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: ..., HostID: ..., api error InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records."

The e2e build-log.txt then shows the BSL stuck unavailable until the 180s spec timeout:

BSL <bsl-name> phase is Unavailable
BSL <bsl-name> phase is Unavailable
... (repeated)

Time window: from 2026-11-02 ~17:20 UTC until the new key is in place. The earliest affected run in the catalog below started 2026-11-02 18:55 UTC; the latest started 2026-11-03 21:30 UTC.

How to tell a key failure from a real failure

• Key failure (re-trigger once creds are fixed): velero.log contains InvalidAccessKeyId / StatusCode: 403, and the e2e build-log has many repeated BSL ... phase is Unavailable lines (about 36 per spec at the 180s timeout). Happens on the first spec that creates a BSL.
• Real or other failure (do not just retest because of the key): no InvalidAccessKeyId in velero.log, and no repeated phase is Unavailable. Examples seen in the same window: cluster install failing in ipi-install-install before any test runs (see "Failures not related to the key"), or a genuine unit-test failure.
• Proof the creds work again: any S3-backed e2e job (*-e2e-test-aws, *-e2e-test-cli-aws, virt-kdm, virt-nokdm, hcp, kubevirt) that started after the key was replaced and gets past its first BSL creation. As of this writing, no S3-backed e2e run that started after 2026-11-02T17:20Z has passed, so the credentials do not appear to be fixed yet.

Scope

Runs started on or after 2026-11-02 17:20 UTC, scanned through 2026-11-04 00:15 UTC (all open or recently updated PRs and all periodic-ci-openshift-oadp-operator-* jobs):

• 5 PRs with KEY-FAILED jobs, plus 1 PR (openshift/oadp-operator#2531) whose e2e jobs were still running at scan time
• 49 KEY-FAILED presubmit runs and 40 KEY-FAILED periodic runs (87 of these 89 confirmed directly from velero.log; 2 identified by the BSL-Unavailable signature only because the run left no velero.log)
• 0 passing S3-backed e2e runs after the key was disabled
• Non-e2e jobs (unit-test, images, ci-index, ci-bundle) are unaffected: 34 of 35 runs passed (the one failure was a unit-test flake that passed on the re-run).

Prow run URL template: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_oadp-operator/<PR>/<full job name>/<run id>. The "full job name" is pull-ci-openshift-oadp-operator-<base branch>-<context>.

PRs with KEY-FAILED jobs (presubmits)

All heads below are unchanged as of this writing, so the failing runs are for the current head. A push to any of these PRs re-triggers all jobs, so a fresh push also works as a re-trigger.

Still running at scan time (expected to hit the same failure unless the key is fixed first): #2531 (fix(e2e): read VMB backup type from status.includedVolumes[].type, by kaovilai, base oadp-dev, head a77cc3c710), jobs 4.22-e2e-test-aws, 4.22-e2e-test-cli-aws, 4.23-e2e-test-aws, 5.0-e2e-test-aws, 5.0-e2e-test-cli-aws, 5.0-virt-kdm-e2e-test-aws, 5.0-virt-nokdm-e2e-test-aws, 5.1-e2e-test-aws, 5.1-e2e-test-cli-aws, started 2026-11-03 23:36 UTC. Check these before re-triggering.

Merge-bot retest budget

The merge bot's /retest-required comment reports "Remaining retests: 0 against base HEAD ... and N for PR HEAD ...". Latest values seen: #2515: 0 against base HEAD / 1 for PR HEAD; #2493: 0 / 1; #2527: 0 / 1; #2530: 0 / 2. No such comment on #2529 or #2531. Because the base-HEAD budget is already 0, the bot is unlikely to auto-retest these, so they will need a manual /retest (or a push) after the key is fixed. Do not spend retests before then.

Periodic jobs (periodic-ci-openshift-oadp-operator-*)

All periodic e2e jobs in this repo use AWS. Most recent run in the window, all started 2026-11-03 between 02:01 and 06:01 UTC. Run URL template: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/<full job name>/<run id>.

KEY-FAILED (40): run ids

• oadp-1.3: 4.14-e2e-test-aws-periodic 2117467086262374400; 4.14-e2e-test-cli-aws-periodic 2117436847159250944
• oadp-1.4: 4.18-e2e-test-aws-periodic 2117467098836897792; 4.18-e2e-test-cli-aws-periodic 2117436848841166848
• oadp-1.5: 4.19-e2e-test-aws-periodic 2117467107246477312; 4.19-e2e-test-cli-aws-periodic 2117436850527277056; 4.19-e2e-test-hcp-aws-periodic 2117482106220187648; 4.19-e2e-test-kubevirt-aws-periodic 2117467112271253504; 4.20-e2e-test-aws-periodic 2117467117291835392; 4.20-e2e-test-cli-aws-periodic 2117436852834144256; 4.20-e2e-test-hcp-aws-periodic 2117482108724187136; 4.20-e2e-test-kubevirt-aws-periodic 2117467119816806400
• oadp-1.6: 4.18-e2e-test-hcp-aws-periodic 2117482111257546752; 4.22-e2e-test-aws-periodic 2117467129878941696; 4.22-e2e-test-cli-aws-periodic 2117436853899497472; 4.22-e2e-test-hcp-aws-periodic 2117482113774129152; 4.22-virt-kdm-e2e-test-aws-periodic 2117467134924689408 (no velero.log, BSL signature only); 4.22-virt-nokdm-e2e-test-aws-periodic 2117467137432883200 (no velero.log, BSL signature only); 4.23-e2e-test-aws-periodic 2117467143309103104; 4.23-e2e-test-cli-aws-periodic 2117436856378331136; 4.23-e2e-test-hcp-aws-periodic 2117482116294905856; 4.23-virt-kdm-e2e-test-aws-periodic 2117467146656157696; 4.23-virt-nokdm-e2e-test-aws-periodic 2117467149176934400; 5.0-e2e-test-aws-periodic 2117467151693516800; 5.0-e2e-test-cli-aws-periodic 2117436858894913536; 5.0-e2e-test-hcp-aws-periodic 2117482118794711040; 5.0-virt-kdm-e2e-test-aws-periodic 2117467154210099200; 5.0-virt-nokdm-e2e-test-aws-periodic 2117467156735070208
• oadp-dev: 4.22-e2e-test-aws-periodic 2117467162590318592; 4.22-e2e-test-cli-aws-periodic 2117436861407301632; 4.23-e2e-test-aws-periodic 2117467165106900992; 5.0-e2e-test-aws-periodic 2117467167631872000; 5.0-e2e-test-cli-aws-periodic 2117436863093411840; 5.0-e2e-test-hcp-aws-periodic 2117482120472432640; 5.0-virt-kdm-e2e-test-aws-periodic 2117467170140065792; 5.1-e2e-test-aws-periodic 2117467192806084608; 5.1-e2e-test-cli-aws-periodic 2117436865622577152; 5.1-e2e-test-hcp-aws-periodic 2117482123827875840; 5.1-virt-kdm-e2e-test-aws-periodic 2117467196157333504; 5.1-virt-nokdm-e2e-test-aws-periodic 2117467197830860800

Other periodic results in the window:

• OTHER-FAILED (1): oadp-dev-5.0-e2e-test-hcp-guest-aws-periodic (2117497207895625728) failed creating its HCP cluster: the multi-arch hosted cluster is not enabled and the management cluster and nodepool CPU architectures do not match. Not key related.
• Still RUNNING (1): oadp-dev-5.0-virt-nokdm-e2e-test-aws-periodic (2117467172665036800), started 2026-11-03 04:01 UTC and apparently hung; check or abort it.
• PASSED (5): the unit-test periodics for oadp-1.3, oadp-1.4, oadp-1.5, oadp-1.6, and oadp-dev.

No GCP, Azure, or other-cloud periodics showed this signature (no such jobs were found in the window).

Periodics re-run on their normal schedule, so they should recover on their own once the key is replaced; no manual action is needed unless you want an earlier signal.

Failures not related to the key

• fix(e2e): default empty VMB status.type to full #2529 first runs (2026-11-02 ~18:55 UTC): 4.22-e2e-test-aws, 4.23-e2e-test-aws, 5.0-e2e-test-cli-aws failed in ipi-install-install with baseDomain: Invalid value: ...: the zone already has record sets for the domain of the cluster. This is a CI cluster provisioning problem, not the key. The later runs of these jobs fall under KEY-FAILED above.
• Bump CSV metadata to v1.5.9 #2530: the second runs of 4.19-e2e-test-hcp-aws and 4.20-e2e-test-hcp-aws failed with the same ipi-install-install baseDomain error.
• fix: retry AWS bucket region discovery on transient S3 errors #2515: one oadp-dev-unit-test run failed (exit status 2) and passed on the immediate re-run (flake).
• Add NonAdminSchedule design proposal #2522 (merged), Fix misleading podAnnotations deprecation warning on OADP 1.5 (OADP-8788) #2528 (merged), and [oadp-1.5] fix: add nil guard for BSL LastValidationTime in BSLsAreUpdated #2502 had no job runs started in this window, so nothing to re-trigger.
• Periodic oadp-dev-5.0-e2e-test-hcp-guest-aws-periodic: see above.

Checklist: once the key is rotated

• Confirm the new key works: run a quick S3-backed job (for example /test 5.0-e2e-test-cli-aws on a PR that already needs one) and verify the BSL reaches Available and the spec passes.
• fix(e2e): read VMB backup type from status.includedVolumes[].type #2531: let the running e2e jobs finish, or abort and re-run them after the key is fixed.
• fix: retry AWS bucket region discovery on transient S3 errors #2515: re-trigger the 6 failed e2e contexts (/test <context> or /retest).
• fix(e2e): default empty VMB status.type to full #2529: re-trigger the 9 failed e2e contexts.
• Bump CSV metadata to v1.5.9 #2530: re-trigger the 8 failed e2e contexts (including both hcp jobs).
• OADP-8924: Add HCP guest cluster pod backup test case #2493: re-trigger 5.0-virt-kdm-e2e-test-aws.
• Bug: fix CreateNamespace race when namespace is Terminating #2527: re-trigger 5.0-virt-kdm-e2e-test-aws.
• Re-check any other open PR whose e2e jobs started after 2026-11-02 17:20 UTC (for example via gh pr checks) and look for the signature above before retesting.
• Abort or re-run the stuck periodic oadp-dev-5.0-virt-nokdm-e2e-test-aws-periodic run (2117467172665036800) if it is still running.
• Confirm the next scheduled periodics are green, then close this issue.