Skip to content

Separate Bedrock QA releases and production role images - #2837

Open
iwiznia wants to merge 1 commit into
mainfrom
ionatan/qa-cpp-release-ci
Open

iwiznia wants to merge 1 commit into
mainfrom
ionatan/qa-cpp-release-ci

Conversation

@iwiznia

@iwiznia iwiznia commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Details

Implement the Bedrock CI/release/image portion of the independent Authrock/Webrock QA cutover.

Companion BackupManager PR: https://github.com/Expensify/ExpensifyBackupManager/pull/188

  • Subscribe the existing Bedrock Test Suite (.github/workflows/bedrock.yml) to release: published. This covers stable releases and prereleases, including draft-to-published prereleases, without subscribing to both publication events.
  • Attach bedrock to the event's exact release tag and explicitly pass its prerelease flag. The pinned softprops/action-gh-release version supports both inputs. Grant the uploader contents: write.
  • Add publish-expensidev-bedrock-image.yml: production pushes publish only the matching role, with the exact pushed Bedrock SHA. Manual runs rebuild both roles from their mapped production refs. QA pushes cannot trigger production-image publication.
  • Keep the release workflow name/path expected by Ops' release-test waiting and recovery flow.
Role Bedrock BackupManager Other dependencies
Authrock expensify_prod production Auth production
Webrock expensify_prod_webrock production-webrock TableManager, FuzzyBot and WhitelistDB production

Both images use Expensidev main for their Dockerfiles and publish amd64/arm64 images plus the matching multi-architecture latest manifest.

Cutover sequencing: create/protect the deployment branches manually; land additive dependencies first; merge these repository CI changes into main during the C++ deploy freeze before the final main → qa-authrock/qa-webrock promotion PRs. Synchronize expensify_prod_webrock from the current expensify_prod through a protected PR, then activate the coordinated Ops/Salt mappings and Webrock Jenkins switches. The first production-first cycle promotes each cluster's QA candidate to production; only after production completes does that cluster advance QA from main.

Outstanding rollout prerequisite: confirm Bedrock can use an organization-provided MELVIN_GH_TOKEN or CODE_EXPENSIFY_TOKEN with private dependency checkout and GHCR publishing access. Repository-level secret listing returned no entries; the Actions permissions endpoint returned 403, so inherited organization access could not be established. This PR uses the same token fallback as the existing role-image publishers.

Fixed Issues

For https://github.com/Expensify/Expensify/issues/639785

Tests

AI Tests

Verified on 2026-10-06 at head b551fc8d569f23c629c2bd778221d140295030e6 (base 5ee6fc6b8b8e2659e21b6283a71213a957b20f3a). GitHub PR checks tested the synthetic merge 0f6b51ec56d5cd94cd852e4b362d62491efdba0a.

  • PASS - GitHub CI: Bedrock Test Suite: C++ Styler, Create Bedrock and Test, Bedrock Tests, and Bedrock Tests with HCTree. Upload Bedrock Binaries was correctly skipped for the pull-request event.
  • PASS - GitHub CI: Validate Github Actions and Workflows: actionlint, validateSchemas, and validateImmutableActionRefs. Inspected job results and validation logs; both changed workflow files were covered.
  • PASS - local: git diff --check origin/main...HEAD.
  • PASS - isolated, read-only smoke checks: parsed both workflow YAML files with Ruby/Psych and evaluated their equality/and/or routing expressions with synthetic contexts. Covered each production push, manual dispatch from main and either QA branch, matching role manifests/dependency refs, exact pushed Bedrock SHA, both architectures, exclusion of QA/main/feature pushes, release tag and both prerelease flag values, uploader write permission, and exclusion of PR uploads. This is static/expression verification, not execution of GitHub Actions or Docker. The temporary checker is not a repository test and is removed after this audit.
  • Reviewed: GitHub's release event documentation and the pinned uploader's release update logic / boolean input parsing.
  • Test audit: the complete PR changes only two workflow YAML files. No new test files, fixtures, or orphan tests; nothing to remove. No C++ source/build-script changes, so an additional Auth compatibility build was not run in this workflow-only audit.
  • Review audit: inspected inline comments, submitted reviews, general comments, and review threads. Codex completed without actionable findings; no inline/submitted reviews or threads were present. No code fixes were needed. Verify peer review is excluded from this CI audit as requested.

Manual rollout blockers and checks

Read-only prerequisite checks on 2026-10-06 confirmed that Bedrock expensify_prod_webrock and BackupManager production-webrock do not exist. The mapped Authrock refs and the other plugin production refs exist. Token access remains unverified as described above; an empty repository secret list does not establish whether organization secrets are inherited.

  1. Have the rollout owner create/protect/synchronize the missing production refs and confirm private dependency checkout plus GHCR publishing authorization.
  2. At the planned QA release, publish a prerelease for each QA stream and confirm Bedrock Test Suite runs, uploads bedrock to the same tag, and leaves the release a prerelease without changing the latest stable release.
  3. At each planned production promotion, confirm the exact pushed Bedrock SHA builds only that cluster's image with the dependency refs above, on both architectures.
  4. After both production branch sets are ready, run the manual publisher during the authorized rollout and confirm each role uses its own production refs regardless of the dispatch branch.
  5. Confirm a QA branch push does not publish either production image and Ops discovers the release run by bedrock.yml and the exact tag. The routing portion passed static checks; the end-to-end rollout remains manual.
  6. Complete any required Auth compatibility build in an isolated authorized environment before rollout. It is not covered by the Bedrock PR checks above.

No deployment branches/protections, secrets, promotions, deployments, live images, releases, or PR merges were changed during verification. No production-image workflow was dispatched and no shared VM/services were modified.


Internal Testing Reminder: when changing bedrock, please compile auth against your new changes

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-06T16:56:08.057938Z b551fc8 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant