[Badri-Rajendran]

Problem

CD has never completed a successful run since it was added. Every run either failed, was skipped because CI was red, or sat queued on the approval gate for up to 16 hours. Every deploy so far has been done by hand.

Run	Result
34030331323	client-id/tenant-id empty — ran 23 min before the repo secrets existed. Self-resolved.
34067982447, 34099158004	AADSTS700213 at Azure login
6 runs	queued waiting 6–16 h on required_reviewers, never approved
8 runs	skipped — CI was red on main (fixed by #31)

Five defects, each independently fatal

1. OIDC login (blocked every run). The Entra federated credential carries the legacy subject repo:Badri-Rajendran/Trueup:environment:production. GitHub actually presents the immutable owner-ID/repo-ID form — confirmed at source: gh api repos/.../actions/oidc/customization/sub returns sub_claim_prefix: "repo:Badri-Rajendran@79736071/Trueup@1357531178".

2. Approval gate. required_reviewers on the production environment. Removed by request. environment: production deliberately stays on every job — it is what scopes the OIDC subject claim, so removing it would change the subject to :ref:refs/heads/main and re-break login.

3. Worker readiness check could never pass. It compared runningState to Running, but an app pinned to min=max=1 settles on RunningAtMaxScale. Verified live — backend, frontend and worker all report RunningAtMaxScale, never Running. It also indexed revision list [0], which has no ordering guarantee and can return a deprovisioning revision.

4. mcp-server would have crash-looped instantly. python app/mcp/entrypoint.py puts the script's own directory on sys.path instead of /app. Reproduced against the built image:

$ docker run --rm --entrypoint python trueup-backend:bf3bfaf app/mcp/entrypoint.py
ModuleNotFoundError: No module named 'app'

$ docker run --rm --entrypoint python trueup-backend:bf3bfaf -m app.mcp.entrypoint
<gets past imports, fails only on missing env config — expected>

docker-compose.yml:107 already used the -m form; cd.yml didn't.

5. Health check could report a false green. It curled the ingress immediately after containerapp update, which returns once the new revision is created, not once it serves — so the previous revision could answer while the new image was still starting or already crash-looping.

Also fixed: a live production outage this surfaced

worker — the ADR 13 job_outbox drain — has never run. It was hand-created with only FLASK_APP=wsgi, no secrets and no other env vars, and has been crash-looping on missing database_url/redis_url:

F database_url
F   Field required [type=missing, input_value={}, input_type=dict]
F redis_url
F   Field required [type=missing, input_value={}, input_type=dict]

Critically, no CD run would ever have repaired it — the old exists -> update image only shortcut skips config entirely. So the worker/mcp-server steps are restructured to bootstrap once, reconcile image + secrets + env vars every run. Both secret set and --set-env-vars are idempotent and need only Contributor, so reconciling every run costs nothing and makes this workflow the actual source of truth for their config.

Role assignment stays bootstrap-only and now fails with an actionable operator message instead of a raw AuthorizationFailed. CD's service principal holds Contributor, which cannot create role assignments — and it should stay that way: granting the pipeline roleAssignments/write would let anyone able to merge to main hand out Key Vault roles, exactly the escalation CLAUDE.md's least-privilege rule exists to prevent.

Plus a cd-production concurrency group (cancel-in-progress: false) so two merges landing together cannot race and leave the older commit deployed, and a run is never cancelled mid-migration with the database ahead of the deployed image.

Validation

• cd.yml parses; all 11 run: scripts pass bash -n with GitHub expressions stubbed
• The new readiness matcher was executed against the live apps, not just reasoned about. It initially failed: --query "[a,b,c]" -o tsv returns values on separate lines, not tab-separated, so the shell-built status string could never match a case pattern. Now joined in JMESPath (join('/', [...])) and re-verified: backend and frontend → provisioned/runningatmaxscale/true (pass); worker → provisioned/activating/true (correctly keeps waiting)
• Both python invocation forms tested against the real image (above)
• Federated credential and environment change applied and verified out of band

Security / DB / API impact

• Security: removes the manual production approval gate (requested). Adds one Entra federated credential with the same issuer/audience, differing only in the subject format GitHub now sends. CD's Azure privileges are unchanged — deliberately not escalated. No secret values appear in the workflow; all remain keyvaultref/secretref.
• DB: none. The migrate job is untouched.
• API: none.

⚠️ Two things this PR does not do

1. worker is still down. Its managed identity is missing Key Vault Secrets User and Key Vault Crypto User, so its keyvaultref secrets cannot resolve. That grant needs an operator with roleAssignments/write — commands in the PR discussion.
2. CLAUDE.md:63 still says "Gate production deploys on green CI plus manual approval." Left in place by request; the approval half no longer matches this workflow. A comment at the top of cd.yml records that this is deliberate.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.