fix: preserve concurrent external-update slots with re-apply retry

[joshua-temple]

Problem

Concurrent cascade external update runs (multiple upstream artifacts notifying the same primary repo) lost an external slot to a non-fast-forward push. The losing run failed with ! [rejected] main -> main (fetch first) and dropped its slot from the manifest, a silent state loss.

The generated external-update workflow's concurrency: group only serializes runs; a queued run still holds a stale-parent checkout and is rejected on push. The push had no recovery.

Root cause and why a simple retry is not enough

The verb pushed via the plain git.CommitAndPush (no fetch/rebase/retry). Swapping to the existing git.CommitAndPushWithRetry was the obvious fix, but verification proved it insufficient: its retry does git pull --rebase, and two external updates both mutate the same region of the re-marshaled YAML (state.<env>.external), so the rebase hits a textual conflict (UU cascade.yaml), wedges the repo mid-rebase, and after the retries returns an error with no state written.

The merge has to happen at the data-structure level, not the text level: different artifacts write different keys under state.<env>.external, so re-applying the mutation onto the fresh remote manifest merges cleanly.

Fix

internal/external/command.go: runUpdate now validates once up front, then drives the manifest mutation through commitWithApplicationRetry. On a rejected push it fetches the remote tip, reset --hard origin/<branch>, re-reads the manifest fresh, re-applies this update's slot, recommits, and retries (up to 5, short backoff). The mutation closure re-parses the file each attempt so it merges onto rebased remote content. A nothing to commit outcome short-circuits to success (idempotent re-runs).

internal/git/git.go: added CurrentBranch() helper. CommitAndPush and CommitAndPushWithRetry are left untouched for their other callers (promote/hotfix).

internal/generate/external.go: updated the writeConcurrency doc comment to reflect that serialization alone is not sufficient and the verb self-heals via rebase-and-retry.

Tests

• internal/git/retry_test.go: proves CommitAndPushWithRetry recovers from a plain non-fast-forward (non-conflicting case) with both slots preserved.
• internal/external/retry_test.go: drives the real runUpdate through the concurrent race (competitor lands cdk from a stale checkout, this run lands lambda) and asserts both slots survive in origin HEAD. Failing-first verified: on the pre-fix code the push is rejected non-fast-forward. Also covers the sequential stale-checkout case and idempotent re-apply.
• e2e/harness/multi_repo_scenario_test.go: TestMultiRepoRunner_ConcurrentExternalUpdatesPreserveBothSlots drives two real external-update dispatches against one primary under act/gitea and asserts both external slots land in the committed manifest.

E2E limitation: gitea has no live workflow_dispatch API, so two truly simultaneous pushes cannot be staged deterministically. The scenario uses sequential dispatches with two distinct external artifacts, which exercises the same fetch+reset+re-apply path a stale checkout hits. The unit/integration tests cover the conflicting concurrent write directly.

Verification

go build ./...        Success
go test ./...         1391 passed in 23 packages
go test -race ./internal/external/ ./internal/git/   41 passed
go vet ./...          clean
golangci-lint run ./...   clean
e2e: go build ./... + go vet ./...   clean (Docker suite not run)

No generated-workflow drift (the only generator change is a doc comment; generate golden tests stay green).

Residual note

A same-deploy-key concurrent write (two notifications for the same deploy) resolves last-writer-wins after the reset and re-read, which is correct: a duplicate notification should land the latest SHA/version in that one slot.