fix: retry manifest state writes on contents API 409 conflicts - #261
Merged
Merged
Conversation
Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
joshua-temple
enabled auto-merge (squash)
June 23, 2026 13:58
Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Concurrent per-environment finalize jobs (hotfix/promote) PUT the shared manifest.yaml via the GitHub Contents API at the same blob SHA; the second writer gets HTTP 409 and its env state is dropped. The live fleet caught this on a multi-env hotfix chain (rc v0.3.0-rc.22), where parallel test and staging finalize runs raced and the loser's environment state was never written.
Fix
Adds a shared optimistic-lock read-modify-write in internal/statewrite: fetch the current manifest, re-apply this caller's env-scoped mutation, PUT, and retry on 409 (bounded, backed off) so the loser re-reads the winner's state and both merge. Promote and hotfix finalize both route through it; orchestrate/rollback inherit it via promote. As defense-in-depth, the generated hotfix finalize path uses a per-repo concurrency group so finalize runs queue instead of racing.
Verification
New internal/statewrite package with a stubbed-client test proving a 409-then-success sequence merges both writers' state and that exhaustion surfaces a clean error; merge-semantics and generator concurrency assertions added. go build/test ./..., golangci-lint, and the generator actionlint/generated-bytes tests all green. schema_version unchanged; no manifest field. The stubbed-client unit test is the regression guard (act+gitea cannot reproduce the parallel Contents-API race); the 4env fleet step is the live counterpart.