fix(generate): bind the state-write CAS token to the rendered base blob - #550
Merged
Merged
Conversation
The Contents-API state-write loop rendered manifest content from the freshly reset trunk tip, then separately read the current remote blob sha to satisfy the PUT compare-and-swap. When a sibling component's finalize landed between the render and that read, the sha refreshed to the sibling's blob while the content stayed stale, so the PUT satisfied the CAS with a token decoupled from the content and landed stale bytes as a clean child of the sibling commit, reverting the sibling leaf. Capture the base blob sha with git rev-parse right after the reset, before the content is re-rendered, and use it as the PUT sha. The base blob sha is exactly the value the Contents API sha parameter expects, so a sibling write after the reset makes the current blob differ from the token and GitHub returns 409; the existing retry loop re-fetches, re-renders on the sibling leaf, and converges. One writer wins per round, the loser retries. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The emitted workflow state-write loop (the Contents REST API path in the orchestrate and Release finalize jobs) had a compare-and-swap that was not atomic with the content it wrote. Each attempt did
git reset --hard origin/$BRANCH(base blob B0), rendered the new manifest by patching B0, then in a separate later call readCURRENT_SHAviagh api .../contents?ref=$BRANCH, and PUT the B0-derived content withsha=$CURRENT_SHA.When a concurrent component finalize landed between the render and the
CURRENT_SHAread,CURRENT_SHArefreshed to the sibling's blob while the content stayed stale. The PUT then satisfied the compare-and-swap with a token decoupled from the content's base and landed stale bytes as a clean child of the sibling's commit, reverting the sibling's state leaf. Observed live on a shared-path wave: one component's dev version was reverted while the other advanced. Promote, hotfix, and rollback are unaffected because they finalize through thecascade <verb> finalizeCLI, whose Go CommitWithRetry reads content and sha together (a true CAS).Fix
internal/generate/state_write.go: capture the base blob sha the content is rendered from, right after the reset, withBASE_SHA=$(git rev-parse "origin/$BRANCH:$MANIFEST_FILE"), and PUT withsha=$BASE_SHA. Remove the decoupledCURRENT_SHAread. If a sibling lands after the reset, the current blob no longer matchesBASE_SHA, GitHub returns 409, and the existing retry re-derives onto fresh trunk and converges. Exactly one writer wins per round; the loser retries. A brand-new file (no existing blob) yields an emptyBASE_SHAand omits the sha, creating the file as before. The GitHub Contents API contract is thatshamust equal the current blob sha or the write is rejected 409, and that sha is the git blob shagit rev-parseyields.Verification
Regression test
TestStateWriteBindsCASTokenToBaseBlob(red before, green after) asserts the emitted script capturesBASE_SHAafter the reset, PUTs withsha=$BASE_SHA, and no longer does the decoupledCURRENT_SHAread, for both the orchestrate and Release outputs. Goldens regenerated (orchestrate byte-identical baseline and cascade's own drift-locked orchestrate workflow); diffs limited to the CAS-token binding.go build ./...,go test ./...(2707 pass),go test ./... -race,golangci-lint run ./...all clean; e2e module builds and vets.Caught by the v1-readiness fleet's shared-path concurrency wave.