fix(orchestrate): skip unchanged builds via build-state base ladder - #351
Merged
Merged
Conversation
Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
…nd masking condition Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
joshua-temple
added a commit
to stablekernel/cascade-example-4env
that referenced
this pull request
Jun 25, 2026
Producing a genuinely change-free orchestrate dispatch on the live fleet is unreliable: the dispatch SHA advances past the last finalized envState.SHA, so cascade correctly detects a change and runs the build (verified: the underlying skip behaviour is correct and fixed in stablekernel/cascade#351). The skip is proven deterministically at the right layers (the internal/orchestrate unit tests and the act+gitea scenario 06-no-change-skip). Step 12 keeps the live concurrency-cancel assertions and the no-change run-concludes-success check. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
joshua-temple
added a commit
to stablekernel/cascade-example-4env
that referenced
this pull request
Jun 25, 2026
Producing a genuinely change-free orchestrate dispatch on the live fleet is unreliable: the dispatch SHA advances past the last finalized envState.SHA, so cascade correctly detects a change and runs the build (verified: the underlying skip behaviour is correct and fixed in stablekernel/cascade#351). The skip is proven deterministically at the right layers (the internal/orchestrate unit tests and the act+gitea scenario 06-no-change-skip). Step 12 keeps the live concurrency-cancel assertions and the no-change run-concludes-success check. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
cascade orchestrate decides whether a build callback runs by computing
run_build_<name> = detectChanges(baseSHA, headSHA, triggers). For a build withno dependent deploy carrying its state SHA (every no-environment / library
project, and any build whose deploys do not
depends_onit),calculateBaseSHAsfell back to
defaultBase = HEAD~1.On a genuine no-change re-dispatch (HEAD unchanged since the last orchestrate),
HEAD~1..HEADstill contains the last source commit, sodetectChangesreturnstrue and the build re-runs on every dispatch. This matches a live failure where a
no-change orchestrate cut a prerelease and its
Build (app)callback concludedsuccessinstead of being skipped.The documented intent (docs
Change Detection Algorithm/Per-Deployable Tracking) is that a build is skipped when no trigger files changed since the lastbuild of that build. The old ladder never consulted the build's own last-built
SHA, nor the env-level last-orchestrated SHA.
Fix
calculateBaseSHAsnow resolves the build base SHA in priority order:envState.Builds[name].SHA),envState.SHA) - the key fallbackthat fixes no-env / no-dependent-deploy builds, since after the first
orchestrate at HEAD
envState.SHA == HEAD, so a no-change re-dispatch diffsHEAD..HEAD(empty) and skips,defaultBase(HEAD~1/ initial commit) as the first-run fallback.Finalizenow records the per-build SHA for each successful build intoenvState.Builds[name](SHA,BuiltAt,BuiltBy), mirroring the existingper-deploy recording. This makes ladder step 1 effective on later dispatches.
Deploy base-SHA logic is intentionally left unchanged: the env-level fallback is
scoped to builds only. The build fix is the required one, and the existing deploy
scenarios (the three-env scenario's
deploy-cdkrun/skip steps) assert deployconclusions that a deploy-side fallback could perturb.
Verification
go build ./...: successgo test ./internal/orchestrate/...: 51 passed (includes 3 new tests)go test ./internal/...: 1769 passedgolangci-lint run ./...: no issuescd e2e && go vet ./...: no issuese2e/scenarios/06-no-change-skip.yamlpasses under the real act+giteaharness (build-app skipped on the no-change re-orchestrate).
The bug is isolated by the new unit test
TestSetup_NoChangeReDispatch_SkipsBuild,which builds a real git repo and asserts the build is skipped on a genuine
same-HEAD re-dispatch (RED before the fix, GREEN after), with
TestSetup_NewCommit_RunsBuildas the positive counterpart andTestFinalize_RecordsPerBuildSHAcovering the new state record. The e2e scenariolocks in the end-to-end no-change-skip behavior; it cannot isolate the base-ladder
fix on its own because the harness's finalize state commit always advances HEAD
between consecutive orchestrate steps, so HEAD~1 is the state commit rather than a
source commit. The same-HEAD re-dispatch that triggers the bug is therefore only
expressible at the unit layer.