Skip to content

fix(orchestrate): skip unchanged builds via build-state base ladder - #351

Merged
joshua-temple merged 2 commits into
mainfrom
fix/no-change-skip-investigate
Jun 25, 2026
Merged

joshua-temple merged 2 commits into
mainfrom
fix/no-change-skip-investigate

Conversation

@joshua-temple

Copy link
Copy Markdown
Collaborator

Problem

cascade orchestrate decides whether a build callback runs by computing
run_build_<name> = detectChanges(baseSHA, headSHA, triggers). For a build with
no dependent deploy carrying its state SHA (every no-environment / library
project, and any build whose deploys do not depends_on it), calculateBaseSHAs
fell back to defaultBase = HEAD~1.

On a genuine no-change re-dispatch (HEAD unchanged since the last orchestrate),
HEAD~1..HEAD still contains the last source commit, so detectChanges returns
true and the build re-runs on every dispatch. This matches a live failure where a
no-change orchestrate cut a prerelease and its Build (app) callback concluded
success instead of being skipped.

The documented intent (docs Change Detection Algorithm / Per-Deployable Tracking) is that a build is skipped when no trigger files changed since the last
build of that build. The old ladder never consulted the build's own last-built
SHA, nor the env-level last-orchestrated SHA.

Fix

  • calculateBaseSHAs now resolves the build base SHA in priority order:
    1. the build's own recorded state SHA (envState.Builds[name].SHA),
    2. else the dependent-deploy state SHA (unchanged from before),
    3. else the env-level last-orchestrated SHA (envState.SHA) - the key fallback
      that fixes no-env / no-dependent-deploy builds, since after the first
      orchestrate at HEAD envState.SHA == HEAD, so a no-change re-dispatch diffs
      HEAD..HEAD (empty) and skips,
    4. else defaultBase (HEAD~1 / initial commit) as the first-run fallback.
  • Finalize now records the per-build SHA for each successful build into
    envState.Builds[name] (SHA, BuiltAt, BuiltBy), mirroring the existing
    per-deploy recording. This makes ladder step 1 effective on later dispatches.

Deploy base-SHA logic is intentionally left unchanged: the env-level fallback is
scoped to builds only. The build fix is the required one, and the existing deploy
scenarios (the three-env scenario's deploy-cdk run/skip steps) assert deploy
conclusions that a deploy-side fallback could perturb.

Verification

  • go build ./...: success
  • go test ./internal/orchestrate/...: 51 passed (includes 3 new tests)
  • go test ./internal/...: 1769 passed
  • golangci-lint run ./...: no issues
  • cd e2e && go vet ./...: no issues
  • New e2e/scenarios/06-no-change-skip.yaml passes under the real act+gitea
    harness (build-app skipped on the no-change re-orchestrate).

The bug is isolated by the new unit test TestSetup_NoChangeReDispatch_SkipsBuild,
which builds a real git repo and asserts the build is skipped on a genuine
same-HEAD re-dispatch (RED before the fix, GREEN after), with
TestSetup_NewCommit_RunsBuild as the positive counterpart and
TestFinalize_RecordsPerBuildSHA covering the new state record. The e2e scenario
locks in the end-to-end no-change-skip behavior; it cannot isolate the base-ladder
fix on its own because the harness's finalize state commit always advances HEAD
between consecutive orchestrate steps, so HEAD~1 is the state commit rather than a
source commit. The same-HEAD re-dispatch that triggers the bug is therefore only
expressible at the unit layer.

Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
…nd masking condition

Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
@joshua-temple
joshua-temple merged commit b88c57e into main Jun 25, 2026
14 of 16 checks passed
@joshua-temple
joshua-temple deleted the fix/no-change-skip-investigate branch June 25, 2026 15:20
joshua-temple added a commit to stablekernel/cascade-example-4env that referenced this pull request Jun 25, 2026
Producing a genuinely change-free orchestrate dispatch on the live fleet is unreliable: the dispatch SHA advances past the last finalized envState.SHA, so cascade correctly detects a change and runs the build (verified: the underlying skip behaviour is correct and fixed in stablekernel/cascade#351). The skip is proven deterministically at the right layers (the internal/orchestrate unit tests and the act+gitea scenario 06-no-change-skip). Step 12 keeps the live concurrency-cancel assertions and the no-change run-concludes-success check.

Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
joshua-temple added a commit to stablekernel/cascade-example-4env that referenced this pull request Jun 25, 2026
Producing a genuinely change-free orchestrate dispatch on the live fleet is unreliable: the dispatch SHA advances past the last finalized envState.SHA, so cascade correctly detects a change and runs the build (verified: the underlying skip behaviour is correct and fixed in stablekernel/cascade#351). The skip is proven deterministically at the right layers (the internal/orchestrate unit tests and the act+gitea scenario 06-no-change-skip). Step 12 keeps the live concurrency-cancel assertions and the no-change run-concludes-success check.

Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant