Skip to content

fix: make workflow generation deterministic - #174

Merged
joshua-temple merged 1 commit into
mainfrom
fix/generator-determinism
Jun 16, 2026
Merged

joshua-temple merged 1 commit into
mainfrom
fix/generator-determinism

Conversation

@joshua-temple

Copy link
Copy Markdown
Collaborator

Problem

Repeated runs of the same manifest produced workflows whose ordering varied run to run: job blocks, needs: [...] entries, and the OR'd contains(fromJSON(...), needs.X.result) checks inside finalize if: expressions. Root cause is Go randomizing map range order per process. DependencyGraph.TopologicalSort seeded its walk by ranging g.Nodes (a map), so the emitted job order, the finalize needs: list, the summary table, the failure-check conditions, and the output aggregation all inherited a random seed. This is cosmetic (jobs run by the needs: DAG, needs is a set, || is commutative) but it makes any committed-vs-regenerated drift gate flaky, including this repo's own Workflow Drift Check.

Fix

Record manifest declaration order while building the graph and seed the topological sort from it instead of from a map range. Declaration order is the most intuitive read for a human (validate, then builds, then deploys, each in the order they appear in the manifest); the DAG still constrains the result wherever real dependencies exist, so only the free ordering is pinned. Two map-seeded lookups in the generator (the release-tag job lookup) and two validation loops (warning/error emission) were switched to the same declaration order so their output is stable as well. No semantics change: only ordering is stabilized. Sets that have no meaningful declaration order (matrix dimensions, dispatch inputs, secrets, finalize outputs) were already alphabetically sorted and are untouched.

Verification

  • go build ./..., go test ./... (1349 pass), go vet ./..., golangci-lint run ./... all green. e2e module builds and vets clean.
  • New determinism test generates a representative 4-environment manifest (matrix image build + dependent bundle build + independent docs build, dependent app deploy + independent sidecar deploy, two external repos) 20 times in one process and asserts every generated file (orchestrate, promote, hotfix, rollback, external-update) is byte-identical across all runs. It fails reliably before the fix (multiple parallel graph roots surface the random seed) and passes after.
  • Ordering-only proof: capturing orchestrate.yaml before and after the fix and diffing sorted lines leaves only two lines differing, the finalize needs: list and the failure-check if:. Tokenizing each of those two lines and sorting shows the member sets are identical; only their order within the line changed. Every other line is byte-identical.
  • Regenerating this repo's dogfood workflows from .github/manifest.yaml produces no change (its manifest is a linear chain whose order was already stable), so the Workflow Drift Check stays green and the determinism test is the coverage for the multi-root case the dogfood manifest does not exercise.

Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
@joshua-temple
joshua-temple merged commit 7164ef5 into main Jun 16, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant