V2.10.S8: CI pipeline consolidation and cascading gate design

[deeprnd]

Replace the 10 scattered GitHub Actions workflow files with a single ci.yml that implements a 5-stage cascading gate pipeline: quality checks → platform builds → unit tests → integration tests → system + e2e tests. Each stage gates on the previous one using needs: so that failures are caught immediately and downstream jobs are auto-skipped by GitHub Actions.

Leverages the V2.10 substrate: fully-qualified just recipe names (S4), contrib/platform.sh OS/arch detection (S1), setup-* installation recipes (S5), and a declarative build.zig (S2). Strategy matrices eliminate the 712-line duplication currently in tests-short.yml.

Product Outcome

CI behaves as a fast-fail pipeline: a build failure on any platform blocks all testing; a unit test failure blocks integration; an integration failure blocks system and e2e. Developers see the failing stage immediately without waiting for the entire matrix to complete. The pipeline is described in a single file, making it trivial to audit, extend, or add new stages.

User Story

As a developer or CI reviewer, I want the entire CI pipeline to live in one file with clear stage boundaries, so that I can understand the gate chain at a glance, see which stage failed, and trust that downstream stages were never wasted on broken code.

Scope

In Scope

• Consolidation: merge build-fd.yml, build-tk.yml, tests-short.yml, tests-long.yml, tests-xlong.yml, quality.yml, security.yml into a single ci.yml.
• Pipeline stages: 5 stages with needs: chains:
  • Stage 1 — Quality gates: just quality-check-all (format, lint, proto) + just security-check-all (gitleaks, sanitize, seccomp, proof). Runs on Linux once before builds.
  • Stage 2 — Builds: parallel build jobs for each platform/compiler combo using strategy matrix (Linux GCC, Linux Clang, macOS Clang, Windows MSVC/Clang). Each job calls a fully-qualified recipe like just build-fd-linux-x86-gcc.
  • Stage 3 — Unit tests: strategy matrix over os × arch, gated on all builds passing. Calls just test-unit-{fd,tk}-{os}-{arch}.
  • Stage 4 — Integration tests: reduced matrix (Linux x86_64, macOS x86_64, Windows x86_64), gated on all unit tests passing. Calls just test-integration-{fd,tk}-{os}-{arch}.
  • Stage 5 — System + e2e: serial or lightly parallel, gated on integration. Calls just test-system-{fd,tk}, just test-e2e-{fd,tk}, just test-demo-tk.
• Matrix design: use strategy.matrix.include for os/arch/compiler triples to avoid the 712-line duplication. Each platform lane maps to one matrix entry.
• Deletion: after validation, remove build-fd.yml, build-tk.yml, tests-short.yml, tests-long.yml, tests-xlong.yml, quality.yml, security.yml.
• Keep separate: coverage.yml (orthogonal, not part of the cascade), benchmark.yml (benchmarking, not testing), book.yml (docs, unrelated).
• Composite actions: update .github/actions/setup-public-gh-runner/action.yml and .github/actions/deps/action.yml to use setup-* recipes from S5 (coordinated with S6 cutover).
• Documentation: update doc/execution/ci.md to document the new pipeline structure, stage gates, and matrix entries.

Out Of Scope

• Changing just recipe implementations (recipes are consumed as-is from S4).
• Adding new test lanes or removing existing ones (same coverage as today).
• CI scheduling/polling changes (same triggers as existing files, collapsed).
• Windows ARM test lanes beyond what already exists (preserve current matrix, no new platforms).
• Docker/container CI changes (container recipes remain untouched).

Preconditions And Assumptions

• S1 (contrib/platform.sh) has shipped — workflows use platform.sh or just variables for OS/arch instead of runner.os/runner.arch branching.
• S4 (justfile fully-qualified recipe names) has shipped — every build/test/quality/security lane has a recipe like test-unit-tk-linux-x86.
• S5 (setup-* recipes) has shipped — each workflow job calls just setup-{os}-{arch} before building/testing.
• S6 (CI cutover to fully-qualified names) has shipped — all existing lanes use the new naming convention.
• The 7 files to consolidate: build-fd.yml, build-tk.yml, tests-short.yml, tests-long.yml, tests-xlong.yml, quality.yml, security.yml.
• 3 files to keep: coverage.yml, benchmark.yml, book.yml.

Acceptance Criteria

• Given a single PR, when GitHub Actions runs ci.yml, then all 5 stages execute in order: quality → builds → unit → integration → system/e2e.
• Given a build failure on one platform, then unit tests, integration tests, system tests, and e2e tests are all auto-skipped (greyed out with "required by" label).
• Given a unit test failure on one platform, then integration, system, and e2e tests are auto-skipped.
• Given an integration test failure, then system and e2e tests are auto-skipped.
• The ci.yml file uses strategy.matrix.include to define platform/OS/arch/compiler triples — no duplicated job definitions.
• Every run: line in ci.yml calls a fully-qualified recipe (e.g., just test-unit-tk-linux-x86), not a bare dispatcher.
• The old 7 workflow files are deleted after the new ci.yml passes validation.
• coverage.yml, benchmark.yml, and book.yml are unchanged and still functional.
• doc/execution/ci.md documents the 5-stage pipeline, gate logic, and matrix entries.
• All existing CI lanes have a corresponding matrix entry in the new ci.yml — no coverage regression.

Conditional Acceptance

Financial capability and policy

• N/A — CI workflow changes; no policy changes.

Audit and replay

• N/A — CI workflow changes; no audit/replay changes.

Runtime topology and tile ownership

• N/A — CI workflow changes; no topology changes.

Model, tool, adapter, or execution boundary

• N/A — CI workflow changes; no model/adapter/execution changes.

CaseOps API or UI

• N/A — CI workflow changes; no API/UI changes.

Storage role: Memory, Analytics, Ledger

• N/A — CI workflow changes; no storage changes.

Metrics, diagnostics, and operations

• N/A — CI workflow changes; no metrics/telemetry changes.

Security and fail-closed behavior

• Given ci.yml, when a run: line calls a just recipe, then it uses the fully-qualified form.
• Given ci.yml, when quality or security checks fail, then no downstream stage executes (no wasted compute on potentially insecure/broken code).

Evidence Plan

• Demo or command: open a PR that intentionally breaks the build (e.g., a syntax error in a .c file) and verify that only the build stage fails while downstream stages are auto-skipped.
• Demo: open a PR that breaks unit tests on Linux but not macOS, verify that integration/system/e2e are skipped after the Linux unit failure.
• Tests: merge ci.yml and verify all existing CI lanes pass. Then merge a clean PR and verify the full cascade completes in expected time.
• Fixtures or samples: the new ci.yml file and updated doc/execution/ci.md.
• Audit/replay evidence: N/A — CI workflow only.
• Blocked-flow evidence: intentionally failing a stage and verifying auto-skip behavior.

Quality Gate

• ci.yml is a single file under 800 lines (strategy matrices eliminate duplication from the current 1,793-line total across 10 files).
• All 5 stages are visible in the GitHub Actions UI with clear gate transitions.
• Auto-skip behavior works: failed stages show red, downstream stages show grey with "required by".
• No if: success() or if: always() conditions — needs: alone provides the gate logic.
• 7 old workflow files deleted.
• 3 files (coverage.yml, benchmark.yml, book.yml) unchanged.
• All existing CI lanes preserved (no coverage regression).
• doc/execution/ci.md updated.
• Documentation and roadmap status are updated.

Notes And Open Questions

• Quality gates placement: Should quality+security run before or after builds? Running before builds catches formatting/lint issues faster (seconds vs minutes), but some security checks (e.g., seccomp) require built artifacts. Recommendation: format/lint/proto run before builds; security scans (sanitize, gitleaks) run in parallel with builds since they don't require binaries.
• Windows ARM test lanes: Currently exist for unit, integration, and system tests (test-unit-tk-windows-arm, test-integration-tk-windows-arm, test-system-tk-windows-arm). Preserve as-is in the matrix.
• Demo conformance tests: tests-short.yml has 8 demo-conformance jobs (macOS 15/26 × x86/arm, Linux, Windows). These are the most expensive tests (~10-15 min each). In the cascade, they should run as part of Stage 5 (system/e2e) since they require built binaries and are the slowest lane.
• System tests on Windows: Currently tests-xlong.yml has system-tests-windows-x86 and system-tests-windows-arm. These should move to Stage 5 or Stage 4 depending on whether they depend on integration results.
• Matrix granularity: Should fd and tk builds/tests share the same matrix entry (with conditional just calls), or be separate entries? Recommendation: separate entries for clarity — fd lanes and tk lanes have different build artifacts and test fixtures.
• E2E demo tests: test-demo-tk currently runs on macOS only in tests-short.yml. After V2.10, consider if it should run on Linux too (it's a CLI demo, not platform-specific).

Dependencies

• Depends on S1 (platform.sh), S4 (fully-qualified recipe names), S5 (setup-* recipes), S6 (CI cutover to fully-qualified names). These stories establish the substrate that makes the cascade possible — without them, ci.yml would need the same conditional branching that the old files have.
• S6 (CI cutover) and S8 (pipeline consolidation) share the same set of workflow files. S8 can proceed in parallel with S6 as long as the new ci.yml uses the fully-qualified recipe names that S4 produces. S6's acceptance criteria (all existing lanes pass with new names) validates the substrate; S8's acceptance criteria (cascade behavior works) validates the orchestration.
• S2 (build.zig refactor) is independent — the cascade consumes whatever recipe names S4 produces, regardless of how build.zig discovers tests.
• S3 (Qt build POC) is independent — Qt builds are not in scope for S8.
• S7 (evidence/docs closure) depends on S8 since it includes updating all roadmap docs with final pipeline structure.

Implementation Notes

Recommended file structure

.github/workflows/
  ci.yml              ← 5-stage cascade (replaces 7 files)
  coverage.yml        ← unchanged
  benchmark.yml       ← unchanged
  book.yml            ← unchanged

Recommended matrix pattern

strategy:
  matrix:
    include:
      - {platform: linux, arch: x86, compiler: gcc,  runner: ubuntu-24.04}
      - {platform: linux, arch: x86, compiler: clang, runner: ubuntu-24.04}
      - {platform: linux, arch: arm, compiler: gcc,   runner: ubuntu-24.04}
      - {platform: macos, arch: x86, compiler: clang, runner: macos-15}
      - {platform: macos, arch: arm, compiler: clang, runner: macos-15}
      - {platform: windows, arch: x86, compiler: msvc, runner: windows-2025}
      - {platform: windows, arch: arm, compiler: msvc, runner: windows-2025}

Each matrix entry maps to a fully-qualified recipe like just build-fd-{platform}-{arch}-{compiler}. No if: conditions needed — the matrix is the conditional routing.

Stage gate chain

jobs:
  quality-gates:    # Stage 1 — runs once on Linux
  build-linux-gcc:  # Stage 2 — matrix entries
  build-linux-clang:
  build-linux-arm:
  build-macos-x86:
  build-macos-arm:
  build-windows-x86:
  build-windows-arm:
  unit-tests:       # Stage 3 — matrix over os/arch, needs: build-*
  integration-tests:  # Stage 4 — reduced matrix, needs: unit-tests
  system-e2e:        # Stage 5 — serial or light parallel, needs: integration-tests

No needs: on the first stage. Each stage's needs: references all jobs in the previous stage. GitHub Actions auto-skips downstream jobs when any dependency fails.