Skip to content

ci: add fail-closed validation aggregates - #551

Merged
codeforester merged 4 commits into
mainfrom
ci/540-20261006-enforce-validation-gates
Oct 7, 2026
Merged

codeforester merged 4 commits into
mainfrom
ci/540-20261006-enforce-validation-gates

Conversation

@codeforester

@codeforester codeforester commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add stable, fail-closed merge-policy contexts for the framework validation contract.

Issue

Refs #540

Validation

  • tests/ci-policy-contract.sh
  • tests/quality-contract.sh
  • shellcheck --severity=warning tests/ci-policy-contract.sh tests/validate.sh tests/lint-warnings.sh
  • ./tests/validate.sh reached all 708 BATS tests and the Python contract suite successfully; the later artifact-contract fixture failed because the sandboxed api.github.test endpoint reset its connection.
  • Hosted actionlint remains to be verified by GitHub Actions; Docker is unavailable in this environment.

Demo Impact

None.

Docs Impact

Updated docs/ci-policy.md with the aggregate contexts, coverage, fail-closed behavior, accepted same-workflow trust boundary, and post-merge ruleset readback procedure.

API Impact

None.

CI Impact

Product validation aggregates platform, minimum-runtime, compatibility, release, and Beacon lanes. Quality contract aggregates the quality lane. Both run with always() and require every dependency to report exactly success. Both are intended to become required after the post-merge ruleset readback.

Security Notes

No additional permissions; aggregate jobs remain contents: read.

Notes

This PR intentionally does not mutate live branch protection. After merge, add Product validation and Quality contract with Actions integration ID 15368 to the effective ruleset and read back the existing controls, per the documented procedure.

@codeforester

Copy link
Copy Markdown
Collaborator Author

Review at 8da3ea6. The aggregates are correct: always() plus an exact == success comparison means a skipped, cancelled or failed dependency cannot leave a green context. Both workflows have no path or branch filters on pull_request, so the required contexts will always report. All five product lanes are covered, and hosted CI is green. No blockers. Notes, in rough priority order:

  1. Fixes #540 will close the issue before its acceptance criteria are met. The PR deliberately does not touch the live ruleset, so these criteria stay open after merge: "Exact contexts and Actions integration identity are verified in effective live rulesets" and "Re-running Base repository configuration preserves the controls" (which depends on base#2451). Consider Refs #540 (or Part of) and closing the issue after the ruleset readback is recorded.
  2. The contract test does not pin all the comparisons. tests/ci-policy-contract.sh checks that all five *_RESULT env vars are wired, but only asserts the == success comparison for VALIDATE_RESULT and DOWNSTREAM_DEMO_RESULT. Someone could drop the Bash 4.2, representative-matrix or release-gates comparison and the test would still pass, leaving that lane silently non-blocking. Adding the other three greps closes this. Also, nothing forces a new job added to tests.yml into the needs: list. A check that every job ID other than product-validation appears in the aggregate's needs: would catch that.
  3. The aggregate is editable by the PRs it gates. It lives in the same pull_request workflow, and ci-policy-contract.sh is in the same diff, so a PR can weaken both together. For a solo-maintainer repo with same-repo PRs this is probably an acceptable trade-off. It is weaker than the trusted pull_request_target approach used for the demo gate in base-bash-libs-demo#45. Worth one sentence in docs/ci-policy.md stating the accepted scope.
  4. Docs read as both current and target state. The "Default-branch baseline" bullets describe today's live ruleset and the following paragraph describes the intended one. The "not yet merge-blocking" sentence is easy to miss. Consider labeling the new section "Planned" until the readback is done, or adding a "ruleset updated on " line after.
  5. Readback detail is missing. The post-merge procedure doesn't say how to read back the ruleset. Including the gh api repos/.../rulesets and .../branches/main/protection commands would make the readback repeatable. Minor.

Also noting that the PR description says the full local ./tests/validate.sh hit a sandbox network reset in the artifact fixture. Hosted Validate (ubuntu-24.04) and (macos-14) are green, which covers it.

@codeforester

Copy link
Copy Markdown
Collaborator Author

Review follow-up applied in b6d158c and 443ab43:

  • changed the PR to Refs #540 so the issue remains open until live ruleset acceptance criteria are satisfied
  • strengthened tests/ci-policy-contract.sh to require == success for all five product lanes and to fail if any top-level tests job is omitted from the aggregate needs: list
  • documented the accepted same-pull_request workflow trust boundary
  • separated current default-branch policy from planned aggregate enforcement and added repeatable ruleset/branch-protection readback commands
  • corrected the contract script to satisfy the repository shfmt profile

Validation:

  • focused policy, quality, Bash syntax, shellcheck, and shfmt checks pass
  • local ./tests/validate.sh completed all 708 BATS tests and the Python suite; its later artifact fixture hit the known sandbox api.github.test connection reset
  • hosted Quality is green; hosted Tests is still running its macOS Validate lane. The PR remains open and unmerged.

@codeforester
codeforester merged commit f72059c into main Oct 7, 2026
12 checks passed
@codeforester
codeforester deleted the ci/540-20261006-enforce-validation-gates branch October 7, 2026 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant