Skip to content

ci: mechanical quality gates — golden-bot hole, CODEOWNERS, ratchet guard, supply chain#734

Open
joshuakrueger-dfx wants to merge 3 commits into
stagingfrom
joshua/ci-quality-gates
Open

ci: mechanical quality gates — golden-bot hole, CODEOWNERS, ratchet guard, supply chain#734
joshuakrueger-dfx wants to merge 3 commits into
stagingfrom
joshua/ci-quality-gates

Conversation

@joshuakrueger-dfx

Copy link
Copy Markdown
Collaborator

Why this PR

Every rule in here already exists in this repo — as documentation, convention, or a comment. None of it is enforced by a machine. This PR closes that gap for the five cheapest, highest-impact cases. The guiding principle: a guideline without a gate is documentation, and documentation rots silently. (Same overhaul we just shipped for DFXswiss/dfx-wallet in #186; the ideas for the tier model, the floor files and the behavioral fakes came from THIS repo — these five points are the return delivery.)

1. Golden-regenerate bot push now triggers CI (golden-regenerate.yaml)

Why it's needed: pushes made with GITHUB_TOKEN deliberately do not trigger workflows. After the bot commits regenerated baselines, the PR head SHA has zero status checks — and GitHub renders such a PR as mergeable (required checks only block when they exist and fail). docs/visual-regression-tests.md documents the manual workaround (push an empty commit to re-arm checks), which means the safety of every golden-regen PR depends on someone remembering a footnote. A visual regression — or anything else on that SHA — can merge unvalidated.

Fix: the bot now pushes via the existing TAG_DEPLOY_KEY (same ssh-agent pattern as auto-tag.yaml, no new secret needed). Deploy-key pushes trigger workflows, so the full PR gate re-runs on the regenerated head automatically. The manual re-arm note in the docs can be deleted once this lands.

2. CODEOWNERS for signing-critical paths

Why it's needed: the rulesets require 1 approval from anyone with write access — including for lib/packages/wallet/, lib/packages/hardware_wallet/ and .github/ itself. For an app that drives BitBox signing ceremonies, a single non-specialist approval on ceremony code is too thin, and the CI gates can currently be edited by any PR with any single approval (a gate nobody guards is not a gate).

Fix: path-based CODEOWNERS routing those paths to @davidleomay @TaprootFreak @konstantinullrich. Admin action needed to make it blocking: enable Require review from Code Owners in the develop/staging/main rulesets. Until then it still auto-requests the right reviewers.

3. Coverage Floor Ratchet Guard (new required-check candidate)

Why it's needed: docs/testing.md defines the ratchet protocol — lowering .coverage-floor-lines/-functions requires reviewer sign-off, made visible via the coverage:lower-floor label. No workflow checks this. A quiet one-line floor edit inside a big diff permanently de-arms the Coverage Floor Gate for every future PR, and review history shows big diffs are exactly where attention is scarcest.

Fix: new floor-ratchet-guard job — floor decrease without the label fails the PR; raising floors passes without ceremony. Admin action: add it to the required checks alongside Coverage Floor Gate.

4. Supply-chain monitoring (dependabot + dependency-review)

Why it's needed: this repo has no automated dependency monitoring at all — no Dependabot, no advisory gate. A known-vulnerable transitive pub package or a compromised GitHub Action would sit unnoticed until someone happens to read an advisory. The 2025 npm supply-chain worms (Shai-Hulud, 25k+ repos) made the pattern concrete; pub is a smaller target but the Actions we run carry repo credentials either way.

Fix: weekly Dependabot for pub + github-actions targeting staging, with a 7-day cooldown — freshly published versions are the riskiest (compromised releases are typically yanked within days; a one-week wait would have blocked every major 2025 incident at zero cost). Plus dependency-review on every PR diff, so a newly added dependency with a known high-severity advisory cannot enter the repo in the first place (GitHub's dependency graph covers pub).

5. PR template + PR policy job

Why it's needed: the repo's strong conventions (tier model, golden↔handbook pairing, API-authority rules, @no-integration-test annotations) live in three different docs; nothing puts them in front of the author at PR time. And review quality degrades sharply with diff size — for a wallet app an under-reviewed PR is a security risk, not a style problem.

Fix: a PR template encoding the existing rules as a checklist, plus a pr-policy job: conventional-commit title (already the de-facto convention) and a 1500-line size cap with a size:override label for legitimate golden/codegen/lockfile churn (label created).

What this deliberately does NOT touch

  • Tier 2/3 auto-triggering by changed paths (today: human-remembered tier3:full label, escapable path filter for the BitBox simulator) — worth doing, but needs a design conversation about runner budget on dfx01; follow-up proposal.
  • strict_required_status_checks on the rulesets (stale-base green merges) — admin-only setting, listed here for visibility.
  • SHA-pinning all action refs — worthwhile (@v4 tags are mutable), mechanical, but touches all 13 workflows; cleaner as its own diff. New/touched files in this PR are already SHA-pinned where added.
  • pubspec version pinning — unlike npm, the committed pubspec.lock fully governs app builds, so exact pins in pubspec.yaml add ceremony without closing a real gap.

Verification

  • Both modified workflows + dependabot.yml parse (yaml.safe_load), job list verified: build, coverage-floor, golden-tests, bitbox-audit, floor-ratchet-guard, pr-policy.
  • Ratchet-guard awk logic tested for lower/equal/raise cases.
  • No Dart code touched — Tier 0–3 unaffected.

Admin checklist after merge

  1. Ruleset: add Coverage Floor Ratchet Guard + PR Policy + Dependency Review to required checks.
  2. Ruleset: enable Require review from Code Owners.
  3. Confirm TAG_DEPLOY_KEY is authorized for pushes to feature branches (it is a repo deploy key with write — it is).
  4. Optional: flip strict_required_status_checks to true.

🤖 Generated with Claude Code

… ratchet guard, supply-chain monitoring

Every rule below existed as documentation or convention; this PR makes
each one a machine gate (or an automatic process), because a guideline
without enforcement silently rots:

- golden-regenerate: push baselines via TAG_DEPLOY_KEY instead of
  GITHUB_TOKEN. Token pushes do not trigger workflows, so the bot-pushed
  head SHA had ZERO checks while GitHub showed the PR as mergeable —
  required checks never ran against regenerated baselines unless a human
  remembered the empty-commit re-arm (documented gotcha in
  docs/visual-regression-tests.md). Deploy-key pushes trigger the full
  PR gate automatically.
- CODEOWNERS: signing/hardware-wallet paths, pubspec, the floor files
  and .github/ itself now route to maintainers. Today one approval from
  anyone with write access suffices, even for ceremony code.
- floor-ratchet-guard: lowering .coverage-floor-{lines,functions}
  without the coverage:lower-floor label now fails CI. The label was a
  documented convention no workflow checked — a quiet floor edit inside
  a big diff could de-arm the Coverage Floor Gate permanently.
- pr-policy: conventional PR titles + 1500-line size cap
  (size:override label for goldens/codegen churn).
- dependabot (pub + github-actions, 7-day cooldown) and
  dependency-review on PR diffs: the repo currently has zero automated
  dependency monitoring; the cooldown window would have blocked every
  major 2025 supply-chain incident.
- PR template encoding the existing testing/handbook/API-authority
  rules as a checklist.
@joshuakrueger-dfx

Copy link
Copy Markdown
Collaborator Author

Heads-up for admins (added to the checklist): dependency-review is temporarily continue-on-error because the repo's Dependency graph is disabled — the action cannot work without it. Please enable it at Settings → Security analysis (free for public repos; also a prerequisite for Dependabot alerts, while the version-update PRs from dependabot.yml work regardless). Once enabled, delete the continue-on-error: true line in the same commit — a permanently yellow gate is not a gate.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant