Skip to content

[quality] Gate proposal: scheduled Stryker run over the whole src/ tree (five advisory shards) #594

Closed

Description

@hivecommons-hive

Finding

Every module under src/ now has a full StrykerJS pass, and at main @ 9acb046 the survivors are all equivalent:

set mutants killed timeout survived score wall time (this host)
ownership and merge plugins + their two owner utilities (4 modules) 1420 1266 114 40 97.18% 112m @ concurrency 14
the ten slash-command handlers, fixed labels and five small utilities (16 modules) 1269 1159 26 83 93.45% 94m @ 12
default npm run test:mutation set (4 merge-decision modules) 151 143 2 6 96.03% 7m @ 16
dispatcher / cron / command cluster (#592) 1189 1015 61 113 90.50% 61m @ 12
label writers, config loader and the two label plugins (7 modules) 1357 — — — — 45m @ 12
five reviewer/label plugins (#590) 718 — — 78 — —

The 129 survivors of today's three runs were each sliced at their exact column span: required: true folds, payload ?. chains already null-checked upstream, debug strings, removeLabels([]) / ownersInclude([]) no-ops, a re-run counter whose only consumer tests it against zero, a right-hand .trim() on a rendered notifier that never has outer whitespace, and a lookup !== undefined guard every caller satisfies. Nothing left to pin.

That clean state was reached through fourteen survivor-pinning issues and PRs (#517, #521, #526, #534, #538, #573–#585, #590, #592, consolidated by #588 and #589). Each of those is the same class of finding — a branch that runs but is never asserted — found by hand-running the harness #572 shipped. Nothing in CI runs it, so the next regression of this kind will only be found when someone next runs it by hand. docs/contributing.md ("Mutation checks") already calls it "a reproducible local and nightly-style check, not a per-push gate".

Proposal

Run the harness on a schedule, non-blocking, as one matrix job per cluster. Sharding is needed because the full tree (~6,100 mutants) took over five hours of wall time at concurrency 12–14 on a 32-core host; a single ubuntu-latest job would exceed GitHub's six-hour job limit.

Two parts:

1. Outside .github/workflows/ (PR-able by this agent once accepted):

  • stryker.nightly.config.mjs extending stryker.config.mjs with mutate taken from a STRYKER_MUTATE env var, incremental: false, reporters: ['clear-text', 'json'], and vitest.configFile: 'vitest.nightly.config.mjs'.
  • vitest.nightly.config.mjs extending vitest.mutation.config.mjs and adding __tests__/workflows.test.ts, __tests__/commands.test.ts, the configurationDocs / docsYamlExamples / dogfoodConfig text-scan suites to test.exclude — Stryker instruments the mutated files, so these suites (which read src/ as text) fail the dry run whenever the dispatcher or an input's only reader is mutated.
  • A "Scheduled run" paragraph in docs/contributing.md under "Mutation checks".

2. In .github/workflows/ (needs a human or an ISSUES_PRS_MERGE agent — this agent's App token is minted at the contributor tier and GitHub rejects any push touching .github/workflows/** without the workflows permission):

.github/workflows/mutation.yml:

name: mutation

on:
  schedule:
    - cron: '17 3 * * 1'   # weekly, Monday 03:17 UTC
  workflow_dispatch:

permissions:
  contents: read

jobs:
  stryker:
    runs-on: ubuntu-latest
    timeout-minutes: 355
    strategy:
      fail-fast: false
      matrix:
        include:
          - name: merge-gate
            mutate: src/labels/*.ts,src/utils/mergeGate.ts,src/utils/labelMatch.ts,src/utils/duration.ts
          - name: ownership
            mutate: src/plugins/approve*.ts,src/utils/*wners*.ts,src/utils/auth.ts
          - name: tide
            mutate: src/plugins/tide.ts,src/utils/mergeQueue.ts,src/cronJobs/*.ts
          - name: plugins
            mutate: src/plugins/blunderbuss.ts,src/plugins/lgtmBinding.ts,src/plugins/ownersLabel.ts,src/plugins/requireMatchingLabel.ts,src/utils/config.ts,src/utils/label*.ts
          - name: commands
            mutate: src/issueComment/*.ts,src/pullReq/*.ts,src/issues/*.ts,src/utils/command.ts,src/utils/comments.ts,src/utils/events.ts,src/utils/pulls.ts,src/run.ts
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
        with:
          node-version-file: package.json
          cache: npm
      - run: npm ci
      - run: npx stryker run stryker.nightly.config.mjs
        env:
          STRYKER_MUTATE: ${{ matrix.mutate }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: mutation-${{ matrix.name }}
          path: reports/mutation/report.json

(actions/upload-artifact should be pinned to a SHA the way the other two actions are; the repository's existing pin for it, if any, wins.)

The run is advisory: thresholds.break stays null, so a survivor never fails the workflow — the JSON artifact is the signal, and the quality lane reads it instead of re-running two-hour probes by hand.

Decision requested

  • Accept the scheduled run as proposed (weekly, five shards, advisory), or
  • Decline, in which case the quality lane keeps re-running the harness by hand after test-heavy merges and keeps filing survivor pins one issue at a time.

Risk: governance — this adds roughly 5 × 2–3 h of hosted-runner time per week to a CNCF repository, and docs/contributing.md currently frames the harness as local-only.

Decision taken by hive: shard by functional cluster rather than by file count, weekly rather than nightly (the tree changes a few times a week and each run is hours), advisory rather than a break threshold (the docs' stated position).

Priority

  • Impact: medium
  • Effort: low (the non-workflow half is two config files and a docs paragraph; the workflow half is the text above)

Filed by quality agent (hold-gated mode)


🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5q9t | SHA: 9acb046

— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

Activity

  1. added
    testingCreated by Hive for agent-filed issue provenance
    agent/qualityCreated by Hive for agent-filed issue provenance
    stage/045-needs-decisionIssue parked on a person: needs-human or a maintainer decision is owed before the hive picks it up
    on Oct 27, 2026
  2. hivecommons-hive commented on Oct 27, 2026

    @hivecommons-hive
    ContributorAuthor

    What to reply

    • /hive approve: go with the recommendation
    • /hive decision <your instructions>: go ahead, following your instructions
    • /hive help: show these again

    If what this issue waited on has been done (a merged fix, or an action taken outside this repository), a maintainer can reply /fixed to close it as completed; Hive clears its park labels.

    Only a maintainer with write access can un-park; everything else is ignored. Full command reference: https://github.com/hivecommons/hive/blob/v5/src/docs/maintainer-commands.md.

  3. hivecommons-hive commented on Oct 27, 2026

    @hivecommons-hive
    ContributorAuthor

    Folded finding. This was filed as a separate issue, but 1 open agent-filed issue (plus 3 closed as completed in the last 14 days) already cite docs/contributing.md, so it was folded into the oldest of them instead of opening another. Treat this issue as the tracker for that component: prefer one structural fix that covers every folded variant over a PR per variant.

    [quality] config.ts type-guard and 24h-bound Stryker survivors, and toThrow passing on a nullish throw

    Finding

    Two assertion-strength gaps that line coverage cannot see (both suites already report 100 % lines for src/utils/config.ts), found by a full Stryker run over the label/config cluster at main @ 9acb046 (1357 mutants, 95.36 %, 63 survived; each survivor hand-applied at its exact column span).

    1. Test infrastructure: toThrow(message) passes when undefined or null is thrown

    Vitest delegates a string/RegExp toThrow argument to chai's throws, and chai compares the message only against a truthy caught value (upstream: vitest-dev/vitest#7260, open). Verified in this checkout: expect(() => { throw undefined }).toThrow('some message') passes under vitest 5.0.5 / chai 6.3.0. The block mutant on malformedSection (src/utils/config.ts:485, throw malformedSection(key) → throw undefined) therefore survived all seven full-message assertions that reach it, and the suite holds 217 message-form toThrow assertions that carry the same blind spot.

    2. Five src/utils/config.ts guards whose existing cases only exercise one half

    site surviving mutant what it changes why no test notices
    :472 typeof value.color === 'string' → true a yaml-number color: 123456 is accepted and later sent to GitHub as a number only the emptiness/missing-color half is tested
    :512 typeof missing_label === 'string' → true a list or mapping missing_label is accepted only the empty-string half is tested
    :578 typeof hold.label === 'string' → true a numeric or boolean hold.label is accepted only the empty-string half is tested
    :621 ms > maxBlunderbussDelayMs → >= blunderbuss.delay: 24h — the documented maximum — is rejected the bound is tested only well above and well below 24h
    :342 AbortSignal.timeout(5000) argument dropped an https:// config source is fetched without the 5 s timeout the https path is tested with an immediate response

    Evidence and provenance

    • Unit: npm run test:coverage on main @ 9acb046 — 191 files / 2473 tests, 100 / 99.76 / 100 / 100; src/utils/config.ts has no zero-hit line.
    • End-to-end: npm run test:coverage:e2e on the same revision — 90 files / 542 tests, 100 / 99.82 / 100 / 100; no zero-hit line in the file.
    • Mutation: npx stryker run with a probe config whose mutate set is the label/config cluster, main @ 9acb046, local run 2026-10-27 (report archived by the quality lane).

    This is Priority 2 (unit- and e2e-covered, assertion strength only). Per #554 a repository-wide mutation gate is not required; the standing gate question is tracked in #594, so these are pinned individually here.

    Recommendation

    One pull request, no src/ or dist/ change:

    • __tests__/setup.ts: a chai overwriteMethod guard so toThrow / .rejects.toThrow fail when the thrown value is undefined or null; __tests__/setupToThrowGuard.test.ts pins the guard and the unchanged negated, Error and string forms
    • __tests__/utils/configTypeGuards.test.ts: yaml-number color refused (six and two digits) while a quoted string is kept; number/list/mapping missing_label refused; number/list/boolean hold.label refused; delay: 24h and 1440m accepted while 86401s is refused; an https:// source is fetched with AbortSignal.timeout(5000) and the abort surfaces in the load error
    • docs/contributing.md: one paragraph naming the toThrow guard so contributors know why a nullish throw fails

    Each mutant above, applied by hand at its exact span, fails the new case that names it and passes on main.

    Priority

    • Impact: medium (config validation is user-facing: a bad .prow.yaml should be refused with the documented message, and the 24h bound is documented)
    • Effort: low

    Filed by quality agent (hold-gated mode)


    🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5q9t | SHA: da5625e

    — hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88


    Folded finding. This was filed as a separate issue, but 1 open agent-filed issue (plus 3 closed as completed in the last 14 days) already cite docs/contributing.md, so it was folded into the oldest of them instead of opening another. Treat this issue as the tracker for that component: prefer one structural fix that covers every folded variant over a PR per variant.

    [quality] e2e branch floor (98) does not require sweep.ts's delayed-pass failure arm; drive it and raise the floor to 99

    Finding

    The bundle acceptance run (npm run test:coverage:e2e, main @ 9acb046: 90 files, 542 tests, 100 / 99.82 / 100 / 100) reaches every src/ branch but one that is not a known artifact. In coverage-e2e/lcov.info:

    SF:src/cronJobs/sweep.ts BRDA:85,5,1,0
    

    That is the String(e) arm of the catch in the delayed-reviewers pass (forEachLimited(delayedOnly, …) in src/cronJobs/sweep.ts:85). The unit run covers it (coverage/lcov.info BRDA:85,5,1,1, npx vitest run --coverage, same revision), and the candidate pass's equivalent fold at :132 is already driven end-to-end by __tests__/bundle/nonErrorRejectionPluginArms.test.ts — but no bundle scenario makes a delayed-only pull request's blunderbuss visit throw, with an Error or otherwise. The remaining e2e misses (approve.ts:504, auth.ts:225) are the negative-count await artifacts docs/contributing.md says not to test, and were excluded from this finding.

    The two runs' maps are deliberately not combined (docs/contributing.md, "vite vs. tsc+webpack"); unit and e2e evidence above are reported separately.

    Gate

    The repository already gates this run: vitest.e2e-coverage.config.mjs fails below 98 % branches, a floor "set just under what main reaches". So this is filed against the gate's configuration rather than as a bare coverage gap: with the one scenario added, the run reaches 1686/1688 branches (99.88 %, the two misses being the documented artifacts) and the e2e branch floor can rise to 99, the same floor vitest.config.mjs already enforces for the unit run, with ~0.88 pp (≈14 branches) of headroom — the same margin the unit floor has today (99 vs 99.76).

    Recommendation

    • __tests__/bundle/blunderbussDelayFailure.test.ts: two delayed-only pull requests; a 500 on #2's timeline read, and #2's review read rejecting with a string via nonErrorRejectionPreload.cjs, are each recorded as blunderbuss: … against #2 only, #1 still gets reviewers, and the run fails once naming it; a control case with both healthy passes
    • vitest.e2e-coverage.config.mjs: branches: 98 → 99, with the figure history in the comment extended
    • docs/contributing.md: the "Fails below …" sentence for npm run test:coverage:e2e updated to 99 % branches

    Decision taken by hive: raise the floor to 99 rather than 99.5 — one floor shared with the unit run is simpler to reason about, and 99.5 would leave under 7 branches of headroom.

    Priority

    • Impact: medium (coverage-reporting: the e2e floor could not notice a delayed-pass regression; the fold itself is unit-covered)
    • Effort: low

    Filed by quality agent (hold-gated mode)


    🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5q9t | SHA: 8c71ae9

    — hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88


    Folded finding. This was filed as a separate issue, but 1 open agent-filed issue (plus 3 closed as completed in the last 14 days) already cite docs/contributing.md, so it was folded into the oldest of them instead of opening another. Treat this issue as the tracker for that component: prefer one structural fix that covers every folded variant over a PR per variant.

    [quality] Stryker survivors: prefixed.ts:254 protected guard leaks onto a do-not-merge section; tide.ts:332 position-less enqueued verdict unpinned

    Finding

    Two Stryker survivors in merge-adjacent code that change an observable effect (a request made or not made; the shape a caller receives), found with a fresh probe over src/plugins/tide.ts, src/labels/prefixed.ts, src/issueComment/trigger.ts and src/labels/fixed.ts at 9acb046 (1124 mutants, 40 min, coverageAnalysis: perTest, Vitest 5 separator shim active, bundle harness and the src-text-scanning docs tests excluded as docs/contributing.md describes).

    1. src/labels/prefixed.ts:254 cmd.prefix === '' → true survives. requestedLabels applies the protected-label refusal (lgtm, hold, approved, do-not-merge/*, hold.label) only to the bare /label and /remove-label commands. A configured do-not-merge label section produces exactly the labels that guard refuses, and is meant to label through its own /do-not-merge <value> command. No test drives such a section, so under the mutant every /do-not-merge value is refused with is managed by its own command and nothing asserts otherwise.
    2. src/plugins/tide.ts:332 outcome.position === undefined → false survives. evaluateInQueue returns { result: 'enqueued' } with no position key when the queue reports no entry. __tests__/plugins/tideMergeQueue.test.ts:102 asserts that with toEqual, which treats a missing key and position: undefined alike.

    Reviewed and not filed (equivalent, or already in the docs/contributing.md reviewed-survivors classes): every { required: true }/optional-chaining-on-payload/present.length === 0 mutant; prefixed.ts:54 some→every (one protected prefix); trigger.ts:40/44 run.status === 'completed' (an in-progress run has no conclusion); trigger.ts:255 count++→count-- (only === 0 is read, and the mutant stays non-zero whenever the original does); tide.ts:233/320/323/455/472/515/564 (debug-only or unreachable arms). fixed.ts:23 (the /good-first-issue entry → {}) is reported Survived but actually crashes module load (removeCommandFor on undefined) so Vitest runs 0 tests and @stryker-mutator/vitest-runner counts the empty run as a survival — a runner artifact, not a gap.

    Coverage evidence: unit line coverage reaches both sites (npm run test:coverage at 9acb046, thresholds pass); the e2e bundle harness (npm run test:coverage:e2e) also reaches prefixed.ts label paths but drives no do-not-merge section either. Both sites are therefore reached by unit and e2e suites but unasserted — a mutation gap, not a line-coverage gap.

    Recommendation

    • Add a test driving a .prowlabels.yaml do-not-merge section through /do-not-merge <value> and /remove-do-not-merge <value> (asserting the label POST/DELETE), with /label do-not-merge/... kept as the refused control.
    • Change tideMergeQueue.test.ts:102 to toStrictEqual({ result: 'enqueued' }).

    Decision taken by hive: a new file __tests__/label/dynamicProtectedPrefix.test.ts rather than extending dynamic.test.ts, whose shared fixture has no do-not-merge section.

    Priority

    • Impact: medium (a regression that applied the guard to prefixed commands would silently refuse a repository's own do-not-merge allowlist)
    • Effort: low

    Filed by quality agent (hold-gated mode)


    🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5q9t | SHA: ef4c2b4

    — hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88


    Folded finding. This was filed as a separate issue, but 1 open agent-filed issue (plus 3 closed as completed in the last 14 days) already cite docs/contributing.md, so it was folded into the oldest of them instead of opening another. Treat this issue as the tracker for that component: prefer one structural fix that covers every folded variant over a PR per variant.

    [quality] Unit suite is test-order dependent: 11 tests in 8 files fail under vitest --sequence.shuffle; CI never shuffles

    Finding

    The unit suite only passes in source order. On main@9acb046 (Node 26.12.0, Vitest 5):

    npx vitest run --sequence.shuffle --sequence.seed=20261027 --exclude '__tests__/bundle/**'
    #  Test Files  5 failed | 96 passed (101)
    #       Tests  6 failed | 1925 passed | 75 skipped (2006)

    Across 12 seeds, 4 fail (20261027: 6 tests / 5 files; 424242: 2; 7: 1; 11: 1) — 11 distinct tests in 8 files. The default order and the shuffled bundle suite (__tests__/bundle, 542 tests, seed 1) are green. Every coupling is within one file (Vitest isolates files), and each has one of three causes:

    Cause Where What the neighbour saw
    A shared imported JSON fixture mutated and never restored __tests__/issueCommentTest/assign.test.ts, unassign.test.ts, handleIssueComment.test.ts (comment.body = …); __tests__/cronJobTest/lgtm.test.ts (listPullReqs[0].labels.push(hold)) the "self assign" test dispatched /assign @some-user; the squash/rebase merge tests got a PR carrying hold and never merged
    A loadProwConfig probe chain outliving its test __tests__/utils/utilsErrorPaths.test.ts (explicit-source rejections), __tests__/utils/auth.test.ts (closePolicyAllows and the review fallback), __tests__/cronJobTest/labelSync.test.ts src/utils/config.ts:232 probes the organization and repository tiers under Promise.all; when the exercised tier rejects, the other tier's sequential probes keep going after the test's await returned. The next test's failTestOnUnhandledRequests or request:start listener records them. auth.test.ts's afterEach "wait for inFlight 0" is satisfied in the gap between two probes, so the leak chains forward test after test
    A module memo not reset __tests__/utils/auth.test.ts outer describes (configuration cache → inherited a trusted policy), __tests__/utils/owners.test.ts expandOwnersSet (group memo), __tests__/utils/mergeQueueGuards.test.ts (dynamic import('@actions/core') after a sibling's vi.resetModules() while calling the static queueState) wrong refusal message; stale team members; core.warning asserted on a different instance than the module warns through

    Nothing in CI would catch a new instance: test.yml runs npm run test:coverage in source order only.

    Recommendation

    • Fix the eleven couplings (tests only), add npm run test:shuffle (vitest run --sequence.shuffle) and a utils.configFoundAt(source) helper that ends the sibling tier on its first probe, and document the three patterns under a "Test order" heading in docs/contributing.md — the companion PR.
    • Add a shuffled run to CI so a new coupling fails the push rather than a later reorder. This is a .github/workflows/test.yml change, which the hive's contributor-tier token cannot push (GitHub refuses workflow edits without the workflows permission); it needs a human or an ISSUES_PRS_MERGE agent to land. The exact addition, after the e2e-coverage job:
      # Order independence: the unit suite with every file's tests shuffled
      # (npm run test:shuffle, see docs/contributing.md "Test order"). The seed
      # is printed in the run log; repeat an order with --sequence.seed=<n>.
      shuffle:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
          - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
            with:
              node-version-file: package.json
              cache: npm
          - run: npm ci
          - run: npm run test:shuffle -- --exclude '__tests__/bundle/**'

    Decision taken by hive: a separate non-required job rather than replacing test:coverage's order, so coverage numbers stay comparable run to run and the shuffled job can become required once it has been green for a while.

    Observed once and not reproduced (3/3 and a full re-run passed with the same seed): __tests__/utils/ownersGroupPagination.test.ts › authorizes a member on page one once every page has been read failed under seed 424242 while another suite was running on the host — a load flake, not an order coupling; noted here so it is recognisable if it recurs.

    Priority

    • Impact: medium — a green suite that depends on source order hides coupling; the next test added to one of these files can go red for reasons unrelated to its own change
    • Effort: low (tests only; the workflow job is eleven lines)

    Filed by quality agent (hold-gated mode)


    🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5q9t | SHA: b7a7f8d

    — hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

  4. hivecommons-hive commented on Oct 29, 2026

    @hivecommons-hive
    ContributorAuthor

    A merged PR referenced this issue without closing it. Carrying the remainder back here so the issue timeline shows what is still open.

    Remainder from the PR body:

    the finding was folded onto the scheduled-Stryker gate proposal, which stays open for the maintainers' accept/decline decision; this PR lands the two survivor pins and leaves nothing else of the folded finding behind

    This comment is edited in place by the task-list sweep on every cycle; it is not duplicated.

  5. mrbobbytables commented on Oct 31, 2026

    @mrbobbytables
    Member

    /kind cleanup

  6. added
    kind/cleanupCategorizes issue or PR as related to cleaning up code, process, or technical debt.
    and removed on Oct 31, 2026
  7. mrbobbytables commented on Oct 31, 2026

    @mrbobbytables
    Member

    Decision: landing the single-shard form in #635 — a mutation job on the existing daily schedule (and workflow_dispatch) running the default merge-decision set through stryker.nightly.config.mjs with break: 95, tally in the step summary, reports/mutation/ as an artifact.

    The whole-tree five-shard matrix is declined for now: 5+ hours of runner time nightly for a tree whose survivors are all equivalent today. If the single shard starts catching regressions, widening mutate to the next cluster is a one-line change to the nightly config. #635 closes this issue on merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/qualityCreated by Hive for agent-filed issue provenancekind/cleanupCategorizes issue or PR as related to cleaning up code, process, or technical debt.testingCreated by Hive for agent-filed issue provenance

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions