Repository navigation
Description
Finding
Every module under src/ now has a full StrykerJS pass, and at main @ 9acb046 the survivors are all equivalent:
| set | mutants | killed | timeout | survived | score | wall time (this host) |
|---|---|---|---|---|---|---|
| ownership and merge plugins + their two owner utilities (4 modules) | 1420 | 1266 | 114 | 40 | 97.18% | 112m @ concurrency 14 |
| the ten slash-command handlers, fixed labels and five small utilities (16 modules) | 1269 | 1159 | 26 | 83 | 93.45% | 94m @ 12 |
default npm run test:mutation set (4 merge-decision modules) |
151 | 143 | 2 | 6 | 96.03% | 7m @ 16 |
| dispatcher / cron / command cluster (#592) | 1189 | 1015 | 61 | 113 | 90.50% | 61m @ 12 |
| label writers, config loader and the two label plugins (7 modules) | 1357 | — | — | — | — | 45m @ 12 |
| five reviewer/label plugins (#590) | 718 | — | — | 78 | — | — |
The 129 survivors of today's three runs were each sliced at their exact column span: required: true folds, payload ?. chains already null-checked upstream, debug strings, removeLabels([]) / ownersInclude([]) no-ops, a re-run counter whose only consumer tests it against zero, a right-hand .trim() on a rendered notifier that never has outer whitespace, and a lookup !== undefined guard every caller satisfies. Nothing left to pin.
That clean state was reached through fourteen survivor-pinning issues and PRs (#517, #521, #526, #534, #538, #573–#585, #590, #592, consolidated by #588 and #589). Each of those is the same class of finding — a branch that runs but is never asserted — found by hand-running the harness #572 shipped. Nothing in CI runs it, so the next regression of this kind will only be found when someone next runs it by hand. docs/contributing.md ("Mutation checks") already calls it "a reproducible local and nightly-style check, not a per-push gate".
Proposal
Run the harness on a schedule, non-blocking, as one matrix job per cluster. Sharding is needed because the full tree (~6,100 mutants) took over five hours of wall time at concurrency 12–14 on a 32-core host; a single ubuntu-latest job would exceed GitHub's six-hour job limit.
Two parts:
1. Outside .github/workflows/ (PR-able by this agent once accepted):
stryker.nightly.config.mjsextendingstryker.config.mjswithmutatetaken from aSTRYKER_MUTATEenv var,incremental: false,reporters: ['clear-text', 'json'], andvitest.configFile: 'vitest.nightly.config.mjs'.vitest.nightly.config.mjsextendingvitest.mutation.config.mjsand adding__tests__/workflows.test.ts,__tests__/commands.test.ts, the configurationDocs / docsYamlExamples / dogfoodConfig text-scan suites totest.exclude— Stryker instruments the mutated files, so these suites (which readsrc/as text) fail the dry run whenever the dispatcher or an input's only reader is mutated.- A "Scheduled run" paragraph in
docs/contributing.mdunder "Mutation checks".
2. In .github/workflows/ (needs a human or an ISSUES_PRS_MERGE agent — this agent's App token is minted at the contributor tier and GitHub rejects any push touching .github/workflows/** without the workflows permission):
.github/workflows/mutation.yml:
name: mutation
on:
schedule:
- cron: '17 3 * * 1' # weekly, Monday 03:17 UTC
workflow_dispatch:
permissions:
contents: read
jobs:
stryker:
runs-on: ubuntu-latest
timeout-minutes: 355
strategy:
fail-fast: false
matrix:
include:
- name: merge-gate
mutate: src/labels/*.ts,src/utils/mergeGate.ts,src/utils/labelMatch.ts,src/utils/duration.ts
- name: ownership
mutate: src/plugins/approve*.ts,src/utils/*wners*.ts,src/utils/auth.ts
- name: tide
mutate: src/plugins/tide.ts,src/utils/mergeQueue.ts,src/cronJobs/*.ts
- name: plugins
mutate: src/plugins/blunderbuss.ts,src/plugins/lgtmBinding.ts,src/plugins/ownersLabel.ts,src/plugins/requireMatchingLabel.ts,src/utils/config.ts,src/utils/label*.ts
- name: commands
mutate: src/issueComment/*.ts,src/pullReq/*.ts,src/issues/*.ts,src/utils/command.ts,src/utils/comments.ts,src/utils/events.ts,src/utils/pulls.ts,src/run.ts
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with:
node-version-file: package.json
cache: npm
- run: npm ci
- run: npx stryker run stryker.nightly.config.mjs
env:
STRYKER_MUTATE: ${{ matrix.mutate }}
- uses: actions/upload-artifact@v4
if: always()
with:
name: mutation-${{ matrix.name }}
path: reports/mutation/report.json(actions/upload-artifact should be pinned to a SHA the way the other two actions are; the repository's existing pin for it, if any, wins.)
The run is advisory: thresholds.break stays null, so a survivor never fails the workflow — the JSON artifact is the signal, and the quality lane reads it instead of re-running two-hour probes by hand.
Decision requested
- Accept the scheduled run as proposed (weekly, five shards, advisory), or
- Decline, in which case the quality lane keeps re-running the harness by hand after test-heavy merges and keeps filing survivor pins one issue at a time.
Risk: governance — this adds roughly 5 × 2–3 h of hosted-runner time per week to a CNCF repository, and docs/contributing.md currently frames the harness as local-only.
Decision taken by hive: shard by functional cluster rather than by file count, weekly rather than nightly (the tree changes a few times a week and each run is hours), advisory rather than a break threshold (the docs' stated position).
Priority
- Impact: medium
- Effort: low (the non-workflow half is two config files and a docs paragraph; the workflow half is the text above)
Filed by quality agent (hold-gated mode)
🐝 Hive Agent: quality | Instance: hosted-available-lke648397-260827-5q9t | SHA: 9acb046
— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88
Activity
hivecommons-hive commentedon Oct 27, 2026
What to reply
/hive approve: go with the recommendation/hive decision <your instructions>: go ahead, following your instructions/hive help: show these againIf what this issue waited on has been done (a merged fix, or an action taken outside this repository), a maintainer can reply
/fixedto close it as completed; Hive clears its park labels.Only a maintainer with write access can un-park; everything else is ignored. Full command reference: https://github.com/hivecommons/hive/blob/v5/src/docs/maintainer-commands.md.
hivecommons-hive commentedon Oct 27, 2026
Folded finding. This was filed as a separate issue, but 1 open agent-filed issue (plus 3 closed as completed in the last 14 days) already cite
docs/contributing.md, so it was folded into the oldest of them instead of opening another. Treat this issue as the tracker for that component: prefer one structural fix that covers every folded variant over a PR per variant.[quality] config.ts type-guard and 24h-bound Stryker survivors, and toThrow passing on a nullish throw
Finding
Two assertion-strength gaps that line coverage cannot see (both suites already report 100 % lines for
src/utils/config.ts), found by a full Stryker run over the label/config cluster atmain@ 9acb046 (1357 mutants, 95.36 %, 63 survived; each survivor hand-applied at its exact column span).1. Test infrastructure:
toThrow(message)passes whenundefinedornullis thrownVitest delegates a string/RegExp
toThrowargument to chai'sthrows, and chai compares the message only against a truthy caught value (upstream: vitest-dev/vitest#7260, open). Verified in this checkout:expect(() => { throw undefined }).toThrow('some message')passes under vitest 5.0.5 / chai 6.3.0. The block mutant onmalformedSection(src/utils/config.ts:485,throw malformedSection(key)→throw undefined) therefore survived all seven full-message assertions that reach it, and the suite holds 217 message-formtoThrowassertions that carry the same blind spot.2. Five
src/utils/config.tsguards whose existing cases only exercise one half:472typeof value.color === 'string'→truecolor: 123456is accepted and later sent to GitHub as a number:512typeof missing_label === 'string'→truemissing_labelis accepted:578typeof hold.label === 'string'→truehold.labelis accepted:621ms > maxBlunderbussDelayMs→>=blunderbuss.delay: 24h— the documented maximum — is rejected:342AbortSignal.timeout(5000)argument droppedhttps://config source is fetched without the 5 s timeoutEvidence and provenance
npm run test:coverageonmain@ 9acb046 — 191 files / 2473 tests, 100 / 99.76 / 100 / 100;src/utils/config.tshas no zero-hit line.npm run test:coverage:e2eon the same revision — 90 files / 542 tests, 100 / 99.82 / 100 / 100; no zero-hit line in the file.npx stryker runwith a probe config whosemutateset is the label/config cluster,main@ 9acb046, local run 2026-10-27 (report archived by the quality lane).This is Priority 2 (unit- and e2e-covered, assertion strength only). Per #554 a repository-wide mutation gate is not required; the standing gate question is tracked in #594, so these are pinned individually here.
Recommendation
One pull request, no
src/ordist/change:__tests__/setup.ts: a chaioverwriteMethodguard sotoThrow/.rejects.toThrowfail when the thrown value isundefinedornull;__tests__/setupToThrowGuard.test.tspins the guard and the unchanged negated, Error and string forms__tests__/utils/configTypeGuards.test.ts: yaml-numbercolorrefused (six and two digits) while a quoted string is kept; number/list/mappingmissing_labelrefused; number/list/booleanhold.labelrefused;delay: 24hand1440maccepted while86401sis refused; anhttps://source is fetched withAbortSignal.timeout(5000)and the abort surfaces in the load errordocs/contributing.md: one paragraph naming thetoThrowguard so contributors know why a nullish throw failsEach mutant above, applied by hand at its exact span, fails the new case that names it and passes on
main.Priority
.prow.yamlshould be refused with the documented message, and the 24h bound is documented)Filed by quality agent (hold-gated mode)
🐝 Hive Agent:
quality| Instance:hosted-available-lke648397-260827-5q9t| SHA:da5625e— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88
Folded finding. This was filed as a separate issue, but 1 open agent-filed issue (plus 3 closed as completed in the last 14 days) already cite
docs/contributing.md, so it was folded into the oldest of them instead of opening another. Treat this issue as the tracker for that component: prefer one structural fix that covers every folded variant over a PR per variant.[quality] e2e branch floor (98) does not require sweep.ts's delayed-pass failure arm; drive it and raise the floor to 99
Finding
The bundle acceptance run (
npm run test:coverage:e2e,main@ 9acb046: 90 files, 542 tests, 100 / 99.82 / 100 / 100) reaches everysrc/branch but one that is not a known artifact. Incoverage-e2e/lcov.info:That is the
String(e)arm of the catch in the delayed-reviewers pass (forEachLimited(delayedOnly, …)insrc/cronJobs/sweep.ts:85). The unit run covers it (coverage/lcov.infoBRDA:85,5,1,1,npx vitest run --coverage, same revision), and the candidate pass's equivalent fold at:132is already driven end-to-end by__tests__/bundle/nonErrorRejectionPluginArms.test.ts— but no bundle scenario makes a delayed-only pull request's blunderbuss visit throw, with an Error or otherwise. The remaining e2e misses (approve.ts:504,auth.ts:225) are the negative-count await artifactsdocs/contributing.mdsays not to test, and were excluded from this finding.The two runs' maps are deliberately not combined (
docs/contributing.md, "vite vs. tsc+webpack"); unit and e2e evidence above are reported separately.Gate
The repository already gates this run:
vitest.e2e-coverage.config.mjsfails below 98 % branches, a floor "set just under what main reaches". So this is filed against the gate's configuration rather than as a bare coverage gap: with the one scenario added, the run reaches 1686/1688 branches (99.88 %, the two misses being the documented artifacts) and the e2e branch floor can rise to 99, the same floorvitest.config.mjsalready enforces for the unit run, with ~0.88 pp (≈14 branches) of headroom — the same margin the unit floor has today (99 vs 99.76).Recommendation
__tests__/bundle/blunderbussDelayFailure.test.ts: two delayed-only pull requests; a 500 on#2's timeline read, and#2's review read rejecting with a string vianonErrorRejectionPreload.cjs, are each recorded asblunderbuss: …against#2only,#1still gets reviewers, and the run fails once naming it; a control case with both healthy passesvitest.e2e-coverage.config.mjs:branches: 98→99, with the figure history in the comment extendeddocs/contributing.md: the "Fails below …" sentence fornpm run test:coverage:e2eupdated to 99 % branchesDecision taken by hive: raise the floor to 99 rather than 99.5 — one floor shared with the unit run is simpler to reason about, and 99.5 would leave under 7 branches of headroom.
Priority
Filed by quality agent (hold-gated mode)
🐝 Hive Agent:
quality| Instance:hosted-available-lke648397-260827-5q9t| SHA:8c71ae9— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88
Folded finding. This was filed as a separate issue, but 1 open agent-filed issue (plus 3 closed as completed in the last 14 days) already cite
docs/contributing.md, so it was folded into the oldest of them instead of opening another. Treat this issue as the tracker for that component: prefer one structural fix that covers every folded variant over a PR per variant.[quality] Stryker survivors: prefixed.ts:254 protected guard leaks onto a do-not-merge section; tide.ts:332 position-less enqueued verdict unpinned
Finding
Two Stryker survivors in merge-adjacent code that change an observable effect (a request made or not made; the shape a caller receives), found with a fresh probe over
src/plugins/tide.ts,src/labels/prefixed.ts,src/issueComment/trigger.tsandsrc/labels/fixed.tsat9acb046(1124 mutants, 40 min,coverageAnalysis: perTest, Vitest 5 separator shim active, bundle harness and the src-text-scanning docs tests excluded asdocs/contributing.mddescribes).src/labels/prefixed.ts:254cmd.prefix === ''→truesurvives.requestedLabelsapplies the protected-label refusal (lgtm,hold,approved,do-not-merge/*,hold.label) only to the bare/labeland/remove-labelcommands. A configureddo-not-mergelabel section produces exactly the labels that guard refuses, and is meant to label through its own/do-not-merge <value>command. No test drives such a section, so under the mutant every/do-not-mergevalue is refused withis managed by its own commandand nothing asserts otherwise.src/plugins/tide.ts:332outcome.position === undefined→falsesurvives.evaluateInQueuereturns{ result: 'enqueued' }with nopositionkey when the queue reports no entry.__tests__/plugins/tideMergeQueue.test.ts:102asserts that withtoEqual, which treats a missing key andposition: undefinedalike.Reviewed and not filed (equivalent, or already in the
docs/contributing.mdreviewed-survivors classes): every{ required: true }/optional-chaining-on-payload/present.length === 0mutant;prefixed.ts:54some→every(one protected prefix);trigger.ts:40/44run.status === 'completed'(an in-progress run has no conclusion);trigger.ts:255count++→count--(only=== 0is read, and the mutant stays non-zero whenever the original does);tide.ts:233/320/323/455/472/515/564(debug-only or unreachable arms).fixed.ts:23(the/good-first-issueentry →{}) is reported Survived but actually crashes module load (removeCommandForonundefined) so Vitest runs 0 tests and@stryker-mutator/vitest-runnercounts the empty run as a survival — a runner artifact, not a gap.Coverage evidence: unit line coverage reaches both sites (
npm run test:coverageat9acb046, thresholds pass); the e2e bundle harness (npm run test:coverage:e2e) also reachesprefixed.tslabel paths but drives nodo-not-mergesection either. Both sites are therefore reached by unit and e2e suites but unasserted — a mutation gap, not a line-coverage gap.Recommendation
.prowlabels.yamldo-not-mergesection through/do-not-merge <value>and/remove-do-not-merge <value>(asserting the label POST/DELETE), with/label do-not-merge/...kept as the refused control.tideMergeQueue.test.ts:102totoStrictEqual({ result: 'enqueued' }).Decision taken by hive: a new file
__tests__/label/dynamicProtectedPrefix.test.tsrather than extendingdynamic.test.ts, whose shared fixture has nodo-not-mergesection.Priority
do-not-mergeallowlist)Filed by quality agent (hold-gated mode)
🐝 Hive Agent:
quality| Instance:hosted-available-lke648397-260827-5q9t| SHA:ef4c2b4— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88
Folded finding. This was filed as a separate issue, but 1 open agent-filed issue (plus 3 closed as completed in the last 14 days) already cite
docs/contributing.md, so it was folded into the oldest of them instead of opening another. Treat this issue as the tracker for that component: prefer one structural fix that covers every folded variant over a PR per variant.[quality] Unit suite is test-order dependent: 11 tests in 8 files fail under vitest --sequence.shuffle; CI never shuffles
Finding
The unit suite only passes in source order. On
main@9acb046(Node 26.12.0, Vitest 5):Across 12 seeds, 4 fail (
20261027: 6 tests / 5 files;424242: 2;7: 1;11: 1) — 11 distinct tests in 8 files. The default order and the shuffled bundle suite (__tests__/bundle, 542 tests, seed 1) are green. Every coupling is within one file (Vitest isolates files), and each has one of three causes:__tests__/issueCommentTest/assign.test.ts,unassign.test.ts,handleIssueComment.test.ts(comment.body = …);__tests__/cronJobTest/lgtm.test.ts(listPullReqs[0].labels.push(hold))/assign @some-user; the squash/rebase merge tests got a PR carryingholdand never mergedloadProwConfigprobe chain outliving its test__tests__/utils/utilsErrorPaths.test.ts(explicit-source rejections),__tests__/utils/auth.test.ts(closePolicyAllowsand the review fallback),__tests__/cronJobTest/labelSync.test.tssrc/utils/config.ts:232probes the organization and repository tiers underPromise.all; when the exercised tier rejects, the other tier's sequential probes keep going after the test'sawaitreturned. The next test'sfailTestOnUnhandledRequestsorrequest:startlistener records them.auth.test.ts'safterEach"wait for inFlight 0" is satisfied in the gap between two probes, so the leak chains forward test after test__tests__/utils/auth.test.tsouter describes (configuration cache → inherited atrustedpolicy),__tests__/utils/owners.test.tsexpandOwnersSet(group memo),__tests__/utils/mergeQueueGuards.test.ts(dynamicimport('@actions/core')after a sibling'svi.resetModules()while calling the staticqueueState)core.warningasserted on a different instance than the module warns throughNothing in CI would catch a new instance:
test.ymlrunsnpm run test:coveragein source order only.Recommendation
npm run test:shuffle(vitest run --sequence.shuffle) and autils.configFoundAt(source)helper that ends the sibling tier on its first probe, and document the three patterns under a "Test order" heading indocs/contributing.md— the companion PR..github/workflows/test.ymlchange, which the hive's contributor-tier token cannot push (GitHub refuses workflow edits without theworkflowspermission); it needs a human or an ISSUES_PRS_MERGE agent to land. The exact addition, after thee2e-coveragejob:Decision taken by hive: a separate non-required job rather than replacing
test:coverage's order, so coverage numbers stay comparable run to run and the shuffled job can become required once it has been green for a while.Observed once and not reproduced (3/3 and a full re-run passed with the same seed):
__tests__/utils/ownersGroupPagination.test.ts › authorizes a member on page one once every page has been readfailed under seed424242while another suite was running on the host — a load flake, not an order coupling; noted here so it is recognisable if it recurs.Priority
Filed by quality agent (hold-gated mode)
🐝 Hive Agent:
quality| Instance:hosted-available-lke648397-260827-5q9t| SHA:b7a7f8d— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88
hivecommons-hive commentedon Oct 29, 2026
A merged PR referenced this issue without closing it. Carrying the remainder back here so the issue timeline shows what is still open.
Remainder from the PR body:
the finding was folded onto the scheduled-Stryker gate proposal, which stays open for the maintainers' accept/decline decision; this PR lands the two survivor pins and leaves nothing else of the folded finding behind
This comment is edited in place by the task-list sweep on every cycle; it is not duplicated.
mrbobbytables commentedon Oct 31, 2026
/kind cleanup
mrbobbytables commentedon Oct 31, 2026
Decision: landing the single-shard form in #635 — a
mutationjob on the existing daily schedule (andworkflow_dispatch) running the default merge-decision set throughstryker.nightly.config.mjswithbreak: 95, tally in the step summary,reports/mutation/as an artifact.The whole-tree five-shard matrix is declined for now: 5+ hours of runner time nightly for a tree whose survivors are all equivalent today. If the single shard starts catching regressions, widening
mutateto the next cluster is a one-line change to the nightly config. #635 closes this issue on merge.ci: run the shuffled suite on every pull request, the mutation check …