Skip to content

ci: run the shuffled suite on every PR, the mutation check nightly, and guard the Worker's entry points - #635

Merged
mrbobbytables merged 1 commit into
cncf:mainfrom
mrbobbytables:ci/shuffle-nightly-mutation-worker-guard
Oct 31, 2026
Merged

mrbobbytables merged 1 commit into
cncf:mainfrom
mrbobbytables:ci/shuffle-nightly-mutation-worker-guard

Conversation

@mrbobbytables

Copy link
Copy Markdown
Member

Description

Three CI gaps the last month of quality findings kept hitting by hand, closed in one place:

  1. shuffle job — npm run test:shuffle on every pull request (with the Redis service so the store tests are in the order). test(server): settle the abandoned installation-token request so stores.test.ts passes in any order (__tests__/server/stores.test.ts) #633 found an order-poisoning pair in stores.test.ts only because someone ran the shuffle locally; the suite passes in source order by luck and nothing in CI knew.
  2. mutation job — on the daily schedule and workflow_dispatch, runs npm run test:mutation:nightly → stryker.nightly.config.mjs: the default merge-decision set of stryker.config.mjs, no incremental file, break: 95 (main reaches 96 with six equivalent survivors). Per-file tally in the step summary; reports/mutation/ uploaded as an artifact. The schedule trigger already existed and gated nothing extra. This is the single-shard form of [quality] Gate proposal: scheduled Stryker run over the whole src/ tree (five advisory shards) #594; the whole-tree five-shard matrix (5+ h of runner time nightly) is declined for now.
  3. __tests__/worker/entryPoints.test.ts — src/server/worker.ts is excluded from both coverage runs because it only executes inside workerd, so scheduled(), queue() and deferViaQueue shipped untested and no number moved ([quality] worker.ts scheduled(), queue consumer and deferViaQueue run under no test (unit excludes the file; miniflare suite only drives fetch()) #629). The guard parses the file's default-export handlers, Hono routes and top-level functions, requires each to be listed with the evidence in __tests__/worker that drives it, and requires that evidence to be present. A new entry point or a deleted driver fails the unit run. Same pattern as coverageIgnoreHints.test.ts (test(coverage): guard the v8-ignore hints in src/ so each ignored arm keeps the unit test it names #601).

docs/contributing.md records both jobs and the new script.

Fixes #594

Testing

  • npx eslint .github/workflows/test.yml stryker.nightly.config.mjs vitest.config.mjs __tests__/worker/entryPoints.test.ts docs/contributing.md package.json — passed
  • npm run typecheck — passed
  • npx vitest run __tests__/worker/entryPoints.test.ts __tests__/workflows.test.ts __tests__/runtime.test.ts __tests__/releaseWorkflows.test.ts __tests__/configurationDocs.test.ts — 5 files / 139 tests passed
  • Negative check: appending function orphan(): void {} to worker.ts fails lists every entry point with the test that drives it, and no other; reverted
  • npx stryker run stryker.nightly.config.mjs --mutate src/utils/duration.ts — config loads, 32 mutants, exit 1 on 93.75 < 95 as designed (the full default set is at 96); the summary node -e script rendered the table from the resulting report.json
  • npm run test:shuffle — 228 files / 2722 passed, seed 1793408552616
  • The mutation job itself only runs on schedule/dispatch; trigger it with gh workflow run test.yml after merge

Checklist

  • I ran npm run all to lint and build my code — lint and typecheck run on the changed files; no src/ change
  • I ran npm run pack and committed dist/ — no src/ change, dist/ is unaffected
  • Commits are signed off (DCO)
  • Note any new dependencies these changes bring in — none
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generates no new warnings (please describe new warnings if unavoidable)

…nightly, and guard the Worker's entry points

Three gaps the last month of quality findings kept hitting by hand:

- test.yml gains a `shuffle` job running `npm run test:shuffle` (with the
  Redis service, so the store tests are in the order too). cncf#633 found an
  order-poisoning pair only because someone ran the shuffle locally; the
  suite passes in source order by luck and nothing in CI knew.

- test.yml gains a `mutation` job on the daily schedule and on
  workflow_dispatch, running `npm run test:mutation:nightly` —
  stryker.nightly.config.mjs, the default merge-decision set of
  stryker.config.mjs with no incremental file and a `break` score of 95
  (main reaches 96 with six equivalent survivors). The job writes a
  per-file tally to the step summary and uploads reports/mutation/. The
  schedule trigger existed and gated nothing extra. This is the
  single-shard form of cncf#594's proposal; the whole-tree five-shard matrix
  (5+ hours of runner time nightly) is declined for now.

- __tests__/worker/entryPoints.test.ts: src/server/worker.ts is excluded
  from both coverage runs because it only executes inside workerd, so
  scheduled(), queue() and deferViaQueue shipped with no test and no
  number moved (cncf#629). The guard parses the file's default-export
  handlers, Hono routes and top-level functions, requires each to be
  listed with the evidence in __tests__/worker that drives it, and
  requires that evidence to be present — a new entry point or a deleted
  driver fails the unit run. The exclusion comment in vitest.config.mjs
  points at it.

docs/contributing.md records the two jobs and the new script.

Closes cncf#594

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Bob Killen <bkillen@linuxfoundation.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[quality] Gate proposal: scheduled Stryker run over the whole src/ tree (five advisory shards)

1 participant