[napetrov]

AGENTS.md asks authors to break the code each test covers and confirm that the test fails. Until now that happened only by hand. CI gates coverage of changed lines, but not whether a test notices when one of those lines breaks. mutation.yml runs mutmut weekly, on three modules only.

Change

• dokimi/scripts/mutate_changed.py works through these steps:

  1. It takes the changed lines from git diff -U0 BASE...HEAD over dokimi/dokimi/**.py.
  2. Candidate tests are the test files that import a changed module, with files named after the module first.
  3. A coverage pass with per-test contexts (--cov-context=test, COVERAGE_CORE=ctrace) runs one pytest process per file and stops at half of --budget. It maps each changed line to the tests that execute it.
  4. A small AST mutator builds mutants on changed lines only: flip a comparison, swap and/or, negate an if/while condition, return None, flip a bool, bump an int, swap +/-, drop a not. Each mutant is spliced into the source text, so the rest of the file is untouched. At most 40 are made, spread round-robin over the changed lines.
  5. Each mutant runs only its covering tests, with -x, in a temp copy of the tracked tree, so subprocess tests see the mutant too. If pytest exits 0, the mutant survived.

  Tests that fail without a mutant are left out and listed in the report. The output is a markdown summary, a JSON file, and a ::warning annotation per survivor. --fail-on-survivor turns those into ::error annotations and exits 1.

• ci.yml has a new job, mutation-diff: pull_request only, after static, py3.13. The step stops at 22 min (no new mutant after 900 s, at most 120 s per mutant) and the job at 30. The summary is written in its own !cancelled() step, so it survives a step timeout. Warn-only for now.

• make ci-local runs the job, because ci_local.py reads the workflows (checked: ci_local.py run --jobs mutation-diff). To leave it out locally, pass a --jobs list without it.

• AGENTS.md documents the job next to the by-hand step. test_ci_label_triggers: a push runs every PR job except mutation-diff, which has no base to diff against on a push. .gitignore covers its output files.

• The weekly mutmut pilot stays as is. It scores whole core modules; this job looks only at what a PR changes.

Measured on 6 recent merged PRs

Run locally (Windows, Python 3.15, serial) at each PR's merge commit against its parent. A GitHub Linux runner should be faster.

PR	changed pkg lines	candidate test files (not run: out of time)	mutants run	killed	survived	wall
#721	4	0	0 (no mutable operator)	–	–	0.1 s
#641	31	21 (11)	16	14	2	1125 s
#635	9	18 (14)	7	6	1	781 s
#634	3	1	1	1	0	35 s
#665	6	8	1	1	0	398 s
#666	18	48 (31)	6	6	0	715 s

In total 31 mutants ran: 28 were killed and 3 survived. All 3 survivors are equivalent or cosmetic:

• zip(..., strict=True) → False, on inputs whose lengths are already checked (stats.py:403);
• and → or in an overflow fallback that returns the same mean on normal input (stats.py:167);
• sha[:12] → [:13] in an error message (workspace.py:665).

Since every survivor was noise, the job starts warn-only. Two more things showed up while measuring:

• pytest-cov's per-test contexts break under xdist (1338 setup errors), so the coverage pass is serial and bounded per file.
• On Python 3.14+, the default sys.monitoring core records no contexts, hence COVERAGE_CORE=ctrace.

Not run / limits

• zizmor (not installed locally), and the job on a real GitHub runner: the first CI run of this PR is the first real measurement.
• The operator set does not touch call arguments or string literals.
• The candidate tests are found from imports, so a module that is tested only through cli.py can show its lines as uncovered.

Local self-review (pr-review, lenses: correctness, silent-failures, tests, comments, design): no blocking findings. A design finding, that the step could overrun the job timeout and lose the summary, is fixed in the second commit.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Shared temporary-tree bytecode caches can cause mutants to execute stale code and produce incorrect results, while incomplete timeout cleanup can leave child processes running during subsequent runs.

2 open issues

• Prevent stale bytecode from contaminating coverage and mutant runs · New
• Clean up subprocess descendants after timeout or interruption · New

What changed in this PR

2 new findings.

This PR adds warn-only mutation testing for changed package lines, complementing changed-line coverage and the existing weekly whole-module mutation job.

Changes:

• Adds a budgeted mutation runner that selects covering tests and reports survivors in Markdown, JSON, and annotations.
• Adds a pull-request-only CI job with bounded runtime and a separate summary step.
• Documents local usage, adds focused tests, and ignores generated reports.

File	Description
dokimi/​tests/​test_mutate_changed.py	Tests mutation generation, source edits, selection, diff parsing, and reporting.
dokimi/​tests/​test_ci_label_triggers.py	Excludes the PR-only mutation job from expected push jobs.
dokimi/​scripts/​mutate_changed.py	Implements changed-line mutation testing and reports.
AGENTS.md	Documents the warn-only job and local command.
.gitignore	Ignores generated mutation reports.
.github/​workflows/​ci.yml	Adds the mutation job and summary publication step.

_{Review effort: Balanced}

[exai-bot]

OpenCodeReview: Commit 0c8e87d was not reviewed. This PR has reached its review limit (6 reviews).

CI at `0c8e87d`: 12 still running, 4 passing, 1 skipped

Check	Status
Build docs (strict)	still running
CodeQL (actions)	still running
CodeQL (python)	still running
External catalog smoke test (uxlfoundation/skills)	still running
Platforms and Python versions	still running
Static checks (ruff, schema, zizmor, licenses, pre-commit, bandit, secrets, changelog, schema com...	still running
benchmark	still running
conda-forge dependency solve (py3.11)	still running
conda-forge dependency solve (py3.14)	still running
mypy	still running

2 more not listed.

To review this commit: re-request review from exai-bot in the Reviewers list.

<details><summary>Last review (19619fa): no findings</summary>

Open threads from the bot: 0

</details>

[xaleryb]

• Ran: head merged with main on Linux py3.13 (full suite, ruff, mypy, bandit, deptry, check_growth). Also the script with CI parameters on the merges of fix(stats): finite values near the float range keep finite results or are refused, and tiny totals keep their variance #641 and refactor(judge): one default-judge rule that takes the answerer into account #621.
• --max-tests 30 runs the first 30 covering ids by name. refactor(judge): one default-judge rule that takes the answerer into account #621: judge_rule.py:41/77/93/95/96 are reported survived, but test_judge_provider_default.py kills each. fix(stats): finite values near the float range keep finite results or are refused, and tiny totals keep their variance #641: two stats.py:167 survivors are killed by test_stats_float_range.py. Expected: killed.
• Pushed (a85b04d..c36d0a4):
  • 312bcc4: exec bit, deptry exclude, _edits split (CI red).
  • 39c554c: drop the module's .pyc on each write (stale bytecode; forced same-mtime test).
  • 0a34195: run every covering test, read from a pytest args file. fix(stats): finite values near the float range keep finite results or are refused, and tiny totals keep their variance #641 now gives 14 killed / 2 survived, as the table says; refactor(judge): one default-judge rule that takes the answerer into account #621 gives 38 / 2 in 755 s.
  • 2d308d6: tests for the verdict, the unmutated-failure exclusion and the exit code.
  • c36d0a4: dokimi.process.run kills the process tree on timeout (Linux-tested).
• Left for the author: a survivor whose file still has unrun candidate tests could be reported inconclusive rather than survived.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.72%. Comparing base (f12057b) to head (0c8e87d).
⚠️ Report is 2 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #739   +/-   ##
=======================================
  Coverage   95.71%   95.72%           
=======================================
  Files         415      415           
  Lines       55346    55362   +16     
  Branches     9514     9516    +2     
=======================================
+ Hits        52977    52993   +16     
  Misses       1466     1466           
  Partials      903      903           

see 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[xaleryb]

Pushed f26652d (c36d0a4..f26652d): CI was red on our four run() tests for two reasons. The conda test environments do not install coverage (ci/config.yml leaves the coverage tools out), and in the wheel environments the inherited PYTHONSAFEPATH=1 made the copied tree import the installed dokimi, so the test file did not collect. The tests now pass the covering-test map directly and clear PYTHONSAFEPATH (as tests/test_intel_ci_conda_lock.py does); the exclusion of tests that fail unmutated has its own test with a stand-in coverage module. Checked in a conda env built from ci/conda-lock/linux-64-py3.11.txt without coverage and in a wheel-style venv: 14 passed in both; each earlier mutant still turns a test red.

[antonwolfy]

Review at f26652d. The process-tree kill and the stale-bytecode fixes hold up. Two defects make the report wrong: a mutant is counted as killed when no test ran, and import-time lines are never mutated. The inline comments also cover candidate discovery, a local-run gap and test gaps that let exit-code and annotation changes through.

What I ran, on a Linux review host at f26652d:

• make ci-local: 30 passed, 4 failed, 14 skipped; 19,764 tests passed. The failures are not the PR's. Install gitleaks hit an HTTP 504 and Scan commits failed with it. pre-commit fails on check-added-large-files for dokimi/docs/reference/cli-interface.json (501 KB), which this PR does not touch. Run tests shows 2 pytest-timeout errors in test_audit_test_quality.py, which passes alone (172 passed).
• mutation-diff runs under ci-local and reports "no mutable changed lines", since the PR changes no package file. mypy, deptry, ruff, zizmor and schema_compat pass.
• Reproductions for the code findings, driving run(), covering_tests and candidate_tests on a scratch git repo with real pytest and coverage runs, no model calls.
• A mutation run of test_mutate_changed.py against 96 one-line mutants of the script: 49 pass the file. Four of them decide an exit code, an annotation level or a measurement (inline).
• All 8 commits carry Signed-off-by.

GitHub checks red at this head: Static checks (...) and Test Linux Python 3.11. I could not reproduce either here. schema_compat passes in my run, and the 3.11 failure is a wall-clock assertion in code the PR does not touch. Keep them in view until a re-run is green.

Open for the author: @xaleryb's suggestion that a survivor whose file still has unrun candidate tests be reported as inconclusive, not survived, waits for the author. It still holds at this head: candidate_tests_not_run is listed, but such a survivor is reported as survived.

Bot points I checked that no longer apply at this head (6)

• Copilot's point on stale bytecode from equal-size edits in the same second does not apply: _write_source deletes the module's .pyc on every write, and test_a_same_size_mutant_in_the_same_second_runs_its_own_source forces the same-mtime case.
• Copilot's point on descendants left running after a timeout does not apply: _pytest goes through dokimi.process.run, which kills the whole process tree, and a test checks that no grandchild survives a mutant timeout.
• exai-bot's point on the shebang and file mode (Ruff EXE001) does not apply: the script is mode 100755 at this head and ruff check passes.
• exai-bot's point on the Windows timeout and pipe drain does not apply at this head: _pytest no longer uses subprocess.run, it uses dokimi.process.run, which kills the process tree on timeout; I checked this on Linux only.
• exai-bot's point on deptry reporting coverage as a transitive dependency does not apply: pyproject.toml now excludes the script, and deptry passes.
• exai-bot's point on cached bytecode running a previous mutant does not apply: the .pyc is removed on each write and restore, as above.

[napetrov]

Re: review by antonwolfy

Two defects make the report wrong: a mutant counted as killed when no test ran, and import-time lines never mutated.
Both fixed in a714bbf; the inline replies have the tests.

[napetrov]

Re: comment by xaleryb

CI was red on our four run() tests for two reasons.
Both fixed in 6aea1d3: coverage is optional, PYTHONSAFEPATH cleared.

[napetrov]

Re: comment by xaleryb

a survivor whose file still has unrun candidate tests could be reported inconclusive rather than survived.
Waiting on a decision before I change this: should it be inconclusive? My recommendation: yes, since no test ran, so it is not evidence of a gap. Option b keeps today's report. The thread stays open until then.

[exai-bot]

Must-fix — Impact: a passing test can be reported as baseline-failing and removed from both per-test and import-time coverage, leaving its changed lines unmutated. Trigger: parameter IDs a and a] - b produce nodes tests/test_zz.py::test_f[a] and tests/test_zz.py::test_f[a] - b]; only the latter fails. Evidence: FAILED tests/test_zz.py::test_f[a] - b] - AssertionError satisfies startswith(i + " - ") for both nodes, so named contains both; ids - bad and passed - bad then discard the passing node, and run() skips mutants without remaining coverage. Match each summary line to its longest matching known node rather than independently accepting every prefix, and add a two-ID regression. Workaround: avoid colliding parameter IDs. Severity: medium because this materially corrupts the bounded advisory mutation measurement, without affecting package runtime.

[vchamarthi]

Fixed in 0c8e87d: _failed attributes each summary line to the longest known id it starts with. FAILED ...test_f[a] - b] - AssertionError now returns only test_f[a] - b]. tests/test_mutate_changed.py::test_a_passing_id_that_prefixes_a_failing_id_is_not_failed pins it and fails without the change.

[antonwolfy]

Review at 337f794. The changes since my last review hold up: wrapped imports are read through ast, only exit 1 is a kill, and import-time lines are mutated. One bug is left in the new ' - ' handling, and three test gaps let a change to a verdict or to the choice of mutated lines through. Two nits inline. The merge also waits for the author on @xaleryb's question.

What I ran, on a Linux review host at 337f794:

• make ci-local: 35 passed, 0 failed, 14 skipped; 20,463 tests passed. The changed-line coverage gate passed; mutation-diff runs under ci-local and reports no mutable changed lines, since the PR changes no package file.
• Reproductions for the code finding, calling _failed directly and with real pytest output, with no model calls.
• A mutation run of test_mutate_changed.py against 45 one-line mutants of the script: 12 pass the file. Three decide a verdict or which lines get mutated (inline).
• All 13 non-merge commits carry Signed-off-by.

Still open from the author: @xaleryb's suggestion that a survivor whose file still has unrun candidate tests be reported as inconclusive, not survived, waits for the author. The author handed the choice back ("waiting on a decision") and the code is unchanged.

Checked, no issues

• Workflow wiring: needs: static, if: pull_request, the 30 and 22 minute timeouts and the flags passed to the script match the argparse flags; fetch-depth: 0 makes git merge-base origin/<base> work. .gitignore covers both output files.
• Exit codes: 0 is survived, 1 and a collection error (2 or 4 with ERROR collecting) are killed, -1 is timeout, and 3, 5 and a bare 2 or 4 are error. I ran each.
• Mixed files: an import-time line in a file with a failing test is credited to the tests that passed, and a file where nothing ran credits nothing.
• Docs: AGENTS.md names the job, the script, the cap of 40 and the local command correctly. No changelog fragment is required; no generated file is committed.

[antonwolfy]

_failed marks a passing id as failed when it is a prefix of a failing id. I agree with exai-bot here; what I add is the run with pytest's own output. @pytest.mark.parametrize("p", ["a", "a] - b"]) gives test_f[a] (passes) and test_f[a] - b] (fails). With -rfEp pytest prints FAILED tests/test_zz.py::test_f[a] - b] - AssertionError, and _failed returns both ids:

_failed("FAILED tests/test_zz.py::test_f[a] - b] - AssertionError\n",
        {"tests/test_zz.py::test_f[a]", "tests/test_zz.py::test_f[a] - b]"})
# ({'tests/test_zz.py::test_f[a]', 'tests/test_zz.py::test_f[a] - b]'}, [])

line.startswith(i + " - ") is true for both ids (names at :271 and named at :274). covering_tests then removes the passing test from every line, so a changed line only that test covers is reported as unexecuted and never mutated, and the test is listed as failing without a mutant.

Suggested fix: attribute each summary line to the longest id that matches it, max((i for i in ids if line == i or line.startswith(i + " - ")), key=len, default=None), and add a two-id regression with the log above.

[vchamarthi]

Fixed in 0c8e87d: _failed attributes each summary line to the longest known id it starts with. FAILED ...test_f[a] - b] - AssertionError now returns only test_f[a] - b]. tests/test_mutate_changed.py::test_a_passing_id_that_prefixes_a_failing_id_is_not_failed pins it and fails without the change.

[antonwolfy]

Nothing pins the merge base: diffing against base itself passes all 30 tests. Mutation check: replacing mb.stdout.strip() with base in this git diff call survives. test_changed_lines_* only uses HEAD~1, where the merge base and the base tip are the same commit. In CI, --base origin/main is the tip of main as it is when the job runs. If main advanced after the PR branched, diffing against the tip makes main's later edits to the same files look like the PR's changed lines, and they get mutated and reported.

Suggestion: add a test with a base branch that gains a commit editing another line of the same file after the PR branch was cut, and assert changed_lines(<base branch>) returns only the branch's lines.

[vchamarthi]

Fixed in 0c8e87d: Added a case where the base branch gains a commit after the branch was cut; changed_lines('main') returns only the branch's line. Diffing against base fails it. tests/test_mutate_changed.py::test_changed_lines_diffs_against_the_merge_base_not_the_base_tip pins it and fails without the change.

[antonwolfy]

Only exit 2 is tested as a collection-error kill. Mutation check: rc in (4,) and rc in (2, 3, 4) both pass the full file. The docstring says the exit code varies by pytest version, so dropping 2 (an older pytest) or adding 3 would turn broken-import kills into error, or an internal error into a kill, with no test red. That is the killed/error verdict --fail-on-survivor and the summary follow from.

Suggestion: a table test of _mutant_status: (4, "ERROR collecting x") is killed, (3, "ERROR collecting x") is error, (2, "") and (4, "") are error, (5, "") is error.

[vchamarthi]

Fixed in 0c8e87d: Added a table test of _mutant_status over exit codes 0, -1, 1, 2, 3, 4, 5 with and without ERROR collecting. rc in (4,) fails it. tests/test_mutate_changed.py::test_mutant_status_by_exit_code pins it and fails without the change.

[antonwolfy]

No test checks that the coverage pass asks for PASSED ids. Mutation check: changing -rfEp to -rfE passes all 30 tests. test_covering_tests_* stubs _pytest and ignores its arguments, and the one real-pytest test has no failing test. Without p, passed is empty on a real run, so in a file with one failing test the import-time constants and defaults are credited to no test and silently dropped from mutation. This is the case the latest commit fixed.

Suggestion: run covering_tests for real on a file with one failing and one passing test and an import-time FLAG = True, and assert line 1 is credited to the passing test.

[vchamarthi]

Fixed in 0c8e87d: Added a real covering_tests run on a file with one passing and one failing test and an import-time FLAG; line 1 is credited to the passing test. -rfE fails it. tests/test_mutate_changed.py::test_the_coverage_pass_asks_for_passed_ids pins it and fails without the change.

[antonwolfy]

I pushed a commit per nit, up to 19619fa, which fix the two nits from my review:

• 19619fa: dokimi/tests/test_mutate_changed.py:121: the summary test now has two killed mutants and one error mutant, and asserts 2 killed and 1 errors; printing the killed count or a hardcoded 0 as errors now fails it.
• 56e1109: dokimi/scripts/mutate_changed.py:11: step 1 of the docstring now says the working tree is diffed against the merge base of BASE and HEAD.

make ci-local at 19619fa: 35 passed, 0 failed, 14 skipped.

Not approving yet. Still open: the bug and the three risks from my review; @xaleryb on report a survivor whose file still has unrun candidate tests as inconclusive, not survived.

[vchamarthi]

Pushed 1 commit to ci-diff-mutation (19619fa..0c8e87d).

• _failed matches each summary line to the longest known id.
• _failed matches each summary line to the longest known id, preferring ids with no PASSED line.
• changed_lines passes -c core.quotePath=false to git diff.
• _pytest captures the whole output (capture_limit=None).
• _failed prefers ids without a PASSED line among those a summary line matches.

Threads: 5 fixed, 0 answered as not a defect.
Run: the full suite with main merged in, 20531 passed.