mutation: AST operator engine for the neutral benchmark - #2
Merged
Conversation
Foundation for the mutation study (paper Section 5, neutral suite). Classical mutation operators applied to the tool-agnostic workloads at real code locations, so injected faults are not hand-written. - src/testbench/mutation/operators.py: ROR, COR, CRP, AOR, SDL implemented on the AST; mutate_source() yields one mutant per applicable location (source + operator + line + before/after), each a single, valid-Python change. Imports nothing external, so the tool-agnostic CI guard stays green. - tests/test_mutations.py: every mutant parses and is single-location; all operators fire across the loop and orchestrator; the key faults are present (ROR on the score threshold, SDL deleting the loop break). Next: registry (MAST-labelled mutants + equivalents), benign edits, and the Chronicle-side eval harness (cut-point vs mock baseline) under integrations/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Tisha Chawla <tisha.chawla2020@vitalum.ac.in>
Completes the neutral mutation package. - registry.py: enumerate_labeled() maps each mutant of the two workloads to a MAST failure category (Cemri et al. 2025): deleting the loop break or flipping the convergence check -> unaware of termination; weakening the accept condition -> no/incorrect verification; altering routing -> disobey task specification; dropping a worker's output -> ignored other agent's input. Yields 7 labelled faults across all four categories, spanning both workloads. - equivalents.py: runs a loop mutant with the offline stub client (via exec so relative imports resolve) and drops mutants observably identical to the baseline. The loop cap is raised above the convergence point so termination faults are observable. - benign.py: behaviour-preserving edits (a dead local per function), the negative cases for the false-positive rate. Still imports nothing external (agnostic guard green). Tests: MAST labelling covers the categories; the baseline loop converges; deleting the break is non-equivalent; benign edits change nothing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Tisha Chawla <tisha.chawla2020@vitalum.ac.in>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Foundation for the neutral mutation study (paper Section 5). Classical mutation operators (ROR, COR, CRP, AOR, SDL) applied to the tool-agnostic workloads on the AST, so faults are injected at real code locations rather than hand-written.
src/testbench/mutation/operators.py:mutate_source()yields one mutant per applicable location (source + operator + line + before/after), each a single valid-Python change. Imports nothing external, so the tool-agnostic CI guard stays green.tests/test_mutations.py: every mutant parses and is single-location; all operators fire across the loop and orchestrator; the key faults are present (ROR on the score threshold, SDL deleting the loop break).19 mutants from the loop alone; neutral suite + agnostic guard green; no em/en dashes.
Next on this line: registry (MAST-labelled mutants + equivalents), benign edits, and the Chronicle-side eval harness (cut-point vs mock baseline) under
integrations/.