test: finish the mutation gate with its report, its measurement and its proof - #123
Merged
Merged
Conversation
Adds the html reporter beside clear-text and json, and uploads reports/mutation as a CI artifact on every run of the mutation job, success or failure, since a failing run is exactly when the report is needed.
ADR 0120: two clean runs over src/ hold 606/606 killed, 0 survived, 0 timed out, score 100.00, so the break score stays 100 with no headroom. Measuring scripts/ in the same scope surfaced two sandbox defects instead of a number worth a threshold: several gate scripts read the tracked tree with git ls-files, which Stryker's sandbox cannot answer, and several gate suites exercise their target only through a spawned child process, which per-test coverage cannot attribute. scripts/ stays out of mutate until both are fixed. Also documents the mutation gate as the docs/architecture.md Gates table's nineteenth row, which had never named it.
Extends REQ-022's suite so a future change to any of the three has to touch this file: the html reporter stays listed, mutate stays exactly src/**/*.ts plus the boundary exclusion until docs/adr/architecture/0120's successor changes it, and the mutation CI job keeps its always-run artifact upload.
#120 added eight src/ modules between the first measurement and the rebase onto it. Two fresh runs hold 21 modules, 868 mutants, 2 ignored, 866 killed, 0 survived, 0 timed out, score 100.00 both times, superseding the pre-#120 numbers; the break score, the scripts/ scope decision and the ten-minute trigger are unchanged by the new count. Also corrects the Gates table count to twenty now that #121's agents gate sits beside this ADR's mutation row.
ExtraToast
added
type: test
Shape
CoverageBaseline:
This pull request changed no covered line. |
ExtraToast
added a commit
that referenced
this pull request
Sep 16, 2026
Root ADR 0120 landed on main in #123 while this branch took 0120 for the parity seam, and test/adr-contract.test.ts refuses a number used in both registers. The branch alone is green, so the collision only appears against merged main. The parity seam becomes 0121 and the two tiers 0122, the next two numbers free in either register, with every citation moved with them.
ExtraToast
added a commit
that referenced
this pull request
Sep 16, 2026
* docs: decide the parity seam and the two tiers under emf/ Record two decisions for the model-driven implementation. 0120 puts the parity suite on the pipeline's file interface. The suite imports Pipeline, Parsed, Diagnostic and Descriptor today, which is why parity/ is an eclipse-plugin bundle while the architecture says a module needing no p2 bundle stays a plain jar. Moving the suite onto the files the pipeline writes closes that drift and narrows the test surface to the one an examiner exercises from a launch configuration. 0121 splits the tree into bundles/ for what Tycho builds and Eclipse imports and tests/ for what only Maven runs, and makes the second tier Kotlin. Ecore generates Java and a bundle compiles through JDT against a target platform with no Kotlin unit, so the bundle tier stays Java. Both are proposed, and the tree is unchanged: emf/docs/architecture.md describes the flat modules it still has and carries each decision as a "proposed, not landed" note beside the section it will rewrite. * docs: drop the em-dash the decision record carries test/emdash.test.ts refuses an em-dash in tracked text, and 0120 carries one, so the Tests job is red on that alone. * docs: drop the em-dashes the proposed-not-landed note carries test/emdash.test.ts refuses an em-dash in tracked text, and the note for 0120 carries two, so the Tests job is red on them alone. * docs: take the next free estate-wide numbers for the two records Root ADR 0120 landed on main in #123 while this branch took 0120 for the parity seam, and test/adr-contract.test.ts refuses a number used in both registers. The branch alone is green, so the collision only appears against merged main. The parity seam becomes 0121 and the two tiers 0122, the next two numbers free in either register, with every citation moved with them.
ExtraToast
added a commit
that referenced
this pull request
Sep 16, 2026
* feat: write the parsed intent, the diagnostics and the descriptor as files the build leaves behind A run of the pipeline now leaves behind everything the parity contract compares, as files, in the canonical JSON the oracles are committed in. For every case under spec/v1/examples/, Outputs writes the parsed intent or the diagnostics that refused it, beside the exit code the run ended on, under cli/target/parity in a tree that mirrors the example tree, so a written file and its oracle are obviously a pair. DescriptorFile writes the source metamodel's descriptor from the same reflective walk Descriptor already performs, under metamodel/target/parity. Neither file is committed. The canonical JSON writer moves down to metamodel/, the lowest module that writes a canonical file, because the pipeline now writes what the parity suite used to serialise. The parity suites are otherwise untouched: this is the expand half of the move recorded in 0120, so a wrong output shape is caught here rather than after the suites depend on it. Closes #128. * refactor: move the parity suite onto the pipeline's file interface The suite decides every case from the files a run leaves behind and the exit code beside them, so it holds no EMF type, calls nothing, and needs nothing from the target platform. The module becomes a plain jar: its META-INF/MANIFEST.MF and build.properties are gone, and it declares the test libraries the parent's bundle profile used to give it. Each case asserts its file exists before comparing it, so a file the run owes and did not write fails as a missing file rather than as a skipped case (0120). Two gates had to survive that change rather than quietly stop enforcing. ArchitectureTest saw the other modules only because the manifest required them; with no dependency left it reads their class directories by path instead. Because withOptionalLayers excuses an empty layer, EMF-010 and EMF-011 would then pass on an empty import while proving nothing, so EMF-017 fails when a module whose classes the build wrote is missing from the import. All three now fail on a fixture that breaks them, as 0104 requires, and the layered rule no longer names the parity layer as a permitted accessor. Also removes an em-dash the decision record carried, which test/emdash.test.ts refuses. Closes #129. * refactor: split emf/ into a bundle tier and a test tier The listing now says which modules a course examiner opens. `bundles/` holds the five Tycho builds Eclipse imports; `tests/` holds the parity tier Maven alone runs, which by 0120 holds no EMF type and needs nothing from the target platform. The directory names stay short, since Eclipse names an imported project from its `artifactId` regardless (0121). Every document that names a module by its old path moves with it: the rule ledger, whose EMF-014 check fails the build the moment a row's named file is not where the row says, the witness list, this tree's architecture, two decision records, the refusal examples' README, the root architecture, and the two root tests that name a parity path in a comment and in a fixture string. `emf/README.md` gains the table that says which artefact lives where and which task grades it. Two things the move would have broken quietly. The module POMs now sit two directories below the parent, so each declares its `relativePath`, and `scripts/lint-codes.ts` matched `emf/<module>/src/test/` one segment deep, which would have stopped counting a Java test as exercising a code for the whole tree rather than only for the tier that moved. Closes #130. * refactor: port the test tier to Kotlin, with the gates the estate runs on it `tests/parity` is written in the estate's JVM language, with the idiom its other repositories already use: JUnit 5, AssertJ, ArchUnit, and a test function named as a sentence in backticks. The bundle tier stays Java, because Ecore generates Java, Xtext's runtime hooks are Java, and a bundle compiles through JDT against a target platform holding no Kotlin unit (0121). The gates move with it. Spotless gains a ktlint step over `src/*/kotlin`, and EMF-018 is its ledger row. The Kotlin compiler runs with `-Werror`, which is EMF-006's rule in the compiler this tier uses. The coverage floor holds unchanged. The mutation threshold also stands at 100, which needed one thing rather than a decision to lower it: `kotlinc` emits a null check for every platform type crossing into Kotlin and a `closeFinally` for every `use`, and removing either call is an equivalent mutation, not an uncovered one, so pitest is told not to mutate calls to them. That is the same treatment `bundles/cli` already gives the one-off OCL language registration. The word that changes in 0114 is "Java", not the rule: a witness is a JUnit test in whichever language its module is written, so `Ledgers.checkWitnesses` now reads a `.kt` function and a `.java` method alike, three of the six witnesses name Kotlin functions and three name Java methods, and a new fixture proves a Java witness in a bundle is still found. The decision record, the witness list, EMF-013's row and the gates sentence say so. Closes #131. * docs: follow the two records to the numbers they landed on Root ADR 0120 landed on main in #123 while #126 was open, so the parity seam became 0121 and the two tiers 0122 in that pull request. This moves the citations that live here: the slugged links, the `docs/adr/emf/NNNN` form a pom comment and a javadoc line use, and the number in a link's own text.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #31
What this branch does
The mutation gate #112 introduced already ran on every pull request and already sat in
Pipeline Complete. It was missing three things the ticket asks for: a report a reviewer can open, a record of the numbers the break score came from, and proof that it fails. This adds all three.mutationjob uploadsreports/mutationas an artifact with a pinnedactions/upload-artifact. The upload runs even when the mutation step fails, since a failing run is when the report is needed, and it errors if the directory is empty.docs/adr/architecture/0120-the-mutation-break-score-is-measured-not-assumed.mdrecords the measurement, the break score and the scope, each with its reason, and the trigger for incremental mode.mutationrow, although the job has existed since feat: parse the minimal project intent in both implementations and match its oracle #112. It has one now, and the table counts twenty gates.htmlreporter, the scope and the pinned upload step, beside the existing check that the agent configuration stays out of the sandbox.The measurement
Two runs over current
main, which mutates 21 modules undersrc/:The two ignored mutants are an existing
// Stryker disable next-lineinjson-schema.ts.The decisions
The break score stays at 100, with no headroom. Two identical runs with no survivor and no timeout is the same bar ADR 0101 accepts for the coverage ratchet. Headroom now would be slack against a failure neither run produced.
The scope stays
src/. The ticket names the gate scripts, so addingscripts/**/*.tswas measured, and it does not work yet, for two structural reasons:git ls-files. Stryker's sandbox has no git index, so their dry run fails outright.That is not a judgement on the scripts' tests, and the record names both fixes as follow-up work.
The incremental-mode trigger is not reached. The ticket sets it at about ten minutes. The run takes under forty seconds.
Verification
Weakening
test/canonical-json.test.tsso it no longer checks the message of the error it expects, the one test that killed that mutant:Restored, the score returns to 100.00 and the exit code to 0. That was on the branch before #120 landed, locally, since a throwaway branch was out of reach.
npm run lint:adrs: 97 files clean.npm run lint:meaning: 166 files clean.npm run verifyexit 0: 705 tests, coverage unchanged.actionlint1.7.7, the version CI pins, clean.Not yet proven
The artifact upload itself runs only inside Actions. This pull request's
mutationjob is its first run.