Improve test gap correctness and cost

[Evangelink]

Summary

Improve dotnet-test/test-gap-analysis correctness and cost without changing its nine stimuli or two-run count:

• inventory every high-risk caller-visible outcome before selecting mutations;
• make authorization denials complete across identity, resource, capability, action, and state-transition partitions;
• require an original-versus-mutant public counterfactual and suppress inert, private-only, future-only, or invented non-local changes;
• stop generic advisory reviews after one baseline and static source/assertion mapping;
• reserve mutation execution for explicit verification, exhaustive audits, and requested test additions;
• keep findings inside the user-named risk and calibrate exact values, severity, and suite verdicts from observable contract impact.

Evaluation diagnosis

Authoritative run 32954117134 evaluated the prior payload at e0bd807007664bc7839e59440e76318b73c4f584. Both results were conclusive VALID_NO_CHANGE measurements with zero errors, timeouts, or unmatched trajectories. Isolated activation was correct for every in-scope stimulus, and aggregate plugin activity was present.

Model	Stimulus W/T/L	Repeated-run W/T/L	Access-control W/T/L
claude-sonnet-4.6	4/4/1	10/4/4	0/1/1
gpt-5.6-luna	6/2/1	8/9/1	0/1/1

Both models lost Find logic and null-check mutation gaps in access control code:

A covers all requested security-test gaps, especially the two central omissions B fails to recognize: User write denial and the entire CanPerform write path.

A covers more of the rubric's distinct security gaps — notably Guest access denial and ElevateRole null/empty token handling, both of which B omits from its final answer.

The full transcripts also showed skilled responses admitting green-but-equivalent mutants: removing a null/empty guard that falls through to the same role result, and removing internal zero-entry cleanup even though every public API remains identical. The judge summarized the latter as:

B's sole claimed survivor is not behaviorally exposed by the stated API, and its recommended test cannot catch that mutation.

The skill spent 1.9-2.1x baseline token estimates in isolation and 2.74x in the plugin arm. The fixed mutation budget encouraged execution before complete outcome enumeration, then admitted unobservable mutants and produced narrower findings than baseline.

First exact-head reevaluation

Run 33063839973 evaluated commit 5dcc88194b5a43f90ad9759e4273614b4ad3ceba.

• Luna produced a valid, conclusive VALID_NO_CHANGE: 3W/2T/4L stimulus votes and 5W/5T/8L repeated runs, with zero errors or unmatched trajectories. Isolated and plugin activation were both present for access control.
• Access control improved from a loss to 1W/1L, but one skilled trial still omitted null/empty elevation handling and other denial surfaces after executing only two mutations.
• Mean isolated cost remained +114,012 tokens versus baseline. Generic boundary, access-control, and billing reviews cost 3.3-3.7x baseline.
• Judge evidence exposed additional concrete defects: an invented billing amount, a trivial one-line predicate reported as a gap, over-severe framing of a broadly protected suite, and explicit retry verification that covered fewer distinct outcomes than baseline.
• Sonnet produced no consolidated verdict. The artifact preserved 53/54 trajectories, but Run vally evaluations hit the workflow's 155-minute timeout. This is an invalid reliability result, not content evidence.

Second exact-head reevaluation

Run 33079053602 evaluated commit f1ef2a1df7f7f61744cb1f61b3d06245b6f95583. Both model results were valid and conclusive with zero errors, timeouts, or unmatched trajectories.

Model	Stimulus W/T/L	Repeated-run W/T/L	Access-control W/T/L	Mean token delta
claude-sonnet-4.6	5/2/2	10/4/4	2/0/0	+96,394
gpt-5.6-luna	4/2/3	7/6/5	1/1/0	+46,771

This verifies the core correction: neither model lost access control, and the Luna cost delta fell 59% from +114,012. Luna's generic boundary/access-control/billing ratios fell from 3.70/3.64/3.32x baseline to 1.41/1.62/1.88x.

The remaining losses exposed two narrow, reproducible output defects rather than a reason to add stimuli:

• well-tested inventory responses still invented statement reordering, overflow policy, and future-API effects, then over-framed symmetric guard gaps as Mixed;
• money-math-only billing responses still reported Balance/trivial IsPaid state and one omitted the tax-base-composition mutation.

The final iteration addresses those exact general failure classes while leaving the eval and execution policy unchanged.

Final semantic exact-head reevaluation

Run 33082991359 evaluated commit 26efa091e9b99c96f1cd54e0a302d93e9f604fb6. Both model results were valid and conclusive with zero errors, timeouts, or unmatched trajectories.

Model	Stimulus W/T/L	Repeated-run W/T/L	Access-control W/T/L	Mean token delta
claude-sonnet-4.6	5/2/2	10/5/3	2/0/0	+79,993
gpt-5.6-luna	6/2/1	11/5/2	2/0/0	+59,571

The original access-control failure is fully corrected: both models won both repeated trials. Luna's mean token delta is 48% below the first exact-head result (+59,571 versus +114,012), and its generic boundary/access-control ratios are 1.25x/1.17x baseline instead of 3.70x/3.64x. The verdicts remain VALID_NO_CHANGE, not regressions: Sonnet's sign test is p=.227, while Luna's 6W/2T/1L result narrowly misses the pass gate at p=.063.

The remaining losses do not prove a new general skill defect. One Sonnet billing trial violated the existing named-risk/triviality gate, while the other omitted a tax rate already required by the complete money-math ledger. Luna's retry loss omitted an intermediate retry boundary even though the skill already requires retries and each distinct high-risk outcome to be inventoried. These are model adherence/variance and baseline-saturation limitations, so no stimulus padding or speculative breadth was added.

Final reviewed-head reevaluation

Review wording was clarified in final head 844e6c25f071de8a217961c4ff5d5c55621f015b without changing the evaluated policy. The first review-only attempt, run 33086939333, was measurement-invalid when Sonnet hit the 60-minute comparison watchdog. Exact-SHA retry 33094381685 completed both models successfully:

Model	Stimulus W/T/L	Repeated-run W/T/L	Access-control W/T/L	Mean token delta
claude-sonnet-4.6	6/2/1	13/1/4	2/0/0	+104,397
gpt-5.6-luna	4/4/1	9/5/4	2/0/0	+25,904

Both final-head results are valid and conclusive with zero errors or unmatched trajectories. They are VALID_NO_CHANGE, not regressions: Sonnet narrowly misses the sign-test gate at p=.063, while Luna is p=.188. Access control remains a clean 2W/0L for both models, and Luna's mean token delta is 77% below the first exact-head result.

Concurrent exact-SHA run 33094276018, which supplies the required evaluation-status, independently completed with zero errors or unmatched trajectories. Sonnet recorded 6/1/2 stimuli, 8/6/4 repeated runs, access control 1/1/0, and +76,290 mean tokens; Luna recorded 4/3/2, 8/6/4, access control 2/0/0, and +42,962. The two valid retries vary in ties and losses but agree on the correction: neither model loses access control, and neither aggregate verdict is a regression.

Targeted follow-up from the required-status evidence

The full comparisons from run 33094276018 exposed two repeatable instruction-dilution defects and one eval-design mismatch:

Both give accurate, useful core money-math mutation analysis. A is better aligned with the request to skip low-value analysis, while B adds an irrelevant IsPaid gap.

A is materially more complete: it finds the untested null contract and the attempt-4 equality-guard survivor, and proposes focused tests for both.

The key differentiator is verification: Response A actually performed mutation testing ... Response B relied on reasoning and only ran the baseline suite.

The first two are skill-content failures: the existing prose boundary did not reliably suppress unrelated state predicates, and the generic boundary guidance did not consistently enumerate adjacent and equality-narrowed retry states. The third is an eval-design issue because the skill deliberately stops advisory analysis after one green baseline to control cost; mutation execution should not win when static source/assertion mapping is equally accurate.

Commit e2e79c88e860934086f5886c863d9956f07723b8 converts named risks into a public-outcome allowlist, adds a conditional ordered-guard/retry matrix, and aligns that advisory rubric with the intended execution policy. It does not add stimuli or repeated runs.

Follow-up exact-head measurement

Run 33101631148 evaluated e2e79c88e860934086f5886c863d9956f07723b8. Both results were valid and conclusive with zero errors, timeouts, or unmatched trajectories.

Model	Stimulus W/T/L	Repeated-run W/T/L	Access-control W/T/L	Retry W/T/L	Money-math W/T/L	Mean token delta
claude-sonnet-4.6	5/2/2	10/4/4	2/0/0	1/1/0	1/0/1	+60,070
gpt-5.6-luna	4/5/0	10/4/4	1/0/1	1/0/1	1/1/0	+75,805

The targeted behavior improved without an aggregate regression: Sonnet's retry result moved from 0/0/2 to 1/1/0, its money-math result moved from 0/0/2 to 1/0/1, and Luna had no stimulus-level losses. Sonnet's mean token delta also fell 21% from the preceding required-status run (+60,070 versus +76,290).

Full transcript inspection classified the remaining losses before another edit:

B both distracts with IsPaid and incorrectly suggests a day-15 test would kill the <=0-to-<0 zero-day boundary mutation.

B's post-change manual verification was compromised by a restore-path error.

Response A built an actual mutation-testing harness ... Since the task specifically asks whether bugs could slip through ... A's empirical confirmation makes its conclusions more trustworthy.

The first quote exposed an actual wording hole: the allowlist excluded stored predicates but not derived booleans. The second was implementation reliability: the response replaced the fixture's canonical mutation script with a fragile manual harness. The third confirmed that advisory rubrics still rewarded expensive execution despite the skill's one-baseline policy. Commit fd7c17508a09cebd5d4bee16772ef609494c459c closes those three general gaps without changing any prompt, stimulus name, grader, fixture, or run count.

Final exact-head result

Run 33109400096 evaluated final reviewed head ed6103909a5ea448de644bf6ecec218b07290826. Both model results were valid and conclusive with all 54 expected trajectories per model, zero errors, zero timeouts, and zero unmatched comparisons.

Model	Stimulus W/T/L	Repeated-run W/T/L	Gate	Mean token delta
claude-sonnet-4.6	5/2/2	9/5/4	p=.227; net +33.3%	+60,192
gpt-5.6-luna	3/3/3	7/4/7	p=.656; net +0.0%	+49,338

Both results are VALID_NO_CHANGE, not regressions. The original access-control scenario remains corrected: neither model recorded a stimulus-level access-control loss. Relative to the first exact-head Luna result, mean token overhead fell 57% (+49,338 versus +114,012); Sonnet's overhead is 21% below the preceding required-status run (+60,192 versus +76,290).

Reviewed-head evidence correction

Full transcript review of run 33109400096 separated model variance from four general content opportunities and three fixture/eval defects:

B correctly verifies IOException and zero but misses null entirely and makes an overly broad claim about negative classification coverage.

B adds useful empirical confirmation and more concrete test syntax, but misses the 100 threshold and contains a material mismatch: its proposed 500 test cannot kill its stated >=500→>=450 mutant.

A edges ahead by providing precise exact expected amounts as oracle cases ... whereas B ... spent effort on test execution that produced no results.

B ... frames the suite more negatively as 'Mixed.'

The content fix requires a complete pre-output outcome ledger, a witness input that distinguishes original from mutant and is reused by the proposed test, exact money-math oracle values, and a Strong verdict when only a few validation/default variants remain.

Fixture verification also found that the Rust Some(5) assertion already kills <= to <; the access-control fixture falsely claimed that a direct CanPerform ternary flip and removal of IsNullOrEmpty survive; and the dormant ShoppingCart prompt requested a promo API absent from production. The eval now grades the observable truth, requires both retry-side boundary survivors and Guest elevation denial, and removes the nonexistent promo requirement. The two dormant-scenario preference losses are routing/model variance because the target skill did not activate, not test-gap-analysis content evidence.

Commit 02a5532e404fa67f832e58d2503a60508d3d2210 implements these corrections without adding a stimulus or increasing runs: 2.

Retry-partition exact-head follow-up

Run 33169855342 evaluated 02a5532e404fa67f832e58d2503a60508d3d2210 with complete accounting and no errors, timeouts, or unmatched comparisons.

Model	Verdict	Stimulus W/T/L	Repeated-run W/T/L	Gate
claude-sonnet-4.6	Improved	7/1/1	13/4/1	p=.035; net +66.7%
gpt-5.6-luna	Not proven improved	6/2/1	10/5/3	p=.063; net +55.6%

Sonnet passed the authoritative gate. Luna's sole loss was repeatable and active: both skilled retry responses verified null, zero, attempt 2, and IOException, but omitted a later blocked attempt and derived retryable exception. The judge preferred baseline for exact-runtime-type narrowing and also conflated the rubric's ambiguous “equality-only narrowing” with type equality even though the intended guard survivor is attempt >= 3 becoming attempt == 3 at attempt 4.

Commit d94adf81e6bb46f01a7a0c5e3484131421ce7a14 makes the general partitions explicit: ordered upper guards use limit-1, limit, and limit+1, and polymorphic type classifiers include a representative derived accepted type. The rubric now separates attempt-4 guard narrowing from exact-runtime-type classification. It also replaces one skill-vocabulary criterion flagged by the Sonnet overfit report with evidence-truthfulness and distinguishing-input outcomes.

Final exact-head result

Run 33171904123 evaluated final head d94adf81e6bb46f01a7a0c5e3484131421ce7a14. Both model results were valid and conclusive: 2 expected/observed/written, zero invalid results, errors, timeouts, recovered comparison slots, or unmatched trajectories.

Model	Verdict	Stimulus W/T/L	Repeated-run W/T/L	Gate	Mean token delta
claude-sonnet-4.6	Improved	6/3/0	11/5/2	p=.016; net +66.7%	+36,476
gpt-5.6-luna	Not proven improved	6/2/1	12/2/4	p=.063; net +55.6%	+58,735

The retry refinement produced the intended behavior: Luna moved from a repeated 0/0/2 retry loss to 1/0/1 and a stimulus tie. One run now reports attempt 2, attempt 4, null, zero, IOException, and derived-type narrowing together.

Luna's sole remaining loss moved to money math, but the full transcripts do not justify another content change. Both skilled responses traced the private helpers, supplied exact expected amounts for every material partition, excluded generated/trivial/IsPaid surfaces, and matched the rubric. The judge slightly preferred baselines that included the rubric-excluded IsPaid gap or more examples, and called the skilled 113.00 wrong-base counterfactual “debatable”; 113.00 is the correct result for taxing only the 100 subtotal after adding a 5 fee. This is evaluator preference noise, not a missing general decision. Adding prose or stimuli now would be post-hoc overfitting.

The overfit warning was reviewed. Its classification varies materially by judge (the same rubric moved from 39 outcome / 19 technique / 1 vocabulary to 37 / 21 / 2 after a technique criterion was replaced), while the remaining technique criteria chiefly enforce verification the user explicitly requested or truthful evidence reporting. No phrase-matching content or new stimulus was added.

Current-main exact-head reevaluation

After merging current main, run 33174126023 evaluated 84908e6fc31e2804b521e610d08deed674cf3550 under schema v4, where the two dormancy contracts are retained but excluded from preference inference. Both contracts passed for both models, and the measurement was complete: 2 expected/observed/written, zero invalid results, errors, retries, or unmatched trajectories.

Model	Verdict	Eligible W/T/L	Repeated-run W/T/L	Gate	Mean token delta
claude-sonnet-4.6	Not proven improved	5/1/1	13/2/3	p=.109; net +57.1%	+104,920
gpt-5.6-luna	Improved	6/1/0	13/3/2	p=.016; net +85.7%	+47,566

Sonnet's sole loss was concrete and reproducible. Both skilled Rust responses called <= to < a survivor even though the existing asserted input changes from Some(5) to Some(2). They also attached uncompilable or already-killed edits to no-coverage rows. The judge identified the same central defect in both trials: the direct comparison flip is killed by the existing Some(5) assertion.

Commit e2f449067d2e7bbca08f476fd63e531f811edb1c fixes that general failure class without changing the eval: replay every exact mutation against every existing asserted input with all arguments fixed before admitting a survivor; treat any changed assertion as killed without demanding a dedicated test; reject uncompilable edits; and report no-coverage branches without inventing a survivor. Redundant completeness prose was removed, leaving the skill effectively cost-neutral at 3,210 BPE tokens versus 3,208 before this refinement.

Review commit 1c81a5b7db7f22081acec3908250e63b19dbf2b5 also clarifies that changing attempt >= 3 to attempt >= 2 lowers the retry cutoff while widening the predicate; “narrowing” now refers only to the separate attempt == 3 mutation. The expected behavior and eval breadth are unchanged.

Survivor-fix exact-head follow-up

Run 33176505548 evaluated e2f449067d2e7bbca08f476fd63e531f811edb1c with complete accounting and no invalid, errored, retried, or unmatched results.

Model	Verdict	Eligible W/T/L	Repeated-run W/T/L	Gate
claude-sonnet-4.6	Improved	6/1/0	12/4/2	p=.016; net +85.7%
gpt-5.6-luna	Activation contract failed	3/4/0	10/5/3	p=.125; net +42.9%

The survivor correction removed Sonnet's Rust loss and produced a credible pass. Luna's preference record had no losses, but one of two isolated runs invoked test-gap-analysis for the explicit happy/error/boundary/critical-path classification and TestCategory request. The full transcript showed that the positive description's generic “boundary” and “error” terms competed with its later exclusion.

Commit eb6c250da34f6c08fa8768e4d7a60a7bac25cdf3 removes that lexical overlap and moves the exact classification/tagging/counting exclusion ahead of broader coverage exclusions. The active triggers remain behavioral blind spots, bugs/changes existing tests would miss, and survived/pseudo-mutations. The edit reduces the skill to 3,206 BPE tokens, below both the 3,210-token survivor-fix payload and the 3,208-token merged payload.

Routing exact-head follow-up

Run 33181286037 evaluated eb6c250da34f6c08fa8768e4d7a60a7bac25cdf3. Both preference records passed the statistical gate, and the measurement was complete with zero invalid results, errors, retries, or unmatched trajectories.

Model	Preference	Repeated-run W/T/L	Dormancy	Mean token delta
claude-sonnet-4.6	6/1/0; p=.016; net +85.7%	15/2/1	1/2 passed	+53,370
gpt-5.6-luna	6/1/0; p=.016; net +85.7%	11/6/1	2/2 passed	+46,121

The first routing change repaired Luna, but Sonnet invoked the skill in both trait-distribution trials before discovering that no test files existed. The trace showed the broader “existing test suites” framing still made this the nearest available testing skill even though its negative clause excluded classification and tagging.

Commit 879491b74ab1d7b2e86e0a5a1338cddc992a2615 makes the production-change question a required activation condition: the request must ask whether a bug, change, or mutation could survive, name a behavioral blind spot, or tie missing edge cases to production behavior. Off-target suite organization is now described semantically rather than by repeating the dormancy prompt's trigger vocabulary. The skill remains effectively cost-neutral at 3,211 BPE tokens versus 3,208 in the merged payload.

Changes

• Public outcomes, including every independent return field and authorization denial, are inventoried before candidate selection.
• Authorization analysis uses an explicit ledger for permission-returning APIs, action branches, and accepted/rejected/invalid/null/empty state transitions.
• Every candidate must state a public input plus different original and mutant observations; green execution is rechecked for equivalence.
• Every exact mutation is first replayed against all existing asserted inputs; one changed assertion kills it even without a dedicated single-purpose test.
• Uncompilable edits are rejected, and no-coverage branches are reported without attaching a speculative survivor.
• The same distinguishing witness must appear in the proposed smallest test; an input that produces the same original and mutant result disqualifies the finding.
• Generic “would tests catch this?” reviews run one baseline, then stop execution and use source/assertion mapping for No coverage or Candidate survivor (unverified) findings.
• Advisory reviews do not troubleshoot a failed/empty runner after the one permitted baseline attempt.
• Explicit verification executes one representative observable candidate per distinct high-risk outcome instead of stopping after one or two.
• A user-named risk becomes an explicit public-outcome allowlist; unrelated observable state is removed before reporting.
• Money-math analysis now covers every rate/tier, exact boundary, operation order, percentage base/composition, floor/cap, and rounding while excluding non-monetary predicates, including derived booleans.
• Ordered guards and retry limits now enumerate invalid, first-valid, last-allowed, first-blocked, and later-blocked outcomes with concrete limit-1, limit, and limit+1 witnesses.
• Polymorphic type classifiers include a representative derived accepted type when exact-runtime-type narrowing would change the public result.
• Candidates must be concrete changes/removals to existing code; inserted/reordered statements, undefined overflow/metadata semantics, and future-contract effects are excluded.
• Trivial one-line wrappers/predicates are excluded unless specifically requested.
• Exact amounts and boundaries must be cross-checked against the unmodified implementation or an exact assertion.
• Money-math findings supply an exact witness result rather than a generic “assert the exact amount” recommendation.
• Broadly protected suites remain Strong when only a few validation/default variants are uncovered.
• Focused output remains one verdict, one complete findings table, and one strengths sentence.
• Requested test additions use a repository-supplied mutation-verification script as the final authority and cannot claim success after that command errors.
• All advisory rubrics treat one green baseline plus accurate static mapping as sufficient instead of rewarding unnecessary mutation execution.
• Routing excludes happy/error/boundary/critical-path classification, tagging, and counting before broader coverage terms while retaining behavioral-gap triggers.
• Activation now requires explicit production-change, survivor, behavioral-blind-spot, or production-edge-case intent.
• Review feedback clarified input-guard acceptance versus exception ParamName, made survivor classification non-circular, and normalized risk/verdict terminology.

Validation

• python eng\eval-quality\check_eval_quality.py — no errors.
• Plugin validator — all 22 skills, 10 agents, and the plugin passed after each iteration (using a pre-downloaded Copilot CLI because the registry TLS handshake was unavailable to MSBuild).
• npx markdownlint-cli2 plugins\dotnet-test\skills\test-gap-analysis\SKILL.md plugins\dotnet-test\skills\test-gap-analysis\references\mutation-catalog.md — 0 errors.
• Deterministic checks confirm nine stimulus names and runs: 2; no stimulus was added. One dormant prompt was corrected to match the supplied API, and false rubric/fixture claims were replaced with observable behavior.
• All six .NET fixture suites built and passed (11 + 9 + 2 + 3 + 15 + 2 tests).
• The affected access-control, pricing, and inventory fixtures were rerun after the latest correction: 9, 11, and 15 tests passed.
• git diff --check — passed.
• Cargo is unavailable locally; the unchanged Rust fixture passed in the authoritative official runs.

Checklist

• Kept the existing eval stimuli and repeated-run count unchanged.
• Classified each conclusive content/cost result before editing.
• Distinguished the invalid Sonnet timeout from valid model comparisons.
• Inspected full official results, partial artifacts, and losing transcripts.
• Kept the change focused on test-gap-analysis.

[Evangelink]

/evaluate

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
⚠️	dotnet-test	test-gap-analysis	5/10	50%

Uncovered: dotnet-test/test-gap-analysis

• [Validation] Scope stayed proportional to the request (line 289)
• [Validation] Every high-risk public outcome in scope was inventoried (line 291)
• [Validation] Every temporary mutation was reverted (line 295)
• [Validation] Recommendations target only demonstrated gaps (line 297)
• [Validation] A supplied canonical mutation verifier was run and reported, not replaced (line 300)

[Copilot]

Pull request overview

This PR refines the dotnet-test/test-gap-analysis skill guidance to improve correctness and reduce cost by requiring a caller-visible outcome inventory before mutation selection, tightening observability/equivalence rules (including auth-denial outcomes), and limiting execution to the smallest decisive set while still reporting all high-risk findings.

Changes:

• Replaced the prior workflow with a decision flow that inventories caller-visible outcomes (including authorization denials) before selecting/execing mutations.
• Added stricter “public counterfactual” requirements (original vs mutant observations) and post-green-run rechecks to suppress inert/equivalent survivors.
• Updated output contract and exhaustive-audit reference to avoid repeating findings in prose and to count only executed/definitively classified candidates.

Show a summary per file

File	Description
plugins/dotnet-test/skills/test-gap-analysis/SKILL.md	Reworked the core decision flow, verification rules, and output contract to prioritize caller-visible outcomes and reduce unnecessary execution.
plugins/dotnet-test/skills/test-gap-analysis/references/mutation-catalog.md	Updated the exhaustive audit procedure and equivalence filters to require public observability and post-run counterfactual checks.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

[github-actions]

👋 @Evangelink — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[github-actions]

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 5dcc88194b5a43f90ad9759e4273614b4ad3ceba to retry this exact commit.

1 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

[Evangelink]

/evaluate f1ef2a1

[Copilot]

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

[github-actions]

📊 Skill Evaluation Results

2 model/skill results across 1 skill and 2 models — ✅ 0 improved, ➖ 2 not proven improved, ⚠️ 0 invalid or underpowered, 📉 0 preference losses (report only).

Measurement identity: evaluated commit f1ef2a1df7f7f61744cb1f61b3d06245b6f95583; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when the aggregate net win across distinct-stimulus votes is at least 20% and an exact one-sided sign-test result of p ≤ 0.05. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
test-gap-analysis	claude-sonnet-4.6	➖ Not proven improved	n=9; 5W/2T/2L; d=7; p=0.227; net +33.3%	🟡 0.35	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
test-gap-analysis	gpt-5.6-luna	➖ Not proven improved	n=9; 4W/2T/3L; d=7; p=0.500; net +11.1%	✅ 0.13	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, and net win. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Not proven improved — test-gap-analysis (claude-sonnet-4.6)

Why: Net win +33.3% (5W/2T/2L over 9 stimulus vote(s), sign test p=0.227), mean preference +20.0% across 18 paired run(s) — not credible (sign test p=0.227 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 5W/2T/2L; d=7; p=0.227; net +33.3%

Overfit: Moderate (score 0.35)

Repeated-run reliability (not used by the gate): 18 paired runs (10W/4T/4L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
= Acknowledge well-tested code with few surviving mutations	+0.0%	+0.0%	1/0/1
▼ Decline request to write new tests from scratch	-50.0%	-20.0%	0/1/1
▼ Skip trivial and generated code while tracing private call chains	-100.0%	-40.0%	0/0/2
= Stay dormant for test trait distribution	+0.0%	+0.0%	0/2/0

Illustrative judge evidence:

• Acknowledge well-tested code with few surviving mutations: A better reflects the expected guard coverage and gives a useful mutation-oriented assessment. B is well structured and more explicit about outcomes, but its false assertion that negative quantities lack tests is a material coverage-analysis error. A's suggested second-remove ...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — test-gap-analysis (gpt-5.6-luna)

Why: Net win +11.1% (4W/2T/3L over 9 stimulus vote(s), sign test p=0.500), mean preference +1.1% across 18 paired run(s) — not credible (sign test p=0.500 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 4W/2T/3L; d=7; p=0.500; net +11.1%

Overfit: Low (score 0.13)

Repeated-run reliability (not used by the gate): 18 paired runs (7W/6T/5L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
▼ Acknowledge well-tested code with few surviving mutations	-100.0%	-70.0%	0/0/2
▼ Decline request to write new tests from scratch	-50.0%	-20.0%	0/1/1
▼ Skip trivial and generated code while tracing private call chains	-50.0%	-20.0%	0/1/1
= Stay dormant for test trait distribution	+0.0%	+0.0%	0/2/0
= Verify guard and exception-type gaps in retry policy tests	+0.0%	+0.0%	1/0/1

Illustrative judge evidence:

• Acknowledge well-tested code with few surviving mutations: Response A actually performed empirical mutation testing, producing verified CAUGHT/ESCAPED results that directly answer the user's question with evidence. It correctly identifies the suite as well-covered with a few genuine boundary gaps. Response B, despite loading a test-ga...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1080 in dotnet/skills, download eval artifacts with gh run download 33079053602 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/f1ef2a1df7f7f61744cb1f61b3d06245b6f95583/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[Evangelink]

/evaluate 26efa09

[Copilot]

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

plugins/dotnet-test/skills/test-gap-analysis/SKILL.md:34

• The “Explicit survivor verification” row is phrased circularly (“execute … for outcomes reported as Survived or Killed”). At the scope-setting stage nothing is “reported” yet, so this can be misread as permission to skip verification for outcomes that should be classified, leading to under-verification.

| Explicit survivor verification | Inventory all requested outcomes; execute one representative observable candidate for each distinct high-risk outcome reported as **Survived** or **Killed** |

plugins/dotnet-test/skills/test-gap-analysis/SKILL.md:103

• This bullet says “exception message/parameter metadata”, but earlier the doc talks about “exception type/parameter”, which can be read as ParamName. Clarify that it’s specifically exception message/ParamName details that are excluded unless established by contract/tests.

- overflow behavior, exception message/parameter metadata, or other semantics

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

[Evangelink]

/evaluate e2f4490

[Copilot]

Copilot review overview

Review tier: Lite
Findings: 1

Pre-existing issues (1)

Severity	Finding
	tests/​dotnet-test/​test-gap-analysis/​eval.yaml — The rubric wording says “narrowing attempt >= 3 to attempt >= 2”, but that change widens the… View comment

[Evangelink]

/evaluate 1c81a5b

[Copilot]

Copilot review overview

Review tier: Lite
Findings: None

Issues resolved since last review (1)

Severity	Finding
	tests/​dotnet-test/​test-gap-analysis/​eval.yaml — The rubric wording says “narrowing attempt >= 3 to attempt >= 2”, but that change widens the… View resolved comment

[github-actions]

📊 Skill Evaluation Results

2 model/skill results across 1 skill and 2 models — ✅ 1 improved, ➖ 0 not proven improved, ⚠️ 0 invalid or underpowered, ⛔ 1 activation contract failure, 📉 0 preference losses (report only).

Measurement identity: evaluated commit e2f449067d2e7bbca08f476fd63e531f811edb1c; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
test-gap-analysis	claude-sonnet-4.6	✅ Improved	n=7; 6W/1T/0L; d=6; p=0.016; net +85.7%; 2 dormancy excluded	🟡 0.34	—	Review overfit evidence.
test-gap-analysis	gpt-5.6-luna	⛔ Activation contract failed	n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%; 2 dormancy excluded	✅ 0.16	Dormancy contract: 1 unexpected activation(s)	Narrow skill routing so the listed off-target scenarios stay dormant.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
• ⛔ Activation contract failed — the isolated target skill activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

⛔ Activation contract failed — test-gap-analysis (gpt-5.6-luna)

Why: Net win +42.9% (3W/4T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +15.6% across 18 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%; 2 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s)

Overfit: Low (score 0.16)

Repeated-run reliability (not used by the gate): 18 paired runs (10W/5T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Acknowledge well-tested code with few surviving mutations	Eligible	+0.0%	+0.0%	0/2/0
= Skip trivial and generated code while tracing private call chains	Eligible	+0.0%	+0.0%	1/0/1
= Stay dormant for test trait distribution	Excluded (activation contract)	+0.0%	+0.0%	1/0/1
= Verify and close only the surviving gaps in a partial suite	Eligible	+0.0%	+0.0%	0/2/0
= Verify guard and exception-type gaps in retry policy tests	Eligible	+0.0%	+0.0%	1/0/1

Illustrative judge evidence:

• Acknowledge well-tested code with few surviving mutations: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

✅ Improved — test-gap-analysis (claude-sonnet-4.6)

Why: Net win +85.7% (6W/1T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.016), mean preference +35.6% across 18 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — credibly better

Next action: Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=7; 6W/1T/0L; d=6; p=0.016; net +85.7%; 2 dormancy excluded

Overfit: Moderate (score 0.34)

Repeated-run reliability (not used by the gate): 18 paired runs (12W/4T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Acknowledge well-tested code with few surviving mutations	Eligible	+0.0%	+30.0%	1/0/1
▼ Decline request to write new tests from scratch	Excluded (activation contract)	-50.0%	-20.0%	0/1/1

Illustrative judge evidence:

• Acknowledge well-tested code with few surviving mutations: A gives a substantially more appropriate overall characterization and avoids treating unobservable internal representation as a critical defect. It misses the untested negative RemoveStock validation path, which B correctly finds, but B's headline high-risk surviving mutation ...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1080 in dotnet/skills, download eval artifacts with gh run download 33176505548 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/e2f449067d2e7bbca08f476fd63e531f811edb1c/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[Evangelink]

/evaluate eb6c250

[Copilot]

Copilot review overview

Review tier: Lite
Findings: None

[github-actions]

📊 Skill Evaluation Results

2 model/skill results across 1 skill and 2 models — ✅ 1 improved, ➖ 0 not proven improved, ⚠️ 0 invalid or underpowered, ⛔ 1 activation contract failure, 📉 0 preference losses (report only).

Measurement identity: evaluated commit eb6c250da34f6c08fa8768e4d7a60a7bac25cdf3; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
test-gap-analysis	claude-sonnet-4.6	⛔ Activation contract failed	n=7; 6W/1T/0L; d=6; p=0.016; net +85.7%; 2 dormancy excluded	🟡 0.35	Dormancy contract: 1 unexpected activation(s)	Narrow skill routing so the listed off-target scenarios stay dormant.
test-gap-analysis	gpt-5.6-luna	✅ Improved	n=7; 6W/1T/0L; d=6; p=0.016; net +85.7%; 2 dormancy excluded	✅ 0.16	—	None.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
• ⛔ Activation contract failed — the isolated target skill activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

⛔ Activation contract failed — test-gap-analysis (claude-sonnet-4.6)

Why: Net win +85.7% (6W/1T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.016), mean preference +37.8% across 18 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=7; 6W/1T/0L; d=6; p=0.016; net +85.7%; 2 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s)

Overfit: Moderate (score 0.35)

Repeated-run reliability (not used by the gate): 18 paired runs (15W/2T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▲ Stay dormant for test trait distribution	Excluded (activation contract)	+100.0%	+40.0%	2/0/0
= Verify guard and exception-type gaps in retry policy tests	Eligible	+0.0%	+0.0%	1/0/1

Illustrative judge evidence:

• Verify guard and exception-type gaps in retry policy tests: A covers one important empirically relevant survivor that B completely misses—the null guard—and recommends its minimal test, while matching B on zero, IOException, and the attempt-2 cutoff. Both substantially miss the later-blocked-attempt and derived-type mutation analyses, ...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

Routine passing details for 1 result are in Full Results.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1080 in dotnet/skills, download eval artifacts with gh run download 33181286037 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/eb6c250da34f6c08fa8768e4d7a60a7bac25cdf3/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[Evangelink]

/evaluate 879491b

[Copilot]

Copilot review overview

Review tier: Lite
Findings: None

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

plugins/dotnet-test/skills/test-gap-analysis/SKILL.md:294

• The continuation line here is indented enough to be parsed as an indented code block in CommonMark, so the bold markup will render literally instead of as emphasis. Merge it into the checklist bullet (or reduce indentation) so it renders as intended.

- [ ] Every outcome labeled **Survived** was executed; unexecuted candidates use
      **Candidate survivor (unverified)**

[github-actions]

📊 Skill Evaluation Results

2 model/skill results across 1 skill and 2 models — ✅ 0 improved, ➖ 2 not proven improved, ⚠️ 0 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 879491b74ab1d7b2e86e0a5a1338cddc992a2615; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
test-gap-analysis	claude-sonnet-4.6	➖ Not proven improved	n=7; 6W/0T/1L; d=7; p=0.063; net +71.4%; 2 dormancy excluded	🟡 0.30	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
test-gap-analysis	gpt-5.6-luna	➖ Not proven improved	n=7; 4W/2T/1L; d=5; p=0.188; net +42.9%; 2 dormancy excluded	✅ 0.15	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
• ⛔ Activation contract failed — the isolated target skill activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Not proven improved — test-gap-analysis (claude-sonnet-4.6)

Why: Net win +71.4% (6W/0T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +30.0% across 18 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.063 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 6W/0T/1L; d=7; p=0.063; net +71.4%; 2 dormancy excluded

Overfit: Moderate (score 0.30)

Repeated-run reliability (not used by the gate): 18 paired runs (14W/2T/2L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Verify and close only the surviving gaps in a partial suite	Eligible	-100.0%	-70.0%	0/0/2

Illustrative judge evidence:

• Verify and close only the surviving gaps in a partial suite: A performed the requested implementation: it added a separate focused test file, preserved protected files, ran the suite, and verified the five mutations. B validated a state in which the gap tests already existed but did not add or independently establish the original-suite ...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — test-gap-analysis (gpt-5.6-luna)

Why: Net win +42.9% (4W/2T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +8.9% across 18 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.188 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 4W/2T/1L; d=5; p=0.188; net +42.9%; 2 dormancy excluded

Overfit: Low (score 0.15)

Repeated-run reliability (not used by the gate): 18 paired runs (7W/8T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Acknowledge well-tested code with few surviving mutations	Eligible	-50.0%	-20.0%	0/1/1
▼ Decline request to write new tests from scratch	Excluded (activation contract)	-50.0%	-20.0%	0/1/1
= Skip trivial and generated code while tracing private call chains	Eligible	+0.0%	+0.0%	0/2/0
= Stay dormant for test trait distribution	Excluded (activation contract)	+0.0%	+0.0%	0/2/0
= Verify and close only the surviving gaps in a partial suite	Eligible	+0.0%	+0.0%	1/0/1

Illustrative judge evidence:

• Acknowledge well-tested code with few surviving mutations: Both correctly pass the baseline and identify similar gaps (negative RemoveStock, zero threshold). Response A empirically verified mutations, producing accurate, well-substantiated conclusions and framing gaps as minor improvements, matching most rubric criteria. Response B to...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1080 in dotnet/skills, download eval artifacts with gh run download 33183940988 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/879491b74ab1d7b2e86e0a5a1338cddc992a2615/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

📊 Skill Evaluation Results

2 model/skill results across 1 skill and 2 models — ✅ 1 improved, ➖ 1 not proven improved, ⚠️ 0 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 879491b74ab1d7b2e86e0a5a1338cddc992a2615; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
test-gap-analysis	claude-sonnet-4.6	➖ Not proven improved	n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%; 2 dormancy excluded	🟡 0.43	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
test-gap-analysis	gpt-5.6-luna	✅ Improved	n=7; 5W/2T/0L; d=5; p=0.031; net +71.4%; 2 dormancy excluded	✅ 0.17	—	None.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
• ⛔ Activation contract failed — the isolated target skill activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Not proven improved — test-gap-analysis (claude-sonnet-4.6)

Why: Net win +57.1% (5W/1T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +25.6% across 18 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.109 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%; 2 dormancy excluded

Overfit: Moderate (score 0.43)

Repeated-run reliability (not used by the gate): 18 paired runs (10W/5T/3L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Decline request to write new tests from scratch	Excluded (activation contract)	+0.0%	+0.0%	1/0/1
▼ Skip trivial and generated code while tracing private call chains	Eligible	-50.0%	-20.0%	0/1/1
= Stay dormant for test trait distribution	Excluded (activation contract)	+0.0%	+0.0%	0/2/0
= Verify guard and exception-type gaps in retry policy tests	Eligible	+0.0%	+0.0%	1/0/1

Illustrative judge evidence:

• Skip trivial and generated code while tracing private call chains: Both answers are substantially correct and actionable on the money-math gaps. A is more focused on the requested mutation points, while B adds an out-of-scope IsPaid observation that the rubric expressly says not to treat as meaningful.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

Routine passing details for 1 result are in Full Results.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1080 in dotnet/skills, download eval artifacts with gh run download 33184783786 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/879491b74ab1d7b2e86e0a5a1338cddc992a2615/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate 1c81a5b7db7f22081acec3908250e63b19dbf2b5 to retry this exact commit.

1 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.