Separate dormancy evidence and improve affected evals - #1079
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR updates the Vally adapter and related tooling/docs to treat expect_activation: false scenarios as activation-contract evidence (dormancy must hold) while excluding them from preference-gate evidence (sign test / net win), introducing adapter schema version 4 and propagating the new semantics through reporting, dashboards, and eval-quality checks.
Changes:
- Introduces schema v4 evidence split: retains dormancy scenarios as
excludedScenarioEvidenceand enforces anactivationContract, while excluding them fromsignTest,netWin, and stimulus vote counts. - Updates reporting/rendering (PR comment consolidation + dashboard) to surface dormancy exclusions and activation-contract failures distinctly.
- Updates eval-quality power checks, self-tests, and repository guidance to define “≥5 stimuli” as preference-eligible stimuli, with dormancy counted separately.
Show a summary per file
| File | Description |
|---|---|
| eng/vally-adapter/README.md | Documents schema v4’s preference-vs-activation split and updated decision rule. |
| eng/vally-adapter/InvestigatingResults.md | Updates triage guidance and schema field semantics for activation contracts and excluded evidence. |
| eng/vally-adapter/consolidate.test.mjs | Adds coverage for separate reporting of dormancy exclusions vs activation-contract failures. |
| eng/vally-adapter/consolidate.mjs | Renders activation-contract failure status, shows dormancy exclusion counts, and annotates scenario preference eligibility. |
| eng/vally-adapter/adapt.test.mjs | Adds tests for dormancy YAML parsing and for schema v4 preference exclusion + activation-contract gating behavior. |
| eng/vally-adapter/adapt.mjs | Implements schema v4 evidence model: preference-eligible vote set, excludedScenarioEvidence, activationContract, and updated gating. |
| eng/eval-quality/selftest_eval_quality.py | Adds a self-test ensuring dormancy evidence does not satisfy the preference floor. |
| eng/eval-quality/README.md | Updates dormancy guidance and the “≥5 stimuli” rationale to preference-eligible stimuli. |
| eng/eval-quality/check_eval_quality.py | Adjusts power checks and reporting to count preference stimuli vs dormancy contracts separately. |
| eng/dashboard/generate-benchmark-data.ps1 | Extends benchmark JSON to include dormancy exclusion count and activationContract; annotates scenario eligibility. |
| eng/dashboard/dashboard.js | Updates UI wording/logic for preference-eligible votes, dormancy exclusions, and activation-contract failures. |
| CONTRIBUTING.md | Updates contributor guidance to define dormancy as an activation contract and to clarify preference floor semantics. |
| .agents/skills/improve-skill-quality/SKILL.md | Updates skill-triage guidance for schema v4 dormancy and preference handling. |
| .agents/skills/improve-skill-quality/references/eval-triage.md | Updates reference triage table to reflect schema v4 evidence split and recommended interpretation. |
| .agents/skills/create-skill-test/SKILL.md | Updates eval-authoring guidance for dormancy contracts and preference-eligible stimulus floor. |
Review details
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 15/15 changed files
- Comments generated: 1
- Review effort level: Lite
Default scenarios without schema v4 preference eligibility metadata to eligible so historical schema v3 records remain consistent with their original gate evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 808c8105-e596-42c6-9dca-4c62cfcd490f
There was a problem hiding this comment.
Review details
Suppressed comments (2)
Previously missed (1) — in code that hasn't changed since the last review.
eng/vally-adapter/consolidate.mjs:293
resultLabel()checksisIndeterminate()(which treatsunderpowered: trueas indeterminate) beforehasActivationContractFailure(). If an activation contract fails while preference is underpowered (a state the adapter explicitly produces), this will display as “⚠️ Underpowered” instead of “⛔ Activation contract failed”.
This issue also appears on line 406 of the same file.
}
if (verdictState(verdict) === STATE.PASS) return "✅ Improved";
if (isObjectiveRegression(verdict)) return "🔻 Objective regression";
if (hasActivationContractFailure(verdict)) return "⛔ Activation contract failed";
if (isPreferenceRegression(verdict)) return "📉 Preference loss (report only)";
eng/vally-adapter/consolidate.mjs:410
activationContractFailureCountexcludes indeterminate verdicts, butverdictState()marksunderpowered: trueas indeterminate. That means activation-contract failures that are also preference-underpowered (which the adapter can emit) get counted as underpowered/invalid instead of as activation-contract failures.
const activationContractFailureCount = verdicts.filter(
(verdict) => !isIndeterminate(verdict)
&& !isObjectiveRegression(verdict)
&& hasActivationContractFailure(verdict),
).length;
- Files reviewed: 15/15 changed files
- Comments generated: 0 new
- Review effort level: Lite
Exercise the combined activation-contract failure and underpowered preference state so consolidation keeps the routing failure as the headline result. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 808c8105-e596-42c6-9dca-4c62cfcd490f
GitHub Actions
Failure
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
github-actions
Bot
added
the
pr-state/ready-for-eval
There was a problem hiding this comment.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
eng/vally-adapter/consolidate.mjs:462
- The summary line always prints "activation contract failures" even when the count is 1 ("1 activation contract failures"), which is a user-facing grammatical error. Use the existing countNoun() helper here so the noun pluralizes correctly.
+ `⛔ **${activationContractFailureCount} activation contract failures**, `
- Files reviewed: 15/15 changed files
- Comments generated: 0 new
- Review effort level: Lite
|
❌ Evaluation did not complete successfully (the evaluate job reported 12 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete. |
GitHub Actions
Failure
Resolve the Vally adapter README conflict by preserving executor timeout retry and dormancy activation-contract guidance in the combined decision flow. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 808c8105-e596-42c6-9dca-4c62cfcd490f
Require complete YAML false tokens so prefix values cannot alter preference eligibility, and keep activation-contract summary grammar correct. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 808c8105-e596-42c6-9dca-4c62cfcd490f
|
/evaluate 86ab860 |
github-actions
Bot
added
pr-state/ready-for-eval
AbhitejJohn
left a comment
AbhitejJohn
left a comment
There was a problem hiding this comment.
Blocking correctness fixes are required before merge. Schema v4 makes dormancy parsing and reporting gate-critical, but the parser still has silent fail-open forms and one report path quotes evidence that the gate excludes. The new xUnit migration grader also rejects valid MSBuild Boolean casing. Each inline comment gives a concrete failure case and fix.
|
There is one additional reporting regression in the per-shard job summary at Schema v4 changes the result semantics, but this summary still uses the schema-v3 presentation:
The verdict and merge-gate calculation remain correct; this regression affects the shard summary's observability and can send an investigator toward preference quality instead of the real routing violation. Please update this summary to use the v4 terminology, render |
Match PyYAML dormancy parsing, preserve unmatched annotations, prioritize gate-eligible report evidence, and make the MTP Boolean grader exact without case sensitivity. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 808c8105-e596-42c6-9dca-4c62cfcd490f
Skill Coverage Report
Uncovered:
|
|
❌ Evaluation did not complete successfully (the evaluate job reported 31 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete. |
|
/evaluate 47bbc32 |
📊 Skill Evaluation Results32 model/skill results across 16 skills and 2 models — ✅ 17 improved, ➖ 15 not proven improved, Measurement identity: evaluated commit Measurement health: 32 expected / 32 observed / 32 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when the aggregate net win across distinct-stimulus votes is at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
➖ Not proven improved — coverage-analysis (claude-sonnet-4.6)Why: Net win +15.4% (4W/7T/2L over 13 stimulus vote(s), sign test p=0.344), mean preference +16.9% across 26 paired run(s) — not credible (sign test p=0.344 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=13; 4W/7T/2L; d=6; p=0.344; net +15.4% Overfit: Moderate (score 0.31) Repeated-run reliability (not used by the gate): 26 paired runs (12W/7T/7L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — coverage-analysis (gpt-5.6-luna)Why: Net win +30.8% (6W/5T/2L over 13 stimulus vote(s), sign test p=0.145), mean preference +9.2% across 26 paired run(s) — not credible (sign test p=0.145 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=13; 6W/5T/2L; d=8; p=0.145; net +30.8% Overfit: Low (score 0.13) Repeated-run reliability (not used by the gate): 26 paired runs (9W/14T/3L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — generate-testability-wrappers (claude-sonnet-4.6)Why: Net win +62.5% (6W/1T/1L over 8 stimulus vote(s), sign test p=0.063), mean preference +40.0% across 24 paired run(s) — not credible (sign test p=0.063 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5% Overfit: Moderate (score 0.41) Repeated-run reliability (not used by the gate): 24 paired runs (17W/5T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — migrate-static-to-wrapper (claude-sonnet-4.6)Why: Net win +33.3% (5W/2T/2L over 9 stimulus vote(s), sign test p=0.227), mean preference +26.7% across 9 paired run(s) — not credible (sign test p=0.227 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=9; 5W/2T/2L; d=7; p=0.227; net +33.3% Warnings: Activation: isolated 9/9; plugin 7/9 Overfit: Moderate (score 0.27) Repeated-run reliability (not used by the gate): 9 paired runs (5W/2T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — migrate-static-to-wrapper (gpt-5.6-luna)Why: Net win +44.4% (4W/5T/0L over 9 stimulus vote(s), sign test p=0.063), mean preference +24.4% across 9 paired run(s) — not credible — 5 of 9 stimulus vote(s) tied, leaving only 4 discordant stimulus vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment. State: Gate evidence: n=9; 4W/5T/0L; d=4; p=0.063; net +44.4% Overfit: Low (score 0.15) Repeated-run reliability (not used by the gate): 9 paired runs (4W/5T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — migrate-xunit-to-mstest (gpt-5.6-luna)Why: Net win +23.1% (5W/6T/2L over 13 stimulus vote(s), sign test p=0.227), mean preference +13.8% across 13 paired run(s) — not credible (sign test p=0.227 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=13; 5W/6T/2L; d=7; p=0.227; net +23.1% Overfit: Low (score 0.14) Repeated-run reliability (not used by the gate): 13 paired runs (5W/6T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — migrate-xunit-to-xunit-v3 (gpt-5.6-luna)Why: Net win +8.3% (4W/5T/3L over 12 stimulus vote(s), sign test p=0.500), mean preference -1.7% across 12 paired run(s) — not credible (sign test p=0.500 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=12; 4W/5T/3L; d=7; p=0.500; net +8.3% Overfit: Low (score 0.11) Repeated-run reliability (not used by the gate): 12 paired runs (4W/5T/3L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — platform-detection (gpt-5.6-luna)Why: Net win +13.3% (4W/9T/2L over 15 stimulus vote(s), sign test p=0.344), mean preference +5.3% across 30 paired run(s) — not credible (sign test p=0.344 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=15; 4W/9T/2L; d=6; p=0.344; net +13.3% Overfit: Low (score 0.19) Repeated-run reliability (not used by the gate): 30 paired runs (7W/20T/3L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — scaffold-dotnet-test-project (claude-sonnet-4.6)Why: Net win +55.6% (6W/2T/1L over 9 stimulus vote(s), sign test p=0.063), mean preference +35.6% across 9 paired run(s) — not credible (sign test p=0.063 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=9; 6W/2T/1L; d=7; p=0.063; net +55.6% Warnings: Activation: isolated 7/9; plugin 6/9 Overfit: Low (score 0.17) Repeated-run reliability (not used by the gate): 9 paired runs (6W/2T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-anti-patterns (claude-sonnet-4.6)Why: Net win +44.4% (5W/3T/1L over 9 stimulus vote(s), sign test p=0.109), mean preference +24.4% across 9 paired run(s) — not credible (sign test p=0.109 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=9; 5W/3T/1L; d=6; p=0.109; net +44.4% Overfit: Moderate (score 0.26) Repeated-run reliability (not used by the gate): 9 paired runs (5W/3T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-anti-patterns (gpt-5.6-luna)Why: Net win +55.6% (7W/0T/2L over 9 stimulus vote(s), sign test p=0.090), mean preference +22.2% across 9 paired run(s) — not credible (sign test p=0.090 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=9; 7W/0T/2L; d=9; p=0.090; net +55.6% Overfit: Low (score 0.06) Repeated-run reliability (not used by the gate): 9 paired runs (7W/0T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-gap-analysis (claude-sonnet-4.6)Why: Net win +33.3% (4W/4T/1L over 9 stimulus vote(s), sign test p=0.188), mean preference +20.0% across 18 paired run(s) — not credible (sign test p=0.188 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=9; 4W/4T/1L; d=5; p=0.188; net +33.3% Overfit: Moderate (score 0.35) Repeated-run reliability (not used by the gate): 18 paired runs (9W/6T/3L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-gap-analysis (gpt-5.6-luna)Why: Net win +44.4% (5W/3T/1L over 9 stimulus vote(s), sign test p=0.109), mean preference +13.3% across 18 paired run(s) — not credible (sign test p=0.109 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=9; 5W/3T/1L; d=6; p=0.109; net +44.4% Overfit: Low (score 0.12) Repeated-run reliability (not used by the gate): 18 paired runs (9W/6T/3L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-smell-detection (gpt-5.6-luna)Why: Net win +50.0% (7W/1T/2L over 10 stimulus vote(s), sign test p=0.090), mean preference +32.0% across 10 paired run(s) — not credible (sign test p=0.090 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=10; 7W/1T/2L; d=9; p=0.090; net +50.0% Overfit: Moderate (score 0.29) Repeated-run reliability (not used by the gate): 10 paired runs (7W/1T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — testability-obstacle (claude-sonnet-4.6)Why: Net win +62.5% (6W/1T/1L over 8 stimulus vote(s), sign test p=0.063), mean preference +23.7% across 16 paired run(s) — not credible (sign test p=0.063 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5% Overfit: High (score 0.62) Repeated-run reliability (not used by the gate): 16 paired runs (10W/4T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — assertion-quality (claude-sonnet-4.6)Why: Net win +87.5% (7W/1T/0L over 8 stimulus vote(s), sign test p=0.008), mean preference +51.3% across 16 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 7W/1T/0L; d=7; p=0.008; net +87.5% Overfit: Moderate (score 0.25) Repeated-run reliability (not used by the gate): 16 paired runs (14W/1T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — detect-static-dependencies (claude-sonnet-4.6)Why: Net win +87.5% (7W/1T/0L over 8 stimulus vote(s), sign test p=0.008), mean preference +42.5% across 8 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 7W/1T/0L; d=7; p=0.008; net +87.5% Overfit: Moderate (score 0.34) Repeated-run reliability (not used by the gate): 8 paired runs (7W/1T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-mstest-v3-to-v4 (claude-sonnet-4.6)Why: Net win +73.3% (13W/0T/2L over 15 stimulus vote(s), sign test p=0.004), mean preference +69.3% across 15 paired run(s) — credibly better Next action: Fix activation gaps; Review overfit evidence. State: Gate evidence: n=15; 13W/0T/2L; d=15; p=0.004; net +73.3% Warnings: Activation: isolated 15/15; plugin 14/15 Overfit: Moderate (score 0.39) Repeated-run reliability (not used by the gate): 15 paired runs (13W/0T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-mstest-v3-to-v4 (gpt-5.6-luna)Why: Net win +46.7% (9W/4T/2L over 15 stimulus vote(s), sign test p=0.033), mean preference +34.7% across 15 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=15; 9W/4T/2L; d=11; p=0.033; net +46.7% Overfit: Moderate (score 0.24) Repeated-run reliability (not used by the gate): 15 paired runs (9W/4T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-xunit-to-mstest (claude-sonnet-4.6)Why: Net win +69.2% (11W/0T/2L over 13 stimulus vote(s), sign test p=0.011), mean preference +46.2% across 13 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=13; 11W/0T/2L; d=13; p=0.011; net +69.2% Overfit: Moderate (score 0.37) Repeated-run reliability (not used by the gate): 13 paired runs (11W/0T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-xunit-to-xunit-v3 (claude-sonnet-4.6)Why: Net win +100.0% (12W/0T/0L over 12 stimulus vote(s), sign test p=0.000), mean preference +80.0% across 12 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=12; 12W/0T/0L; d=12; p=0.000; net +100.0% Overfit: Moderate (score 0.21) Repeated-run reliability (not used by the gate): 12 paired runs (12W/0T/0L). ✅ Improved — platform-detection (claude-sonnet-4.6)Why: Net win +53.3% (10W/3T/2L over 15 stimulus vote(s), sign test p=0.019), mean preference +35.3% across 30 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=15; 10W/3T/2L; d=12; p=0.019; net +53.3% Overfit: Moderate (score 0.32) Repeated-run reliability (not used by the gate): 30 paired runs (20W/6T/4L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — run-tests (claude-sonnet-4.6)Why: Net win +45.0% (12W/5T/3L over 20 stimulus vote(s), sign test p=0.018), mean preference +45.0% across 20 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=20; 12W/5T/3L; d=15; p=0.018; net +45.0% Overfit: Moderate (score 0.37) Repeated-run reliability (not used by the gate): 20 paired runs (12W/5T/3L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — run-tests (gpt-5.6-luna)Why: Net win +40.0% (10W/8T/2L over 20 stimulus vote(s), sign test p=0.019), mean preference +28.0% across 20 paired run(s) — credibly better Next action: Fix activation gaps; Review overfit evidence. State: Gate evidence: n=20; 10W/8T/2L; d=12; p=0.019; net +40.0% Warnings: Activation: isolated 18/20; plugin 19/20 Overfit: Moderate (score 0.27) Repeated-run reliability (not used by the gate): 20 paired runs (10W/8T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-smell-detection (claude-sonnet-4.6)Why: Net win +70.0% (8W/1T/1L over 10 stimulus vote(s), sign test p=0.020), mean preference +52.0% across 10 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=10; 8W/1T/1L; d=9; p=0.020; net +70.0% Overfit: Moderate (score 0.26) Repeated-run reliability (not used by the gate): 10 paired runs (8W/1T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — testability-obstacle (gpt-5.6-luna)Why: Net win +62.5% (5W/3T/0L over 8 stimulus vote(s), sign test p=0.031), mean preference +20.0% across 16 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 5W/3T/0L; d=5; p=0.031; net +62.5% Overfit: Moderate (score 0.25) Repeated-run reliability (not used by the gate): 16 paired runs (10W/4T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — writing-mstest-tests (claude-sonnet-4.6)Why: Net win +71.4% (11W/2T/1L over 14 stimulus vote(s), sign test p=0.003), mean preference +50.0% across 14 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=14; 11W/2T/1L; d=12; p=0.003; net +71.4% Overfit: Moderate (score 0.45) Repeated-run reliability (not used by the gate): 14 paired runs (11W/2T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. Routine passing details for 5 results are in Full Results. 🔍 Full Results - all metrics and investigation details
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
|
/evaluate 47bbc32 |
github-actions
Bot
added
waiting-on-author
|
👋 @Evangelink — this PR has changes requested. When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the |
|
❌ Evaluation did not complete successfully (the evaluate job reported 32 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete. |
📊 Skill Evaluation Results32 model/skill results across 16 skills and 2 models — ✅ 21 improved, ➖ 11 not proven improved, Measurement identity: evaluated commit Measurement health: 32 expected / 32 observed / 32 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when the aggregate net win across distinct-stimulus votes is at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
➖ Not proven improved — coverage-analysis (claude-sonnet-4.6)Why: Net win +23.1% (7W/2T/4L over 13 stimulus vote(s), sign test p=0.274), mean preference +18.5% across 26 paired run(s) — not credible (sign test p=0.274 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=13; 7W/2T/4L; d=11; p=0.274; net +23.1% Overfit: Moderate (score 0.30) Repeated-run reliability (not used by the gate): 26 paired runs (14W/7T/5L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — detect-static-dependencies (gpt-5.6-luna)Why: Net win +37.5% (5W/1T/2L over 8 stimulus vote(s), sign test p=0.227), mean preference +15.0% across 8 paired run(s) — not credible (sign test p=0.227 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=8; 5W/1T/2L; d=7; p=0.227; net +37.5% Overfit: Low (score 0.18) Repeated-run reliability (not used by the gate): 8 paired runs (5W/1T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — migrate-mstest-v3-to-v4 (claude-sonnet-4.6)Why: Net win +46.7% (11W/0T/4L over 15 stimulus vote(s), sign test p=0.059), mean preference +42.7% across 15 paired run(s) — not credible (sign test p=0.059 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=15; 11W/0T/4L; d=15; p=0.059; net +46.7% Warnings: Activation: isolated 15/15; plugin 14/15 Overfit: Moderate (score 0.42) Repeated-run reliability (not used by the gate): 15 paired runs (11W/0T/4L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — migrate-mstest-v3-to-v4 (gpt-5.6-luna)Why: Net win +40.0% (9W/3T/3L over 15 stimulus vote(s), sign test p=0.073), mean preference +16.0% across 15 paired run(s) — not credible (sign test p=0.073 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=15; 9W/3T/3L; d=12; p=0.073; net +40.0% Overfit: Low (score 0.11) Repeated-run reliability (not used by the gate): 15 paired runs (9W/3T/3L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — migrate-static-to-wrapper (claude-sonnet-4.6)Why: Net win +44.4% (6W/1T/2L over 9 stimulus vote(s), sign test p=0.145), mean preference +31.1% across 9 paired run(s) — not credible (sign test p=0.145 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=9; 6W/1T/2L; d=8; p=0.145; net +44.4% Overfit: Moderate (score 0.20) Repeated-run reliability (not used by the gate): 9 paired runs (6W/1T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — migrate-static-to-wrapper (gpt-5.6-luna)Why: Net win +33.3% (3W/6T/0L over 9 stimulus vote(s), sign test p=0.125), mean preference +20.0% across 9 paired run(s) — not credible — 6 of 9 stimulus vote(s) tied, leaving only 3 discordant stimulus vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment. State: Gate evidence: n=9; 3W/6T/0L; d=3; p=0.125; net +33.3% Overfit: Low (score 0.15) Repeated-run reliability (not used by the gate): 9 paired runs (3W/6T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — migrate-xunit-to-xunit-v3 (gpt-5.6-luna)Why: Net win +25.0% (7W/1T/4L over 12 stimulus vote(s), sign test p=0.274), mean preference +20.0% across 12 paired run(s) — not credible (sign test p=0.274 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=12; 7W/1T/4L; d=11; p=0.274; net +25.0% Overfit: Low (score 0.15) Repeated-run reliability (not used by the gate): 12 paired runs (7W/1T/4L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — scaffold-dotnet-test-project (gpt-5.6-luna)Why: Net win +22.2% (4W/3T/2L over 9 stimulus vote(s), sign test p=0.344), mean preference +8.9% across 9 paired run(s) — not credible (sign test p=0.344 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=9; 4W/3T/2L; d=6; p=0.344; net +22.2% Overfit: Low (score 0.06) Repeated-run reliability (not used by the gate): 9 paired runs (4W/3T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-gap-analysis (claude-sonnet-4.6)Why: Net win +22.2% (5W/1T/3L over 9 stimulus vote(s), sign test p=0.363), mean preference +27.8% across 18 paired run(s) — not credible (sign test p=0.363 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=9; 5W/1T/3L; d=8; p=0.363; net +22.2% Overfit: Moderate (score 0.33) Repeated-run reliability (not used by the gate): 18 paired runs (10W/3T/5L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — test-gap-analysis (gpt-5.6-luna)Why: Net win +55.6% (6W/2T/1L over 9 stimulus vote(s), sign test p=0.063), mean preference +23.3% across 18 paired run(s) — not credible (sign test p=0.063 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=9; 6W/2T/1L; d=7; p=0.063; net +55.6% Overfit: Low (score 0.13) Repeated-run reliability (not used by the gate): 18 paired runs (11W/5T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ➖ Not proven improved — writing-mstest-tests (gpt-5.6-luna)Why: Net win +35.7% (6W/7T/1L over 14 stimulus vote(s), sign test p=0.063), mean preference +10.0% across 14 paired run(s) — not credible (sign test p=0.063 > 0.05) Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=14; 6W/7T/1L; d=7; p=0.063; net +35.7% Warnings: Activation: isolated 14/14; plugin 13/14 Overfit: Low (score 0.17) Repeated-run reliability (not used by the gate): 14 paired runs (6W/7T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — assertion-quality (claude-sonnet-4.6)Why: Net win +100.0% (8W/0T/0L over 8 stimulus vote(s), sign test p=0.004), mean preference +41.3% across 16 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 8W/0T/0L; d=8; p=0.004; net +100.0% Overfit: Moderate (score 0.23) Repeated-run reliability (not used by the gate): 16 paired runs (15W/1T/0L). ✅ Improved — detect-static-dependencies (claude-sonnet-4.6)Why: Net win +87.5% (7W/1T/0L over 8 stimulus vote(s), sign test p=0.008), mean preference +65.0% across 8 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 7W/1T/0L; d=7; p=0.008; net +87.5% Overfit: Moderate (score 0.24) Repeated-run reliability (not used by the gate): 8 paired runs (7W/1T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — generate-testability-wrappers (claude-sonnet-4.6)Why: Net win +75.0% (7W/0T/1L over 8 stimulus vote(s), sign test p=0.035), mean preference +41.7% across 24 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 7W/0T/1L; d=8; p=0.035; net +75.0% Overfit: Moderate (score 0.44) Repeated-run reliability (not used by the gate): 24 paired runs (16W/5T/3L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-xunit-to-mstest (claude-sonnet-4.6)Why: Net win +76.9% (11W/1T/1L over 13 stimulus vote(s), sign test p=0.003), mean preference +49.2% across 13 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=13; 11W/1T/1L; d=12; p=0.003; net +76.9% Overfit: Moderate (score 0.44) Repeated-run reliability (not used by the gate): 13 paired runs (11W/1T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — migrate-xunit-to-xunit-v3 (claude-sonnet-4.6)Why: Net win +100.0% (12W/0T/0L over 12 stimulus vote(s), sign test p=0.000), mean preference +80.0% across 12 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=12; 12W/0T/0L; d=12; p=0.000; net +100.0% Overfit: Moderate (score 0.21) Repeated-run reliability (not used by the gate): 12 paired runs (12W/0T/0L). ✅ Improved — platform-detection (claude-sonnet-4.6)Why: Net win +73.3% (12W/2T/1L over 15 stimulus vote(s), sign test p=0.002), mean preference +42.7% across 30 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=15; 12W/2T/1L; d=13; p=0.002; net +73.3% Overfit: Moderate (score 0.22) Repeated-run reliability (not used by the gate): 30 paired runs (22W/6T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — platform-detection (gpt-5.6-luna)Why: Net win +40.0% (7W/7T/1L over 15 stimulus vote(s), sign test p=0.035), mean preference +14.0% across 30 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=15; 7W/7T/1L; d=8; p=0.035; net +40.0% Overfit: Moderate (score 0.23) Repeated-run reliability (not used by the gate): 30 paired runs (11W/17T/2L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — run-tests (claude-sonnet-4.6)Why: Net win +70.0% (15W/4T/1L over 20 stimulus vote(s), sign test p=0.000), mean preference +61.0% across 20 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=20; 15W/4T/1L; d=16; p=0.000; net +70.0% Overfit: Moderate (score 0.42) Repeated-run reliability (not used by the gate): 20 paired runs (15W/4T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — run-tests (gpt-5.6-luna)Why: Net win +60.0% (13W/6T/1L over 20 stimulus vote(s), sign test p=0.001), mean preference +36.0% across 20 paired run(s) — credibly better Next action: Fix activation gaps; Review overfit evidence. State: Gate evidence: n=20; 13W/6T/1L; d=14; p=0.001; net +60.0% Warnings: Activation: isolated 17/20; plugin 18/20 Overfit: Moderate (score 0.27) Repeated-run reliability (not used by the gate): 20 paired runs (13W/6T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — scaffold-dotnet-test-project (claude-sonnet-4.6)Why: Net win +66.7% (7W/1T/1L over 9 stimulus vote(s), sign test p=0.035), mean preference +40.0% across 9 paired run(s) — credibly better Next action: Fix activation gaps. State: Gate evidence: n=9; 7W/1T/1L; d=8; p=0.035; net +66.7% Warnings: Activation: isolated 8/9; plugin 6/9 Overfit: Low (score 0.18) Repeated-run reliability (not used by the gate): 9 paired runs (7W/1T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — test-anti-patterns (claude-sonnet-4.6)Why: Net win +88.9% (8W/1T/0L over 9 stimulus vote(s), sign test p=0.004), mean preference +48.9% across 9 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=9; 8W/1T/0L; d=8; p=0.004; net +88.9% Overfit: Moderate (score 0.27) Repeated-run reliability (not used by the gate): 9 paired runs (8W/1T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — testability-obstacle (claude-sonnet-4.6)Why: Net win +87.5% (7W/1T/0L over 8 stimulus vote(s), sign test p=0.008), mean preference +37.5% across 16 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=8; 7W/1T/0L; d=7; p=0.008; net +87.5% Overfit: Moderate (score 0.48) Repeated-run reliability (not used by the gate): 16 paired runs (13W/2T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — testability-obstacle (gpt-5.6-luna)Why: Net win +62.5% (5W/3T/0L over 8 stimulus vote(s), sign test p=0.031), mean preference +30.0% across 16 paired run(s) — credibly better Next action: Fix activation gaps. State: Gate evidence: n=8; 5W/3T/0L; d=5; p=0.031; net +62.5% Warnings: Activation: isolated 8/8; plugin 7/8 Overfit: Low (score 0.17) Repeated-run reliability (not used by the gate): 16 paired runs (9W/7T/0L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. ✅ Improved — writing-mstest-tests (claude-sonnet-4.6)Why: Net win +85.7% (13W/0T/1L over 14 stimulus vote(s), sign test p=0.001), mean preference +51.4% across 14 paired run(s) — credibly better Next action: Review overfit evidence. State: Gate evidence: n=14; 13W/0T/1L; d=14; p=0.001; net +85.7% Overfit: Moderate (score 0.48) Repeated-run reliability (not used by the gate): 14 paired runs (13W/0T/1L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. Routine passing details for 7 results are in Full Results. 🔍 Full Results - all metrics and investigation details
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
|
@AbhitejJohn all comments have been addressed, there are still 11 evals that needs tweaking but I'd like to merge this one as-is and then look at how to improve the rest. Please approve or suggest other changes! |
Summary
Correctly dormant skilled arms are intentionally behaviorally equivalent to baseline, so their paired preference is a tie or judge noise by construction. Those scenarios were still voting in the exact sign test and could turn a valid skill improvement into a random preference loss.
This change introduces adapter schema version 4:
expect_activation: falsescenarios remain fully retained in raw trials, judge evidence, retry/error and pairing integrity,comparisonTrialEvidence, andcompletionTransitions, but are excluded fromsignTest,netWin, andstimulusVoteCount.excludedScenarioEvidenceand per-scenario preference eligibility make the exclusion explicit rather than silently dropping evidence.activationContractevaluates explicit dormancy against isolated target-skill activation. Unexpected activation blocks a pass and is reported distinctly; plugin activity remains diagnostic because it cannot identify the emitting skill.Evaluation-informed skill fixes
The first complete schema-v4 run produced 48 model/skill results: 24 improved, 20 measured but not proven improved, and 4 activation-contract failures. There were no invalid, underpowered, or preference-regression results. The follow-up reads the exact losing trial evidence and addresses all 16 unique non-passing skills without adding repeated runs or weakening graders.
coverage-analysisnow defers single named method/class/file CRAP work tocrap-score;generate-testability-wrappersstays dormant when code already consumes an interface or built-in abstraction.assertion-quality,detect-static-dependencies,test-anti-patterns,test-gap-analysis, andtest-smell-detectionnow enforce precise matcher semantics, reconciled occurrence counts, coherent severity/counting, complete branch/exception inventories, canonical mutation verification, and clean-framework stop conditions.migrate-static-to-wrapper,platform-detection,run-tests,scaffold-dotnet-test-project,testability-obstacle, andwriting-mstest-testsnow preserve clock-read semantics, distinguish runner selection from command integration, use correct MTP/VSTest syntax, validate the exact requested tree, require executed seam tests, and use version-correct MSTest assertions/cancellation APIs.migrate-mstest-v3-to-v4,migrate-xunit-to-mstest, andmigrate-xunit-to-xunit-v3now resolve package versions, preserve VSTest/MTP intentionally, verify no-op migrations, retain parallelization and base-class behavior, and avoid unrelated file or version changes.Internal rubber-duck and code-review passes identified and drove corrections for MTP-to-MSTest translation, transitive Test.Sdk handling, no-op package-only detection, alternate MSBuild property locations, parameter-name-independent graders,
TestContext.CancellationTokenavailability, and mechanical-versus-behavioral clock guidance.Compatibility and migration
expect_activation: falseautomatically opts into activation-contract-only evidence.wins,ties,losses,winRate,stimulusVoteCount, andtrialCount) mean preference-eligible evidence. Consumers needing an all-stimulus view should combinescenarioEvidencewithexcludedScenarioEvidence, or readscenarios[]/comparisonTrialEvidence.Related issue
N/A
Validation
node --test eng\vally-adapter\adapt.test.mjs eng\vally-adapter\consolidate.test.mjs eng\vally-adapter\retry-executor-timeouts.test.mjs eng\dashboard\dashboard-freshness.test.js— 72 passedpython eng\eval-quality\selftest_eval_quality.py— 27 passedpython eng\eval-quality\check_eval_quality.py— 98 specs checked, no errorsdotnet run --project eng\skill-validator\src\SkillValidator.csproj -p:CopilotSkipCliDownload=true -- check --plugin .\plugins\dotnet-test— passeddotnet run --project eng\skill-validator\src\SkillValidator.csproj -p:CopilotSkipCliDownload=true -- check --plugin .\plugins\dotnet-test-migration— passedgit diff --checkpassedChecklist
eng/known-domains.txtfor any new external domains referenced by skill content.