Harden evaluation result schema handling - #1085
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
Review tier: Lite
Findings: 1
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
 |
eng/skill-validator/src/Evaluate/Models.cs — SchemaVersion currently treats an explicit JSON value of 0 as equivalent to the current legacy… |
 |
eng/skill-validator/src/Evaluate/ConsolidateCommand.cs — The catch block now also handles schema validation failures (e.g., unsupported… |
What changed in this PR
This PR hardens evaluation result/schema handling across the Node-based Vally adapter and the retired C# skill-validator evaluate pipeline, aiming to fail closed on unreadable or unsupported inputs and clearly document schema ownership boundaries.
Changes:
- Make the Vally adapter emit an explicit
eval_spec_unreadablemeasurement-invalid verdict (and skip comparison) when an eval spec cannot be read. - Introduce explicit legacy schema identity (
schemaOwner: skill-validator,schemaVersion: 1) for newskill-validator evaluateoutputs while still accepting unversioned historical result files. - Update legacy consolidation to reject Vally adapter schemas and return a nonzero exit code when any input is malformed/unsupported, alongside updated investigation docs.
| File | Description |
|---|---|
| eng/vally-adapter/README.md | Documents fail-closed behavior when eval specs are unreadable. |
| eng/vally-adapter/InvestigatingResults.md | Explains eval_spec_unreadable as measurement-invalid and why the adapter cannot assume activation expectations. |
| eng/vally-adapter/adapt.test.mjs | Adds coverage for unreadable eval spec behavior and supports passing --repo-root in tests. |
| eng/vally-adapter/adapt.mjs | Throws/records eval_spec_unreadable invalid verdicts before any compare call when eval specs can’t be read. |
| eng/skill-validator/tests/Evaluate/ResultsSchemaTests.cs | Adds tests for legacy schema identity, unversioned compatibility, and Vally schema rejection in consolidation. |
| eng/skill-validator/src/Evaluate/Models.cs | Adds schema owner/version fields to legacy results/verdict models and schema validation helper. |
| eng/skill-validator/src/Evaluate/ConsolidateCommand.cs | Validates schema owner/version during consolidation and returns nonzero exit code on any bad input. |
| eng/skill-validator/src/docs/InvestigatingResults.md | Clarifies Vally vs legacy schema boundaries and documents consolidation’s failure behavior. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3db882d2-1706-4d79-a820-016d895a4992
There was a problem hiding this comment.
Copilot review overview
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
|
eng/skill-validator/tests/Evaluate/ResultsSchemaTests.cs — These tests treat schemaVersion: 4 as representative of the Vally adapter schema, but the adapter… |
Issues resolved since last review (2)
| Severity | Finding |
|---|---|
|
eng/skill-validator/src/Evaluate/ConsolidateCommand.cs — The catch block now also handles schema validation failures (e.g., unsupported… View resolved comment |
|
eng/skill-validator/src/Evaluate/Models.cs — SchemaVersion currently treats an explicit JSON value of 0 as equivalent to the current legacy… View resolved comment |
Suppressed comments (2)
Previously missed (1) — in code that hasn't changed since the last review.
eng/skill-validator/src/docs/InvestigatingResults.md:13
- This document states that Vally adapter results use
schemaVersion: 4, but the adapter’s own documentation and code currently indicateschemaVersion: 3forresults.json(seeeng/vally-adapter/InvestigatingResults.mdandeng/vally-adapter/adapt.mjs). The version here should match the actual emitted schema version to avoid confusing investigations and schema boundary discussions.
> **Vally schema:** Vally adapter results use an independently owned and
> versioned schema. Vally `schemaVersion: 4` records add separate
> activation-contract and preference-eligibility evidence. `state` is
> authoritative:
eng/skill-validator/tests/Evaluate/ResultsSchemaTests.cs:133
- This test uses
schemaVersion: 4as the sample Vally adapter schema, but the adapter currently emitsschemaVersion: 3forresults.json. Aligning the fixture version with the adapter’s current schema version makes this test reflect real-world consolidation input failures.
public async Task ConsolidationFailsForVallyAdapterResults()
{
var paths = CreateTempPaths();
try
{
await File.WriteAllTextAsync(
paths.Input,
"""{"schemaVersion":4,"verdicts":[]}""",
TestContext.Current.CancellationToken);
var exitCode = await ConsolidateCommand.Consolidate([paths.Input], paths.Output);
Assert.Equal(1, exitCode);
}
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3db882d2-1706-4d79-a820-016d895a4992
There was a problem hiding this comment.
Copilot review overview
Review tier: Lite
Findings: None
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
|
eng/skill-validator/tests/Evaluate/ResultsSchemaTests.cs — These tests treat schemaVersion: 4 as representative of the Vally adapter schema, but the adapter… View resolved comment |
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
eng/skill-validator/src/Evaluate/ConsolidateCommand.cs:74
- When one or more input files fail to read/deserialize/schema-validate, consolidation returns exit code 1 but still prints "Consolidated ...". That message reads like a fully successful run and can mislead users (especially in CI) even though the output is only a partial diagnostic summary.
var output = Reporter.GenerateMarkdownSummary(allVerdicts, model, judgeModel);
await File.WriteAllTextAsync(outputPath, output);
Console.WriteLine($"Consolidated {files.Length} result file(s) into {outputPath}");
return inputFailed ? 1 : 0;
eng/skill-validator/tests/Evaluate/ResultsSchemaTests.cs:165
- The test cleanup deletes the temp directory non-recursively and without checking existence. If any additional file is created (or a prior assertion fails before deleting expected files), this can throw during teardown and mask the real test failure. Using a recursive delete guarded by Directory.Exists makes cleanup more robust.
Directory.Delete(paths.Directory);
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3db882d2-1706-4d79-a820-016d895a4992
There was a problem hiding this comment.
Copilot review overview
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
|
eng/skill-validator/src/Evaluate/ConsolidateCommand.cs — consolidate treats a results file that deserializes to null (valid JSON literal null) as a… |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3db882d2-1706-4d79-a820-016d895a4992
There was a problem hiding this comment.
Copilot review overview
Review tier: Lite
Findings: None
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
|
eng/skill-validator/src/Evaluate/ConsolidateCommand.cs — consolidate treats a results file that deserializes to null (valid JSON literal null) as a… View resolved comment |
github-actions
Bot
added
pr-state/ready-for-eval
📊 Skill Evaluation Results22 model/skill results across 11 skills and 2 models — ✅ 8 improved, ➖ 9 not proven improved, Measurement identity: evaluated commit Measurement health: 22 expected / 22 observed / 22 written; 0 missing, 0 unexpected, 4 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when the aggregate net win across distinct-stimulus votes is at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
|
| Scenario | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|
| ▼ Enable NRT in ASP.NET Core Web API with EF Core | -100.0% | -40.0% | 0/0/1 |
| = Enable NRT in a small library with mixed nullability | +0.0% | +0.0% | 0/1/0 |
| = File-by-file migration: only modify the targeted file | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
Enable NRT in ASP.NET Core Web API with EF Core:Both compile cleanly and handle EF navigation and optional text fields well, but A preserves the important required-request API contract for Title and Isbn. Both have an equivalent, complementary defect in BookResponse nullability.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
⚠️ Underpowered — migrate-nullable-references (gpt-5.6-luna)
Why: Net win +0.0% (0W/3T/0L over 3 stimulus vote(s), sign test p=1.000), mean preference +0.0% across 3 paired run(s) — underpowered (3 counted stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth
Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.
State: INVALID_INCONCLUSIVE (underpowered)
Gate evidence: n=3; 0W/3T/0L; d=0; p=1.000; net +0.0%
Warnings: Activation: isolated 3/3; plugin 2/3
Overfit: Low (score 0.08)
Repeated-run reliability (not used by the gate): 3 paired runs (0W/3T/0L).
Weak or warning scenarios:
| Scenario | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|
| = Enable NRT in ASP.NET Core Web API with EF Core | +0.0% | +0.0% | 0/1/0 |
| = Enable NRT in a small library with mixed nullability | +0.0% | +0.0% | 0/1/0 |
| = File-by-file migration: only modify the targeted file | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
Enable NRT in ASP.NET Core Web API with EF Core:Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
📉 Preference loss (report only) — migrate-dotnet9-to-dotnet10 (gpt-5.6-luna)
Why: Net win -52.9% (1W/6T/10L over 17 stimulus vote(s), sign test p=0.006), mean preference -21.2% across 17 paired run(s) — credibly worse
Next action: Inspect losing stimuli and fix skill behavior; this is not objective completion proof.
State: VALID_NO_CHANGE (preference_regression_report_only)
Gate evidence: n=17; 1W/6T/10L; d=11; p=0.006; net -52.9%
Overfit: Low (score 0.08)
Repeated-run reliability (not used by the gate): 17 paired runs (1W/6T/10L).
Weak or warning scenarios:
| Scenario | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|
| ▼ ASP.NET Core app with OpenAPI transformers using Microsoft.OpenApi v1 APIs | -100.0% | -40.0% | 0/0/1 |
| ▼ ASP.NET Core app with WebHostBuilder, OpenAPI, and forwarded headers | -100.0% | -40.0% | 0/0/1 |
| = App using SslStream properties and SystemEvents | +0.0% | +0.0% | 0/1/0 |
| = Blazor WASM app with generic math shift masking and tar operations | +0.0% | +0.0% | 0/1/0 |
| ▼ C# 14 compiler breaking changes — field keyword, extension keyword, disposal | -100.0% | -40.0% | 0/0/1 |
| ▼ Console app with System.Linq.Async, SIGTERM, and BufferedStream | -100.0% | -40.0% | 0/0/1 |
| ▼ Containerized single-file app with P/Invoke and IDispatchEx | -100.0% | -40.0% | 0/0/1 |
| = Cryptography app with OpenSSL, X.509, and Rfc2898DeriveBytes | +0.0% | +0.0% | 0/1/0 |
| ▼ EF Core app with Azure SQL JSON columns and parameterized collections | -100.0% | -40.0% | 0/0/1 |
| = EF Core app with dynamic ExecuteUpdate and complex types | +0.0% | +0.0% | 0/1/0 |
| ▼ Expression tree code broken by C# 14 span overload resolution | -100.0% | -40.0% | 0/0/1 |
| = JSON polymorphism with conflicting property names and XmlSerializer | +0.0% | +0.0% | 0/1/0 |
| ▼ SDK and NuGet obscure tooling changes | -100.0% | -40.0% | 0/0/1 |
| ▼ SQLite app with DateTimeOffset timezone handling | -100.0% | -40.0% | 0/0/1 |
| ▼ WinForms and WPF desktop app with System.Drawing and DynamicResource | -100.0% | -40.0% | 0/0/1 |
| = Worker service with config null array binding and ProviderAlias assembly change | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
ASP.NET Core app with OpenAPI transformers using Microsoft.OpenApi v1 APIs:Both responses are accurate, cover all five rubric points, and give correct migration guidance. They're largely equivalent on most criteria. A is slightly better because it provides a complete, working rewrite of the schema transformer using JsonSchemaType.Null flag manipulati...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
➖ Not proven improved — migrate-dotnet10-to-dotnet11 (claude-sonnet-4.6)
Why: Net win +9.1% (6W/0T/5L over 11 stimulus vote(s), sign test p=0.500), mean preference +20.0% across 11 paired run(s) — not credible (sign test p=0.500 > 0.05)
Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.
State: VALID_NO_CHANGE (no_credible_preference_change)
Gate evidence: n=11; 6W/0T/5L; d=11; p=0.500; net +9.1%
Warnings: Activation: isolated 10/11; plugin 11/11
Overfit: Moderate (score 0.25)
Repeated-run reliability (not used by the gate): 11 paired runs (6W/0T/5L).
Weak or warning scenarios:
| Scenario | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|
| ▼ ASP.NET Core app with OpenAPI customizations and Blazor Virtualize | -100.0% | -40.0% | 0/0/1 |
| ▼ Basic TFM update with Docker and global.json | -100.0% | -40.0% | 0/0/1 |
| ▲ C# 15 dynamic operator and ref readonly delegate issues | +100.0% | +100.0% | 1/0/0 |
| ▼ Cryptography app using DSA on macOS | -100.0% | -40.0% | 0/0/1 |
| ▼ EF Core app with Cosmos DB provider using sync APIs | -100.0% | -40.0% | 0/0/1 |
| ▼ mTLS and AIA certificate chain validation | -100.0% | -40.0% | 0/0/1 |
Illustrative judge evidence:
ASP.NET Core app with OpenAPI customizations and Blazor Virtualize:Both responses answer both upgrade questions correctly and concisely. A is the stronger migration answer because it translates the OpenAPI version change into concrete likely fixes and gives more actionable context for Virtualize, while B is accurate but comparatively high-level.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
➖ Not proven improved — migrate-dotnet10-to-dotnet11 (gpt-5.6-luna)
Why: Net win -18.2% (1W/7T/3L over 11 stimulus vote(s), sign test p=0.312), mean preference -1.8% across 11 paired run(s) — no improvement
Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
State: VALID_NO_CHANGE (no_credible_preference_change)
Gate evidence: n=11; 1W/7T/3L; d=4; p=0.312; net -18.2%
Overfit: Low (score 0.07)
Repeated-run reliability (not used by the gate): 11 paired runs (1W/7T/3L).
Weak or warning scenarios:
| Scenario | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|
| = ASP.NET Core app with OpenAPI customizations and Blazor Virtualize | +0.0% | +0.0% | 0/1/0 |
| ▼ BackgroundService exceptions and ZipArchive CRC32 validation | -100.0% | -40.0% | 0/0/1 |
| = Basic TFM update with Docker and global.json | +0.0% | +0.0% | 0/1/0 |
| = C# 15 compiler breaking changes — Span safe-context, nameof, with() | +0.0% | +0.0% | 0/1/0 |
| = Console app with compression and TAR operations | +0.0% | +0.0% | 0/1/0 |
| ▼ Cryptography app using DSA on macOS | -100.0% | -40.0% | 0/0/1 |
| = Deployment to older hardware with minimum requirement changes | +0.0% | +0.0% | 0/1/0 |
| = EF Core SQL Server with Entra ID auth and Design package dependency | +0.0% | +0.0% | 0/1/0 |
| ▼ EF Core app with Cosmos DB provider using sync APIs | -100.0% | -40.0% | 0/0/1 |
| = mTLS and AIA certificate chain validation | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
ASP.NET Core app with OpenAPI customizations and Blazor Virtualize:Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
➖ Not proven improved — migrate-dotnet8-to-dotnet9 (gpt-5.6-luna)
Why: Net win +8.3% (4W/5T/3L over 12 stimulus vote(s), sign test p=0.500), mean preference +3.3% across 12 paired run(s) — not credible (sign test p=0.500 > 0.05)
Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.
State: VALID_NO_CHANGE (no_credible_preference_change)
Gate evidence: n=12; 4W/5T/3L; d=7; p=0.500; net +8.3%
Overfit: Low (score 0.05)
Repeated-run reliability (not used by the gate): 12 paired runs (4W/5T/3L).
Weak or warning scenarios:
| Scenario | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|
| = App with JsonDocument null deserialization and BinaryFormatter fallback | +0.0% | +0.0% | 0/1/0 |
| = C# 13 compiler breaking changes — InlineArray on record, iterator safe context, collection expressions | +0.0% | +0.0% | 0/1/0 |
| ▼ Containerized app with env var precedence reversal and zlib removal | -100.0% | -100.0% | 0/0/1 |
| = Containerized app with zlib dependency and runtime configuration | +0.0% | +0.0% | 0/1/0 |
| ▼ EF Core Cosmos DB app with discriminator and sync I/O | -100.0% | -40.0% | 0/0/1 |
| = EF Core Cosmos DB app with existing documents and composite id format | +0.0% | +0.0% | 0/1/0 |
| = Library with String.Trim span overload, keyed services, and InlineArray | +0.0% | +0.0% | 0/1/0 |
| ▼ WinForms app with custom UserControls and PictureBox URL loading | -100.0% | -40.0% | 0/0/1 |
Illustrative judge evidence:
App with JsonDocument null deserialization and BinaryFormatter fallback:Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
➖ Not proven improved — migrate-dotnet9-to-dotnet10 (claude-sonnet-4.6)
Why: Net win +41.2% (11W/2T/4L over 17 stimulus vote(s), sign test p=0.059), mean preference +41.2% across 17 paired run(s) — not credible (sign test p=0.059 > 0.05)
Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.
State: VALID_NO_CHANGE (no_credible_preference_change)
Gate evidence: n=17; 11W/2T/4L; d=15; p=0.059; net +41.2%
Warnings: Activation: isolated 16/17; plugin 17/17
Overfit: Low (score 0.05)
Repeated-run reliability (not used by the gate): 17 paired runs (11W/2T/4L).
Weak or warning scenarios:
| Scenario | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|
| ▼ Blazor WASM app with generic math shift masking and tar operations | -100.0% | -40.0% | 0/0/1 |
| ▼ Console app with System.Linq.Async, SIGTERM, and BufferedStream | -100.0% | -40.0% | 0/0/1 |
| ▼ Containerized single-file app with P/Invoke and IDispatchEx | -100.0% | -40.0% | 0/0/1 |
| = JSON polymorphism with conflicting property names and XmlSerializer | +0.0% | +0.0% | 0/1/0 |
| ▼ SDK and NuGet obscure tooling changes | -100.0% | -40.0% | 0/0/1 |
| = SQLite app with DateTimeOffset timezone handling | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
Blazor WASM app with generic math shift masking and tar operations:A is substantially more precise on browser streaming and HTTP/3/WASM context and is solid on TAR timestamps. However, it has a major error on the generic-shift result, whereas B gets that central behavior right; therefore A's aggregate advantage is only slight rather than large.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
➖ Not proven improved — migrate-mstest-v3-to-v4 (gpt-5.6-luna)
Why: Net win +20.0% (8W/2T/5L over 15 stimulus vote(s), sign test p=0.291), mean preference +16.0% across 15 paired run(s) — not credible (sign test p=0.291 > 0.05)
Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.
State: VALID_NO_CHANGE (no_credible_preference_change)
Gate evidence: n=15; 8W/2T/5L; d=13; p=0.291; net +20.0%
Overfit: Low (score 0.11)
Repeated-run reliability (not used by the gate): 15 paired runs (8W/2T/5L).
Weak or warning scenarios:
| Scenario | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|
| ▼ Correctly identify MSTest v3 project and recommend v4 migration | -100.0% | -40.0% | 0/0/1 |
| ▼ Fix Assert.IsInstanceOfType out parameter removal | -100.0% | -40.0% | 0/0/1 |
| ▼ Fix TestMethodAttribute CallerInfo constructor breaking change | -100.0% | -40.0% | 0/0/1 |
| = Fix multiple v4 breaking changes: Assert, ClassCleanup, TestContext, Timeout | +0.0% | +0.0% | 0/1/0 |
| = Full MSTest v3 to v4 migration with multiple breaking changes | +0.0% | +0.0% | 0/1/0 |
| ▼ Migrate MSTest.Sdk v3 project using ManagedType and TestTimeout | -100.0% | -40.0% | 0/0/1 |
| ▼ Understand behavioral changes after MSTest v4 upgrade | -100.0% | -40.0% | 0/0/1 |
Illustrative judge evidence:
Correctly identify MSTest v3 project and recommend v4 migration:Both are strong, accurate responses. A actually inspected the project, verified the true latest version from NuGet (4.3.3 vs B's documented 4.1.0), applied the ExpectedException migration to the real code, and validated it with a passing test run — providing concrete, validate...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
➖ Not proven improved — migrate-vstest-to-mtp (gpt-5.6-luna)
Why: Net win +54.5% (8W/1T/2L over 11 stimulus vote(s), sign test p=0.055), mean preference +32.7% across 11 paired run(s) — not credible (sign test p=0.055 > 0.05)
Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.
State: VALID_NO_CHANGE (no_credible_preference_change)
Gate evidence: n=11; 8W/1T/2L; d=10; p=0.055; net +54.5%
Overfit: Low (score 0.12)
Repeated-run reliability (not used by the gate): 11 paired runs (8W/1T/2L).
Weak or warning scenarios:
| Scenario | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|
| = Configure dotnet test MTP mode on .NET 10 SDK | +0.0% | +0.0% | 0/1/0 |
| ▼ Full VSTest to MTP migration plan for MSTest solution | -100.0% | -40.0% | 0/0/1 |
| ▼ Migrate xUnit.net VSTest filter syntax to MTP | -100.0% | -40.0% | 0/0/1 |
Illustrative judge evidence:
Configure dotnet test MTP mode on .NET 10 SDK:Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
➖ Not proven improved — migrate-xunit-to-mstest (gpt-5.6-luna)
Why: Net win +0.0% (4W/5T/4L over 13 stimulus vote(s), sign test p=0.637), mean preference +9.2% across 13 paired run(s) — no improvement
Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.
State: VALID_NO_CHANGE (no_credible_preference_change)
Gate evidence: n=13; 4W/5T/4L; d=8; p=0.637; net +0.0%
Overfit: Low (score 0.12)
Repeated-run reliability (not used by the gate): 13 paired runs (4W/5T/4L).
Weak or warning scenarios:
| Scenario | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|
| = Convert IClassFixture to ClassInitialize | +0.0% | +0.0% | 0/1/0 |
| ▼ Convert ITestOutputHelper to TestContext | -100.0% | -40.0% | 0/0/1 |
| = Convert MemberData and TheoryData to DynamicData | +0.0% | +0.0% | 0/1/0 |
| ▼ Convert Skip, Trait, and Timeout | -100.0% | -40.0% | 0/0/1 |
| = Handle ICollectionFixture explicitly (do not silently widen scope) | +0.0% | +0.0% | 0/1/0 |
| ▼ Migrate basic xUnit v3 project to MSTest v4 preserving VSTest | -100.0% | -40.0% | 0/0/1 |
| = Preserve type and sequence assertion semantics | +0.0% | +0.0% | 0/1/0 |
| ▼ Preserve xUnit parallelization default with [assembly: Parallelize] | -100.0% | -40.0% | 0/0/1 |
| = Recognize project already on MSTest | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
Convert IClassFixture to ClassInitialize:Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
➖ Not proven improved — migrate-xunit-to-xunit-v3 (gpt-5.6-luna)
Why: Net win +16.7% (6W/2T/4L over 12 stimulus vote(s), sign test p=0.377), mean preference +11.7% across 12 paired run(s) — not credible (sign test p=0.377 > 0.05)
Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.
State: VALID_NO_CHANGE (no_credible_preference_change)
Gate evidence: n=12; 6W/2T/4L; d=10; p=0.377; net +16.7%
Overfit: Low (score 0.10)
Repeated-run reliability (not used by the gate): 12 paired runs (6W/2T/4L).
Weak or warning scenarios:
| Scenario | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|
| ▼ Convert string-based attribute constructors to typeof syntax | -100.0% | -40.0% | 0/0/1 |
| = Migrate Xunit.SkippableFact to xUnit.net v3 built-in skip APIs | +0.0% | +0.0% | 0/1/0 |
| ▼ Migrate project with YTest.MTP.XUnit2 to xUnit.net v3 preserving MTP | -100.0% | -40.0% | 0/0/1 |
| ▼ Recognize project already on xUnit.net v3 — no migration needed | -100.0% | -40.0% | 0/0/1 |
| ▼ Update BeforeAfterTestAttribute overrides with IXunitTest parameter | -100.0% | -100.0% | 0/0/1 |
| = Update custom FactAttribute to include source information parameters | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
Convert string-based attribute constructors to typeof syntax:Both responses correctly converted both attributes to typeof() syntax, got clean builds, and passing tests, and neither fully explained the System.Type rationale. The key differentiator is behavioral correctness: Response A preserved the actual alphabetical ordering logic in i...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
➖ Not proven improved — thread-abort-migration (claude-sonnet-4.6)
Why: Net win +60.0% (3W/2T/0L over 5 stimulus vote(s), sign test p=0.125), mean preference +36.0% across 5 paired run(s) — not credible — 2 of 5 stimulus vote(s) tied, leaving only 3 discordant stimulus vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties
Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
State: VALID_NO_CHANGE (no_credible_preference_change)
Gate evidence: n=5; 3W/2T/0L; d=3; p=0.125; net +60.0%
Overfit: Moderate (score 0.37)
Repeated-run reliability (not used by the gate): 5 paired runs (3W/2T/0L).
Weak or warning scenarios:
| Scenario | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|
| = ASP.NET Response.End and Response.Redirect with Thread.Abort | +0.0% | +0.0% | 0/1/0 |
| = Blocking WaitHandle with Thread.Interrupt | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
ASP.NET Response.End and Response.Redirect with Thread.Abort:Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
✅ Improved — migrate-dotnet8-to-dotnet9 (claude-sonnet-4.6)
Why: Net win +91.7% (11W/1T/0L over 12 stimulus vote(s), sign test p=0.000), mean preference +71.7% across 12 paired run(s) — credibly better
Next action: Fix activation gaps; Review overfit evidence.
State: VALID_PASS (credible_preference_improvement)
Gate evidence: n=12; 11W/1T/0L; d=11; p=0.000; net +91.7%
Warnings: Activation: isolated 12/12; plugin 11/12
Overfit: Moderate (score 0.44)
Repeated-run reliability (not used by the gate): 12 paired runs (11W/1T/0L).
Weak or warning scenarios:
| Scenario | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|
| = App with JsonDocument null deserialization and BinaryFormatter fallback | +0.0% | +0.0% | 0/1/0 |
| ▲ CI pipeline with Terminal Logger parsing and version constraints | +100.0% | +100.0% | 1/0/0 |
Illustrative judge evidence:
App with JsonDocument null deserialization and BinaryFormatter fallback:Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
✅ Improved — migrate-mstest-v1v2-to-v3 (claude-sonnet-4.6)
Why: Net win +100.0% (11W/0T/0L over 11 stimulus vote(s), sign test p=0.000), mean preference +61.8% across 11 paired run(s) — credibly better
Next action: Review overfit evidence.
State: VALID_PASS (credible_preference_improvement)
Gate evidence: n=11; 11W/0T/0L; d=11; p=0.000; net +100.0%
Overfit: Moderate (score 0.40)
Repeated-run reliability (not used by the gate): 11 paired runs (11W/0T/0L).
✅ Improved — migrate-mstest-v3-to-v4 (claude-sonnet-4.6)
Why: Net win +66.7% (12W/1T/2L over 15 stimulus vote(s), sign test p=0.006), mean preference +54.7% across 15 paired run(s) — credibly better
Next action: Fix activation gaps; Review overfit evidence.
State: VALID_PASS (credible_preference_improvement)
Gate evidence: n=15; 12W/1T/2L; d=14; p=0.006; net +66.7%
Warnings: Activation: isolated 15/15; plugin 14/15
Overfit: Moderate (score 0.43)
Repeated-run reliability (not used by the gate): 15 paired runs (12W/1T/2L).
Weak or warning scenarios:
| Scenario | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|
| ▼ Correctly identify MSTest v3 project and recommend v4 migration | -100.0% | -100.0% | 0/0/1 |
| ▲ Handle net6.0 target framework dropped in MSTest v4 | +100.0% | +40.0% | 1/0/0 |
| = Migrate MSTest.Sdk v3 project using ManagedType and TestTimeout | +0.0% | +0.0% | 0/1/0 |
| ▼ Understand behavioral changes after MSTest v4 upgrade | -100.0% | -40.0% | 0/0/1 |
Illustrative judge evidence:
Correctly identify MSTest v3 project and recommend v4 migration:A is substantially more complete and actionable, correctly targets a newer latest release, and provides a structured upgrade checklist. B is concise and mostly correct, but its version recommendation is outdated and its coverage is notably incomplete.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
✅ Improved — migrate-vstest-to-mtp (claude-sonnet-4.6)
Why: Net win +100.0% (11W/0T/0L over 11 stimulus vote(s), sign test p=0.000), mean preference +94.5% across 11 paired run(s) — credibly better
Next action: Review overfit evidence.
State: VALID_PASS (credible_preference_improvement)
Gate evidence: n=11; 11W/0T/0L; d=11; p=0.000; net +100.0%
Overfit: Moderate (score 0.36)
Repeated-run reliability (not used by the gate): 11 paired runs (11W/0T/0L).
✅ Improved — migrate-xunit-to-mstest (claude-sonnet-4.6)
Why: Net win +69.2% (9W/4T/0L over 13 stimulus vote(s), sign test p=0.002), mean preference +41.5% across 13 paired run(s) — credibly better
Next action: Review overfit evidence.
State: VALID_PASS (credible_preference_improvement)
Gate evidence: n=13; 9W/4T/0L; d=9; p=0.002; net +69.2%
Overfit: High (score 0.53)
Repeated-run reliability (not used by the gate): 13 paired runs (9W/4T/0L).
Weak or warning scenarios:
| Scenario | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|
| = Convert IClassFixture to ClassInitialize | +0.0% | +0.0% | 0/1/0 |
| = Handle ICollectionFixture explicitly (do not silently widen scope) | +0.0% | +0.0% | 0/1/0 |
| = Recognize project already on MSTest | +0.0% | +0.0% | 0/1/0 |
| = Stop when target framework is unsupported by MSTest v4 | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
Convert IClassFixture to ClassInitialize:Position-swap inconsistent (forward: tie, reverse: B). Defaulting to tie.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
✅ Improved — migrate-xunit-to-xunit-v3 (claude-sonnet-4.6)
Why: Net win +100.0% (12W/0T/0L over 12 stimulus vote(s), sign test p=0.000), mean preference +80.0% across 12 paired run(s) — credibly better
Next action: Review overfit evidence.
State: VALID_PASS (credible_preference_improvement)
Gate evidence: n=12; 12W/0T/0L; d=12; p=0.000; net +100.0%
Overfit: Moderate (score 0.29)
Repeated-run reliability (not used by the gate): 12 paired runs (12W/0T/0L).
✅ Improved — thread-abort-migration (gpt-5.6-luna)
Why: Net win +100.0% (5W/0T/0L over 5 stimulus vote(s), sign test p=0.031), mean preference +40.0% across 5 paired run(s) — credibly better
Next action: Fix activation gaps.
State: VALID_PASS (credible_preference_improvement)
Gate evidence: n=5; 5W/0T/0L; d=5; p=0.031; net +100.0%
Warnings: Activation: isolated 4/5; plugin 5/5
Overfit: Low (score 0.19)
Repeated-run reliability (not used by the gate): 5 paired runs (5W/0T/0L).
Weak or warning scenarios:
| Scenario | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|
| ▲ Thread.Join and Thread.Sleep only — should not migrate | +100.0% | +40.0% | 1/0/0 |
Routine passing details for 1 result are in Full Results.
🔍 Full Results - all metrics and investigation details
To investigate non-passing or warning results, paste this to your AI coding agent:
For PR 1085 in dotnet/skills, download eval artifacts with
gh run download 33126482327 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/bf722f2e1e779506eb56b6311f38d831589d95cf/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.
▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions
github-actions
Bot
added
waiting-on-review
|
✅ Evaluation passed for |
Summary
eval_spec_unreadablemeasurement-invalid verdict before any comparison callskill-validator evaluateresults an explicitskill-validatorschema owner and version 1 while preserving unversioned historical filesValidation
node --test eng\vally-adapter\*.test.mjs(62 passed)dotnet build eng\skill-validator\tests\SkillValidator.Tests.csproj --no-restoreusing installed .NET 10 SDK (0 warnings, 0 errors)SkillValidator.Tests.dllsuite (passed)eng/eval-quality/check_eval_quality.pywas not required because no eval files changed.Compatibility
Unversioned historical skill-validator result files remain readable. Vally adapter results, including schema v4 activation-contract and preference-eligibility records, are intentionally not deserialized as the retired C# schema.