Pin min-threads and env-var ground truth in migrate-dotnet8-to-dotnet9 rubric - #1046
Conversation
…9 rubric
Strengthen two stimulus rubrics so the judge has explicit ground truth for
the .NET 8->9 runtime-configuration change. In .NET 9 environment variables
take precedence over runtimeconfig.json for GC and ThreadPool settings, so
server GC becomes workstation GC, concurrent GC is disabled, and the minimum
ThreadPool thread count drops from 25 to 4. The rubric now states that a
response claiming any of these values are unchanged (for example that min
threads stay at 25) is factually wrong however confidently stated.
Rubric-only by design: deterministic output-matches/output-not-matches graders
were tried and rejected because a regex cannot distinguish a correct answer
that mentions the old value in a negated or past-tense clause ("no longer 25",
"stayed at 25 in .NET 8") from an incorrect answer, so graders would false-fail
correct text. The rubric guides the judge without hard-failing valid answers.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
Skill Coverage Report
|
|
/evaluate 30b0125 |
📊 Skill Evaluation Results2 model/skill results across 1 skill and 2 models — ✅ 1 improved, ➖ 1 not proven improved, Measurement identity: evaluated commit Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when the aggregate net win across distinct-stimulus votes is at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
➖ Not proven improved — migrate-dotnet8-to-dotnet9 (gpt-5.6-luna)Why: Net win -8.3% (5W/1T/6L over 12 stimulus vote(s), sign test p=0.500), mean preference -13.3% across 12 paired run(s) — no improvement Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=12; 5W/1T/6L; d=11; p=0.500; net -8.3% Overfit: Low (score 0.06) Repeated-run reliability (not used by the gate): 12 paired runs (5W/1T/6L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. Routine passing details for 1 result are in Full Results. 🔍 Full Results - all metrics and investigation details
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
There was a problem hiding this comment.
Pull request overview
This pull request tightens the human-judge rubrics for the migrate-dotnet8-to-dotnet9 eval so the evaluator has explicit ground truth about the .NET 9 change where environment variables take precedence over runtimeconfig.json for GC and ThreadPool settings.
Changes:
- Strengthened a rubric bullet to explicitly mark “runtimeconfig still wins (as in .NET 8)” as incorrect for .NET 9.
- Strengthened a rubric bullet to explicitly mark claims of “no change” (e.g., min threads staying at 25) as factually wrong, while stating the correct concrete impact.
Show a summary per file
| File | Description |
|---|---|
| tests/dotnet-upgrade/migrate-dotnet8-to-dotnet9/eval.yaml | Updates two rubric lines to pin the env-var precedence reversal and its concrete GC/ThreadPool impacts for judge consistency. |
Review details
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Lite
|
✅ Evaluation passed for |
github-actions
Bot
added
waiting-on-author
|
👋 @AbhitejJohn — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
Review tier: Lite
Findings: 2
New issues introduced by this change (3)
| Severity | Finding |
|---|---|
 |
tests/dotnet-upgrade/migrate-dotnet8-to-dotnet9/eval.yaml — The prompt says the .NET 8 app’s runtimeconfig.json sets the ThreadPool minimum thread count to 50,… |
|
tests/dotnet-upgrade/migrate-dotnet8-to-dotnet9/eval.yaml — The prompt states runtimeconfig.json sets Concurrent=true and MinThreads=25, but the referenced… |
 |
tests/dotnet-upgrade/migrate-dotnet8-to-dotnet9/eval.yaml — PR description says the concrete impact is that minimum ThreadPool threads drop from 25 to 4 in… |
Suppressed comments (2)
tests/dotnet-upgrade/migrate-dotnet8-to-dotnet9/eval.yaml:350
- This rubric item claims the minimum ThreadPool thread count remains 50 from runtimeconfig.json, but the fixture used by this stimulus sets
System.Threading.ThreadPool.MinThreadsto 25. The rubric should match the fixture’s configured value so the judge has correct ground truth.
- "Identifies that DOTNET_ThreadPool_MinThreads is not a supported runtime configuration environment variable
and has no effect, so the minimum ThreadPool thread count remains 50 from runtimeconfig.json"
tests/dotnet-upgrade/migrate-dotnet8-to-dotnet9/eval.yaml:462
- This rubric item says the minimum ThreadPool thread count remains 25 from runtimeconfig.json, but the fixture for this stimulus sets
System.Threading.ThreadPool.MinThreadsto 50. To provide correct ground truth to the judge, the rubric should match the fixture’s value.
- "States the correct concrete impact: server GC becomes workstation GC, concurrent GC is disabled, and the
minimum ThreadPool thread count remains 25 from runtimeconfig.json"
📊 Skill Evaluation Results2 model/skill results across 1 skill and 2 models — ✅ 1 improved, ➖ 1 not proven improved, Measurement identity: evaluated commit Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when the aggregate net win across distinct-stimulus votes is at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
➖ Not proven improved — migrate-dotnet8-to-dotnet9 (gpt-5.6-luna)Why: Net win -8.3% (3W/5T/4L over 12 stimulus vote(s), sign test p=0.500), mean preference +6.7% across 12 paired run(s) — no improvement Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior. State: Gate evidence: n=12; 3W/5T/4L; d=7; p=0.500; net -8.3% Overfit: Low (score 0.06) Repeated-run reliability (not used by the gate): 12 paired runs (3W/5T/4L). Weak or warning scenarios:
Illustrative judge evidence:
This is one example, not the aggregate verdict. Open Full Results for every judgment. Routine passing details for 1 result are in Full Results. 🔍 Full Results - all metrics and investigation details
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
github-actions
Bot
added
waiting-on-review
github-actions
Bot
added
ready-to-merge
|
✅ Approved by @Evangelink. cc @dotnet/skills-merge-approvers — ready to merge. |
What
Corrects two container stimuli in
migrate-dotnet8-to-dotnet9/eval.yamlso the judge has explicit and consistent ground truth for .NET 9 runtime configuration.In .NET 9, the supported GC environment variables take precedence over
runtimeconfig.json.DOTNET_ThreadPool_MinThreadsis not a supported counterpart forSystem.Threading.ThreadPool.MinThreads, so it has no effect.The two fixture outcomes are:
DOTNET_gcServer=0switches the first app from server GC to workstation GC; its minimum ThreadPool thread count remains50fromruntimeconfig.json.DOTNET_gcServer=0andDOTNET_gcConcurrent=0switch the second app to workstation GC and disable concurrent GC; its minimum ThreadPool thread count remains25fromruntimeconfig.json.The prompts and rubrics now state those outcomes. The misleading
MinThreads.*4output-matches alternative was removed, and the mitigation now tells users to remove the unsupported variable and configure MinThreads inruntimeconfig.json.Why
The previous rubric treated
DOTNET_ThreadPool_MinThreadsas a supported override and required the judge to reward an incorrect 25-to-4 change. This update keeps the supported GC precedence behavior while making the ThreadPool ground truth match the runtime and both fixtures.Design note
The prompt grader checks the ThreadPool distinction semantically. A numeric regex cannot distinguish a correct statement such as "remains 25" from an incorrect statement such as "drops from 25 to 4."
Scope and validation
python eng/eval-quality/check_eval_quality.pypasses with no errors.dotnet-upgradeplugin validator passes for all six skills.Verification still needed
The manual
/evaluatestatus is pending. A live eval run is needed to measure the rubric's effect on judge consistency./cc @AbhitejJohn