Use shallow fetches for generated dashboard branches - #1099
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
Review tier: Lite
Findings: None
What changed in this PR
This PR optimizes the dashboard publishing workflow by ensuring generated “snapshot” branches are fetched and cloned shallowly, reducing unnecessary history download and local object retention on GitHub Actions runners.
Changes:
- Added
--depth=1to four fetches ofdashboard-token-data/dashboard-eval-databranch tips. - Added
--depth=1to three clones used for publishing updates todashboard-token-data,dashboard-eval-data, andgh-pages. - Normalized the existing
dashboard-session-dataclone to the same--depth=1option style.
| File | Description |
|---|---|
| .github/workflows/evaluation.yml | Adds shallow fetch/clone options for generated dashboard branches to reduce runner-side Git history transfer and storage. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
github-actions
Bot
added
pr-state/ready-for-eval
📊 Skill Evaluation Results22 model/skill results across 11 skills and 2 models — ✅ 5 improved, ➖ 5 not proven improved, Measurement identity: evaluated commit Measurement health: 22 expected / 22 observed / 22 written; 0 missing, 0 unexpected, 12 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots. Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven. A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of
ℹ️ How to read this report
|
| Scenario | Preference gate | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|---|
| = Parse .NET frames and locate dSYMs from an iOS crash log | Eligible | +0.0% | +0.0% | 0/1/0 |
| ▲ Reject Android tombstone passed as iOS crash log | Eligible | +100.0% | +100.0% | 1/0/0 |
Illustrative judge evidence:
Parse .NET frames and locate dSYMs from an iOS crash log:Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
⚠️ Underpowered — apple-crash-symbolication (gpt-5.6-luna)
Why: Net win +100.0% (3W/0T/0L over 3 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +40.0% across 3 paired run(s) — underpowered (3 preference-eligible stimulus vote(s); a credible verdict needs at least 5, and this eval won every one of them) — add distinct, discriminating stimuli; repeated runs do not increase task breadth
Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.
State: INVALID_INCONCLUSIVE (underpowered)
Gate evidence: n=3; 3W/0T/0L; d=3; p=0.125; net +100.0%
Warnings: Activation: isolated 2/3; plugin 3/3
Overfit: Low (score 0.16)
Repeated-run reliability (not used by the gate): 3 paired runs (3W/0T/0L).
Weak or warning scenarios:
| Scenario | Preference gate | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|---|
| ▲ Reject Android tombstone passed as iOS crash log | Eligible | +100.0% | +40.0% | 1/0/0 |
⚠️ Underpowered — configuring-opentelemetry-dotnet (claude-sonnet-4.6)
Why: Net win +66.7% (2W/1T/0L over 3 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +26.7% across 3 paired run(s) — underpowered (3 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth
Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.
State: INVALID_INCONCLUSIVE (underpowered)
Gate evidence: n=3; 2W/1T/0L; d=2; p=0.250; net +66.7%
Overfit: High (score 0.52)
Repeated-run reliability (not used by the gate): 3 paired runs (2W/1T/0L).
Weak or warning scenarios:
| Scenario | Preference gate | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|---|
| = Set up OpenTelemetry tracing and metrics with custom spans in ASP.NET Core | Eligible | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
Set up OpenTelemetry tracing and metrics with custom spans in ASP.NET Core:Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
⚠️ Underpowered — configuring-opentelemetry-dotnet (gpt-5.6-luna)
Why: Net win +66.7% (2W/1T/0L over 3 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +26.7% across 3 paired run(s) — underpowered (3 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth
Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.
State: INVALID_INCONCLUSIVE (underpowered)
Gate evidence: n=3; 2W/1T/0L; d=2; p=0.250; net +66.7%
Overfit: Low (score 0.15)
Repeated-run reliability (not used by the gate): 3 paired runs (2W/1T/0L).
Weak or warning scenarios:
| Scenario | Preference gate | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|---|
| = Propagate trace context across a message queue | Eligible | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
Propagate trace context across a message queue:Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
⚠️ Underpowered — convert-blazor-server-to-webapp (claude-sonnet-4.6)
Why: Net win +100.0% (1W/0T/0L over 1 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +100.0% across 1 paired run(s) — underpowered (1 preference-eligible stimulus vote(s); a credible verdict needs at least 5, and this eval won every one of them) — add distinct, discriminating stimuli; repeated runs do not increase task breadth
Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.
State: INVALID_INCONCLUSIVE (underpowered)
Gate evidence: n=1; 1W/0T/0L; d=1; p=0.500; net +100.0%
Overfit: Moderate (score 0.26)
Repeated-run reliability (not used by the gate): 1 paired run (1W/0T/0L).
⚠️ Underpowered — convert-blazor-server-to-webapp (gpt-5.6-luna)
Why: Net win +100.0% (1W/0T/0L over 1 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +100.0% across 1 paired run(s) — underpowered (1 preference-eligible stimulus vote(s); a credible verdict needs at least 5, and this eval won every one of them) — add distinct, discriminating stimuli; repeated runs do not increase task breadth
Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.
State: INVALID_INCONCLUSIVE (underpowered)
Gate evidence: n=1; 1W/0T/0L; d=1; p=0.500; net +100.0%
Overfit: Low (score 0.10)
Repeated-run reliability (not used by the gate): 1 paired run (1W/0T/0L).
⚠️ Underpowered — dotnet-webapi (claude-sonnet-4.6)
Why: Net win +100.0% (3W/0T/0L over 3 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +80.0% across 3 paired run(s) — underpowered (3 preference-eligible stimulus vote(s); a credible verdict needs at least 5, and this eval won every one of them) — add distinct, discriminating stimuli; repeated runs do not increase task breadth
Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.
State: INVALID_INCONCLUSIVE (underpowered)
Gate evidence: n=3; 3W/0T/0L; d=3; p=0.125; net +100.0%
Overfit: High (score 0.67)
Repeated-run reliability (not used by the gate): 3 paired runs (3W/0T/0L).
⚠️ Underpowered — dotnet-webapi (gpt-5.6-luna)
Why: Net win +66.7% (2W/1T/0L over 3 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +26.7% across 3 paired run(s) — underpowered (3 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth
Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.
State: INVALID_INCONCLUSIVE (underpowered)
Gate evidence: n=3; 2W/1T/0L; d=2; p=0.250; net +66.7%
Overfit: Moderate (score 0.36)
Repeated-run reliability (not used by the gate): 3 paired runs (2W/1T/0L).
Weak or warning scenarios:
| Scenario | Preference gate | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|---|
| = Add a new API endpoint to an existing controller-based project | Eligible | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
Add a new API endpoint to an existing controller-based project:Both agents investigated the workspace, found it empty (or containing only skill docs), and declined to write code, asking for the project files. Neither satisfied any of the implementation rubric criteria. Both reached essentially the same correct-but-unhelpful conclusion. Re...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
⚠️ Underpowered — microbenchmarking (claude-sonnet-4.6)
Why: Net win +100.0% (1W/0T/0L over 1 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +40.0% across 1 paired run(s) — underpowered (1 preference-eligible stimulus vote(s); a credible verdict needs at least 5, and this eval won every one of them) — add distinct, discriminating stimuli; repeated runs do not increase task breadth
Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.
State: INVALID_INCONCLUSIVE (underpowered)
Gate evidence: n=1; 1W/0T/0L; d=1; p=0.500; net +100.0%
Overfit: Moderate (score 0.41)
Repeated-run reliability (not used by the gate): 1 paired run (1W/0T/0L).
⚠️ Underpowered — microbenchmarking (gpt-5.6-luna)
Why: Net win +100.0% (1W/0T/0L over 1 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +100.0% across 1 paired run(s) — underpowered (1 preference-eligible stimulus vote(s); a credible verdict needs at least 5, and this eval won every one of them) — add distinct, discriminating stimuli; repeated runs do not increase task breadth
Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.
State: INVALID_INCONCLUSIVE (underpowered)
Gate evidence: n=1; 1W/0T/0L; d=1; p=0.500; net +100.0%
Overfit: Low (score 0.13)
Repeated-run reliability (not used by the gate): 1 paired run (1W/0T/0L).
⚠️ Underpowered — minimal-api-file-upload (claude-sonnet-4.6)
Why: Net win +100.0% (3W/0T/0L over 3 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +60.0% across 3 paired run(s) — underpowered (3 preference-eligible stimulus vote(s); a credible verdict needs at least 5, and this eval won every one of them) — add distinct, discriminating stimuli; repeated runs do not increase task breadth
Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.
State: INVALID_INCONCLUSIVE (underpowered)
Gate evidence: n=3; 3W/0T/0L; d=3; p=0.125; net +100.0%
Overfit: Moderate (score 0.43)
Repeated-run reliability (not used by the gate): 3 paired runs (3W/0T/0L).
⚠️ Underpowered — minimal-api-file-upload (gpt-5.6-luna)
Why: Net win +33.3% (1W/2T/0L over 3 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +13.3% across 3 paired run(s) — underpowered (3 preference-eligible stimulus vote(s); a credible verdict needs at least 5) — add distinct, discriminating stimuli; repeated runs do not increase task breadth
Next action: Predeclare more independent, discriminating stimuli; repeated runs do not add power.
State: INVALID_INCONCLUSIVE (underpowered)
Gate evidence: n=3; 1W/2T/0L; d=1; p=0.500; net +33.3%
Overfit: Low (score 0.11)
Repeated-run reliability (not used by the gate): 3 paired runs (1W/2T/0L).
Weak or warning scenarios:
| Scenario | Preference gate | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|---|
| = Implement secure file upload in ASP.NET Core 8 minimal API | Eligible | +0.0% | +0.0% | 0/1/0 |
| = Stream very large file uploads without buffering | Eligible | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
Implement secure file upload in ASP.NET Core 8 minimal API:Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
➖ Not proven improved — analyzing-dotnet-performance (claude-sonnet-4.6)
Why: Net win +36.4% (7W/1T/3L over 11 preference-eligible stimulus vote(s), sign test p=0.172), mean preference +20.0% across 11 paired run(s) — not credible (sign test p=0.172 > 0.05)
Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.
State: VALID_NO_CHANGE (no_credible_preference_change)
Gate evidence: n=11; 7W/1T/3L; d=10; p=0.172; net +36.4%
Overfit: Moderate (score 0.32)
Repeated-run reliability (not used by the gate): 11 paired runs (7W/1T/3L).
Weak or warning scenarios:
| Scenario | Preference gate | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|---|
| ▼ Finds branched Replace chain in format string manipulation | Eligible | -100.0% | -100.0% | 0/0/1 |
| ▼ Finds per-call Dictionary allocation not hoisted to static | Eligible | -100.0% | -100.0% | 0/0/1 |
| ▼ Finds repeated enumeration in time and collection formatting | Eligible | -100.0% | -40.0% | 0/0/1 |
| = Flags Span inconsistencies and compound method chains in truncation library | Eligible | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
Finds branched Replace chain in format string manipulation:A delivers a substantive, mostly relevant performance and correctness review with concrete structural and formatting recommendations. B merely requests the code despite it being available to the agent and does not perform the requested review.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
➖ Not proven improved — android-tombstone-symbolication (gpt-5.6-luna)
Why: Net win +0.0% (3W/2T/3L over 8 preference-eligible stimulus vote(s), sign test p=0.656), mean preference +0.0% across 8 paired run(s) — no improvement
Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.
State: VALID_NO_CHANGE (no_credible_preference_change)
Gate evidence: n=8; 3W/2T/3L; d=6; p=0.656; net +0.0%
Warnings: Activation: isolated 7/8; plugin 8/8
Overfit: Low (score 0.19)
Repeated-run reliability (not used by the gate): 8 paired runs (3W/2T/3L).
Weak or warning scenarios:
| Scenario | Preference gate | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|---|
| ▼ Handle .NET frames with no BuildId metadata | Eligible | -100.0% | -40.0% | 0/0/1 |
| ▼ Recognize NativeAOT tombstone with app binary and libSystem.Native.so | Eligible | -100.0% | -40.0% | 0/0/1 |
| = Recognize tombstone with no .NET frames | Eligible | +0.0% | +0.0% | 0/1/0 |
| = Reject iOS crash log as wrong format | Eligible | +0.0% | +0.0% | 0/1/0 |
| ▼ Symbolicate CoreCLR frames in an Android tombstone | Eligible | -100.0% | -100.0% | 0/0/1 |
Illustrative judge evidence:
Handle .NET frames with no BuildId metadata:Both responses reached the same correct conclusion and satisfied all core rubric criteria without hallucinating a BuildId. A is slightly better because it provides more actionable, concrete guidance (specific llvm-symbolizer commands, mention of NuGet runtime pack, .debug file...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
➖ Not proven improved — clr-activation-debugging (claude-sonnet-4.6)
Why: Net win +14.3% (3W/2T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +22.9% across 7 paired run(s) — not credible (sign test p=0.500 > 0.05)
Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.
State: VALID_NO_CHANGE (no_credible_preference_change)
Gate evidence: n=7; 3W/2T/2L; d=5; p=0.500; net +14.3%
Overfit: Moderate (score 0.35)
Repeated-run reliability (not used by the gate): 7 paired runs (3W/2T/2L).
Weak or warning scenarios:
| Scenario | Preference gate | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|---|
| = Analyze healthy managed EXE activation | Eligible | +0.0% | +0.0% | 0/1/0 |
| ▼ Diagnose FOD suppressed but activation still failing | Eligible | -100.0% | -100.0% | 0/0/1 |
| = Diagnose unexpected FOD dialog from native build tool | Eligible | +0.0% | +0.0% | 0/1/0 |
| ▼ Identify multiple activation sequences in a single log | Eligible | -100.0% | -40.0% | 0/0/1 |
Illustrative judge evidence:
Analyze healthy managed EXE activation:The final outputs are identical requests to provide the log rather than an analysis. They therefore have the same, inadequate performance against every substantive criterion.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
➖ Not proven improved — clr-activation-debugging (gpt-5.6-luna)
Why: Net win +71.4% (6W/0T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +37.1% across 7 paired run(s) — not credible (sign test p=0.063 > 0.05)
Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.
State: VALID_NO_CHANGE (no_credible_preference_change)
Gate evidence: n=7; 6W/0T/1L; d=7; p=0.063; net +71.4%
Overfit: Low (score 0.16)
Repeated-run reliability (not used by the gate): 7 paired runs (6W/0T/1L).
Weak or warning scenarios:
| Scenario | Preference gate | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|---|
| ▼ Explain why same binary behaves differently under different launch methods | Eligible | -100.0% | -100.0% | 0/0/1 |
Illustrative judge evidence:
Explain why same binary behaves differently under different launch methods:Response A actually read the logs (via bash after view failed) and confirmed the concrete answer: SEM_FAILCRITICALERRORS = 0x8001 vs 0, quoting the actual FOD suppression log line. It gives a definitive, correct diagnosis. Response B's view tools all failed and it never access...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
➖ Not proven improved — dump-collect (gpt-5.6-luna)
Why: Net win +55.6% (6W/2T/1L over 9 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +35.6% across 9 paired run(s) — not credible (sign test p=0.063 > 0.05)
Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.
State: VALID_NO_CHANGE (no_credible_preference_change)
Gate evidence: n=9; 6W/2T/1L; d=7; p=0.063; net +55.6%
Warnings: Activation: isolated 8/9; plugin 8/9
Overfit: Low (score 0.15)
Repeated-run reliability (not used by the gate): 9 paired runs (6W/2T/1L).
Weak or warning scenarios:
| Scenario | Preference gate | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|---|
| = Advisory: CoreCLR Alpine Docker non-root configuration | Eligible | +0.0% | +0.0% | 0/1/0 |
| ▼ Configure automatic crash dumps for CoreCLR app on Linux | Eligible | -100.0% | -40.0% | 0/0/1 |
| ▲ Decline dump analysis request | Eligible | +100.0% | +40.0% | 1/0/0 |
| = Detect runtime and configure crash dumps for unknown .NET app on Linux | Eligible | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
Advisory: CoreCLR Alpine Docker non-root configuration:Position-swap inconsistent (forward: tie, reverse: B). Defaulting to tie.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
✅ Improved — android-tombstone-symbolication (claude-sonnet-4.6)
Why: Net win +62.5% (5W/3T/0L over 8 preference-eligible stimulus vote(s), sign test p=0.031), mean preference +55.0% across 8 paired run(s) — credibly better
Next action: Review overfit evidence.
State: VALID_PASS (credible_preference_improvement)
Gate evidence: n=8; 5W/3T/0L; d=5; p=0.031; net +62.5%
Overfit: Moderate (score 0.49)
Repeated-run reliability (not used by the gate): 8 paired runs (5W/3T/0L).
Weak or warning scenarios:
| Scenario | Preference gate | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|---|
| = Recognize NativeAOT tombstone with app binary and libSystem.Native.so | Eligible | +0.0% | +0.0% | 0/1/0 |
| = Recognize tombstone with no .NET frames | Eligible | +0.0% | +0.0% | 0/1/0 |
| = Symbolicate tombstone with multiple .NET libraries and different BuildIds | Eligible | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
Recognize NativeAOT tombstone with app binary and libSystem.Native.so:Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
✅ Improved — dotnet-trace-collect (claude-sonnet-4.6)
Why: Net win +76.5% (14W/2T/1L over 17 preference-eligible stimulus vote(s), sign test p=0.000), mean preference +62.4% across 17 paired run(s) — credibly better
Next action: Review overfit evidence.
State: VALID_PASS (credible_preference_improvement)
Gate evidence: n=17; 14W/2T/1L; d=15; p=0.000; net +76.5%
Overfit: High (score 0.56)
Repeated-run reliability (not used by the gate): 17 paired runs (14W/2T/1L).
Weak or warning scenarios:
| Scenario | Preference gate | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|---|
| ▼ Container installation without .NET SDK | Eligible | -100.0% | -40.0% | 0/0/1 |
| = Kubernetes with console access prefers console tools | Eligible | +0.0% | +0.0% | 0/1/0 |
| = Memory leak on Linux (.NET 8) | Eligible | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
Container installation without .NET SDK:Both miss the expected aka.ms direct-download solution and contain questionable assumptions about tool packaging. A is marginally more actionable for the stated transfer problem, with concrete Docker/Kubernetes copy and extraction commands; B's primary recommendation to copy o...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
✅ Improved — dotnet-trace-collect (gpt-5.6-luna)
Why: Net win +76.5% (13W/4T/0L over 17 preference-eligible stimulus vote(s), sign test p=0.000), mean preference +48.2% across 17 paired run(s) — credibly better
Next action: Review overfit evidence.
State: VALID_PASS (credible_preference_improvement)
Gate evidence: n=17; 13W/4T/0L; d=13; p=0.000; net +76.5%
Overfit: Moderate (score 0.21)
Repeated-run reliability (not used by the gate): 17 paired runs (13W/4T/0L).
Weak or warning scenarios:
| Scenario | Preference gate | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|---|
| = Assembly loading failure on Linux (.NET 8) | Eligible | +0.0% | +0.0% | 0/1/0 |
| = Excessive GC on Linux (.NET 8) | Eligible | +0.0% | +0.0% | 0/1/0 |
| = Long-running intermittent issue with PerfView triggers | Eligible | +0.0% | +0.0% | 0/1/0 |
| = Slow requests on Windows with PerfView | Eligible | +0.0% | +0.0% | 0/1/0 |
Illustrative judge evidence:
Assembly loading failure on Linux (.NET 8):Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
✅ Improved — dump-collect (claude-sonnet-4.6)
Why: Net win +100.0% (9W/0T/0L over 9 preference-eligible stimulus vote(s), sign test p=0.002), mean preference +80.0% across 9 paired run(s) — credibly better
Next action: Fix activation gaps; Review overfit evidence.
State: VALID_PASS (credible_preference_improvement)
Gate evidence: n=9; 9W/0T/0L; d=9; p=0.002; net +100.0%
Warnings: Activation: isolated 9/9; plugin 8/9
Overfit: High (score 0.54)
Repeated-run reliability (not used by the gate): 9 paired runs (9W/0T/0L).
Weak or warning scenarios:
| Scenario | Preference gate | Net win | Δ Pref | Runs (W/T/L) |
|---|---|---|---|---|
| ▲ Recover crash dump from macOS NativeAOT without createdump | Eligible | +100.0% | +100.0% | 1/0/0 |
Routine passing details for 1 result are in Full Results.
🔍 Full Results - all metrics and investigation details
To investigate non-passing or warning results, paste this to your AI coding agent:
For PR 1099 in dotnet/skills, download eval artifacts with
gh run download 33450540367 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/442331332522155e11c1306afbfd3f74d23b6c14/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.
▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions
github-actions
Bot
added
waiting-on-review
|
✅ Evaluation passed for |
Problem
The dashboard publishing workflow fetches and clones generated branches that contain append-only snapshots. Consumers only need the current files at each branch tip, but several operations currently allow Git to bring in the branches' full reachable ancestry.
What changed
--depth=1to the four generated-dashboard branch fetches.--depth=1to the three generated-dashboard branch clones.dashboard-session-dataclone to the same--depth=1option style.The data readers use only files from the fetched tip. The publishers create normal child commits from the shallow-cloned tip, so pushes remain fast-forward updates.
This ports the changes from #1089 to an in-repository branch because the original contributor cannot push updates to the source branch.
Validation
go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.7 -shellcheck= -pyflakes= -color .github/workflows/evaluation.yml