Fix bundled skill reference loading

[AbhitejJohn]

Summary

Fix silent degradation when a skill cannot load its bundled reference files. Bundled paths remain relative to the directory that contains SKILL.md; skills no longer treat one failed lookup as proof that the files are unavailable.

Changes

• Update analyzing-dotnet-performance and clr-activation-debugging to:
  • resolve bundled files from the skill root, not the user workspace;
  • use one bounded listing of the known references/ directory after a direct-read failure;
  • avoid workspace file or text search for the skill installation;
  • disclose reduced reference coverage in the final result.
• Update create-skill authoring guidance and its checklist so new skills use relative bundled-file paths and cannot degrade silently.

Validation

• dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- check --plugin ./plugins/dotnet-diag
  • All checks passed: 7 skills, 1 agent, 1 plugin.
• python eng/eval-quality/check_eval_quality.py
  • 99 eval specifications checked; no errors.
• git diff --check

Fixes #1008

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
⚠️	dotnet-diag	analyzing-dotnet-performance	11/21	52.4%
✅	dotnet-diag	clr-activation-debugging	16/19	84.2%

Uncovered: dotnet-diag/analyzing-dotnet-performance

• [Validation] All critical patterns were checked (from reference files or inline recipes) (line 180)
• [Validation] Topic-specific recipes run only when matching signals detected (line 181)
• [Validation] Each finding includes a concrete code fix (line 182)
• [Validation] Scan execution checklist is complete (all recipes run) (line 183)
• [Validation] Summary table included at end (line 184)
• [Pitfall] Suggesting Span in async methods (line 191)
• [Pitfall] Suggesting ConfigureAwait(false) in app code (line 193)
• [Pitfall] Recommending ValueTask everywhere (line 194)
• [Pitfall] Flagging new HttpClient() in DI services (line 195)
• [Pitfall] Suggesting CollectionsMarshal.AsSpan broadly (line 197)

Uncovered: dotnet-diag/clr-activation-debugging

• [Validation] The entry point for each problematic activation was identified (line 291)
• [Validation] SEM_FAILCRITICALERRORS state was noted for FOD-related issues (line 294)
• [Validation] Multiple activations within a single log were individually traced (line 295)

[Copilot]

Pull request overview

This PR updates several skill authoring and runtime instructions so bundled reference files under references/ are resolved relative to the directory containing SKILL.md, preventing silent fallback when the skill is installed outside the user’s workspace (per issue #1008).

Changes:

• Clarify bundled-path resolution for analyzing-dotnet-performance and clr-activation-debugging to be SKILL.md-directory-relative rather than workspace-relative.
• Replace “give up on first miss” behavior with a bounded fallback: one references/ directory listing, then retry only if the expected file is present.
• Require explicit disclosure of reduced reference coverage when references can’t be loaded, and update create-skill guidance/checklist accordingly.

Show a summary per file

File	Description
plugins/dotnet-diag/skills/clr-activation-debugging/SKILL.md	Makes reference loading robust (skill-root relative) and requires reporting when reference coverage is unavailable.
plugins/dotnet-diag/skills/analyzing-dotnet-performance/SKILL.md	Prevents silent degradation by using skill-root relative references, bounded retry behavior, and explicit coverage disclosure.
.agents/skills/create-skill/SKILL.md	Updates authoring guidance/checklist to mandate relative bundled-file paths and visible degradation reporting.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 3/3 changed files
• Comments generated: 0
• Review effort level: Lite

[github-actions]

📊 Skill Evaluation Results

4 model/skill results across 2 skills and 2 models — ✅ 0 improved, ➖ 4 not proven improved, ⚠️ 0 invalid or underpowered, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 0d9c3dff328abeee84a4cb7ca46fd3785972b9cd; 2 judge models.

Measurement health: 4 expected / 4 observed / 4 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when the aggregate net win across distinct-stimulus votes is at least 20% and an exact one-sided sign-test result of p ≤ 0.05. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
analyzing-dotnet-performance	claude-sonnet-4.6	➖ Not proven improved	n=11; 7W/1T/3L; d=10; p=0.172; net +36.4%	🔴 0.74	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
analyzing-dotnet-performance	gpt-5.6-luna	➖ Not proven improved	n=11; 8W/1T/2L; d=10; p=0.055; net +54.5%	✅ 0.12	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
clr-activation-debugging	claude-sonnet-4.6	➖ Not proven improved	n=7; 4W/3T/0L; d=4; p=0.063; net +57.1%	🟡 0.40	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
clr-activation-debugging	gpt-5.6-luna	➖ Not proven improved	n=7; 6W/0T/1L; d=7; p=0.063; net +71.4%	✅ 0.17	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, and net win. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Not proven improved — analyzing-dotnet-performance (claude-sonnet-4.6)

Why: Net win +36.4% (7W/1T/3L over 11 stimulus vote(s), sign test p=0.172), mean preference +14.5% across 11 paired run(s) — not credible (sign test p=0.172 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=11; 7W/1T/3L; d=10; p=0.172; net +36.4%

Overfit: High (score 0.74)

Repeated-run reliability (not used by the gate): 11 paired runs (7W/1T/3L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
▼ Finds branched Replace chain in format string manipulation	-100.0%	-40.0%	0/0/1
= Finds per-call Dictionary allocation not hoisted to static	+0.0%	+0.0%	0/1/0
▼ Finds repeated enumeration in time and collection formatting	-100.0%	-40.0%	0/0/1
▼ Identifies unsealed leaf classes and locale hierarchy patterns	-100.0%	-100.0%	0/0/1

Illustrative judge evidence:

• Finds branched Replace chain in format string manipulation: Both fail to provide the requested performance review and instead ask for absent code. A does so directly and concisely; B adds irrelevant, unsupported discussion of loaded reference material and repeats its request.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — analyzing-dotnet-performance (gpt-5.6-luna)

Why: Net win +54.5% (8W/1T/2L over 11 stimulus vote(s), sign test p=0.055), mean preference +32.7% across 11 paired run(s) — not credible (sign test p=0.055 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=11; 8W/1T/2L; d=10; p=0.055; net +54.5%

Overfit: Low (score 0.12)

Repeated-run reliability (not used by the gate): 11 paired runs (8W/1T/2L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
= Detects Aggregate+Replace chain and struct missing IEquatable	+0.0%	+0.0%	0/1/0
▼ Flags Span inconsistencies and compound method chains in truncation library	-100.0%	-40.0%	0/0/1
▼ Identifies unsealed leaf classes and locale hierarchy patterns	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Detects Aggregate+Replace chain and struct missing IEquatable: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — clr-activation-debugging (claude-sonnet-4.6)

Why: Net win +57.1% (4W/3T/0L over 7 stimulus vote(s), sign test p=0.063), mean preference +48.6% across 7 paired run(s) — not credible — 3 of 7 stimulus vote(s) tied, leaving only 4 discordant stimulus vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 4W/3T/0L; d=4; p=0.063; net +57.1%

Overfit: Moderate (score 0.40)

Repeated-run reliability (not used by the gate): 7 paired runs (4W/3T/0L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
= Analyze healthy managed EXE activation	+0.0%	+0.0%	0/1/0
= Diagnose unexpected FOD dialog from native build tool	+0.0%	+0.0%	0/1/0
= Identify multiple activation sequences in a single log	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Analyze healthy managed EXE activation: The final outputs are substantively equivalent requests for the missing log and fail to provide any of the requested analysis. B's prior skill loading does not improve its final result.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — clr-activation-debugging (gpt-5.6-luna)

Why: Net win +71.4% (6W/0T/1L over 7 stimulus vote(s), sign test p=0.063), mean preference +45.7% across 7 paired run(s) — not credible (sign test p=0.063 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 6W/0T/1L; d=7; p=0.063; net +71.4%

Overfit: Low (score 0.17)

Repeated-run reliability (not used by the gate): 7 paired runs (6W/0T/1L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
▼ Explain why same binary behaves differently under different launch methods	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Explain why same binary behaves differently under different launch methods: Response A actually read and diffed both logs, pinpointing the concrete SEM_FAILCRITICALERRORS values (32769 vs 0) that are the root cause per the rubric. Response B's log reads all failed and it produced a well-structured but purely speculative answer, listing SEM_FAILCRITICA...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1015 in dotnet/skills, download eval artifacts with gh run download 32822112581 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/0d9c3dff328abeee84a4cb7ca46fd3785972b9cd/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

✅ Evaluation passed for 0d9c3df. cc @webreidi @AbhitejJohn @dotnet/dotnet-diag @dotnet/skills-diag-reviewers @marklio @ChrisAhna — please review.

[AbhitejJohn]

/evaluate 0d9c3df

[github-actions]

📊 Skill Evaluation Results

4 model/skill results across 2 skills and 2 models — ✅ 3 improved, ➖ 1 not proven improved, ⚠️ 0 invalid or underpowered, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 0d9c3dff328abeee84a4cb7ca46fd3785972b9cd; 2 judge models.

Measurement health: 4 expected / 4 observed / 4 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when the aggregate net win across distinct-stimulus votes is at least 20% and an exact one-sided sign-test result of p ≤ 0.05. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
analyzing-dotnet-performance	claude-sonnet-4.6	✅ Improved	n=11; 9W/1T/1L; d=10; p=0.011; net +72.7%	🟡 0.41	—	Review overfit evidence.
analyzing-dotnet-performance	gpt-5.6-luna	✅ Improved	n=11; 7W/3T/1L; d=8; p=0.035; net +54.5%	✅ 0.16	—	None.
clr-activation-debugging	claude-sonnet-4.6	➖ Not proven improved	n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%	🟡 0.43	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.
clr-activation-debugging	gpt-5.6-luna	✅ Improved	n=7; 6W/1T/0L; d=6; p=0.016; net +85.7%	✅ 0.15	—	None.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, and net win. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Not proven improved — clr-activation-debugging (claude-sonnet-4.6)

Why: Net win +57.1% (5W/1T/1L over 7 stimulus vote(s), sign test p=0.109), mean preference +40.0% across 7 paired run(s) — not credible (sign test p=0.109 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%

Overfit: Moderate (score 0.43)

Repeated-run reliability (not used by the gate): 7 paired runs (5W/1T/1L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
▼ Diagnose FOD suppressed but activation still failing	-100.0%	-100.0%	0/0/1
= Diagnose unexpected FOD dialog from native build tool	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Diagnose FOD suppressed but activation still failing: A actually locates and analyzes the activation log, provides a substantially correct diagnosis and actionable fix. It misses some important mechanism-specific details (capping and SEM_FAILCRITICALERRORS), but B merely asks for logs that were available and makes no diagnosis.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

✅ Improved — analyzing-dotnet-performance (claude-sonnet-4.6)

Why: Net win +72.7% (9W/1T/1L over 11 stimulus vote(s), sign test p=0.011), mean preference +29.1% across 11 paired run(s) — credibly better

Next action: Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=11; 9W/1T/1L; d=10; p=0.011; net +72.7%

Overfit: Moderate (score 0.41)

Repeated-run reliability (not used by the gate): 11 paired runs (9W/1T/1L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
▼ Finds branched Replace chain in format string manipulation	-100.0%	-40.0%	0/0/1
= Finds repeated enumeration in time and collection formatting	+0.0%	+0.0%	0/1/0

Illustrative judge evidence:

• Finds branched Replace chain in format string manipulation: No struct code is present in the supplied task, so requesting it is appropriate and neither can satisfy the requested technical review. A does so directly; B adds an unsupported claim that it will load reference files and scan code, then still merely requests the code.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

Routine passing details for 2 results are in Full Results.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1015 in dotnet/skills, download eval artifacts with gh run download 33095435663 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/0d9c3dff328abeee84a4cb7ca46fd3785972b9cd/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[Evangelink]

One bundled-reference loading issue needs to be addressed before approval.

[Evangelink]

The topic-specific references aren't actually listed below—the only literal path here is references/critical-patterns.md. Because the fallback listing is allowed only after a direct read fails, that successful read gives the agent no way to discover async-patterns.md, memory-and-strings.md, regex-patterns.md, collections-and-linq.md, io-and-serialization.md, or structural-patterns.md. Please enumerate those relative paths (or explicitly allow listing references/ before topic selection) so the bundled recipes are reliably loaded.

[github-actions]

👋 @AbhitejJohn — this PR has changes requested,1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)