Fix bundled skill reference loading

[AbhitejJohn]

Summary

Fix silent degradation when skills cannot load bundled references outside the user workspace. Bundled paths now resolve from the directory containing SKILL.md; after a direct-read failure, affected skills use one bounded listing of the known references/ directory and report reduced coverage if the files remain unavailable.

The authoring guidance now preserves this behavior for new skills without teaching agents to search for their installation directory.

Related issue

N/A

Validation

• dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- check --plugin ./plugins/dotnet-diag - passed for 7 skills, 1 agent, and 1 plugin.
• python eng/eval-quality/check_eval_quality.py - checked 99 eval specifications with no errors.
• git diff --check - passed.

Checklist

• I searched existing issues and pull requests to avoid duplicates.
• I kept this pull request focused and avoided unrelated refactors.
• I added or updated tests, evals, or documentation when changing skill or agent behavior.
• I updated CODEOWNERS when adding or moving owned content.
• I updated all marketplace manifests when plugin metadata changed.
• I updated eng/known-domains.txt for any new external domains referenced by skill content.

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
⚠️	dotnet-diag	analyzing-dotnet-performance	11/21	52.4%
✅	dotnet-diag	clr-activation-debugging	16/19	84.2%
✅	dotnet-test	assertion-quality	22/22	100%
✅	dotnet-test	code-testing-agent	4/4	100%
ℹ️	dotnet-test	code-testing-extensions	-	N/A (reference-only)
✅	dotnet-test	crap-score	6/6	100%
✅	dotnet-test	detect-static-dependencies	22/22	100%
ℹ️	dotnet-test	filter-syntax	-	N/A (reference-only)
✅	dotnet-test	generate-testability-wrappers	24/24	100%
✅	dotnet-test	grade-tests	29/29	100%
✅	dotnet-test	migrate-static-to-wrapper	26/26	100%
✅	dotnet-test	mtp-hot-reload	16/16	100%
✅	dotnet-test	platform-detection	2/2	100%
✅	dotnet-test	run-tests	18/18	100%
✅	dotnet-test	scaffold-dotnet-test-project	18/21	85.7%
ℹ️	dotnet-test	test-analysis-extensions	-	N/A (reference-only)
✅	dotnet-test	test-anti-patterns	25/25	100%
✅	dotnet-test	test-gap-analysis	31/31	100%
✅	dotnet-test	test-smell-detection	26/26	100%
✅	dotnet-test	test-tagging	28/28	100%
⚠️	dotnet-test	testability-obstacle	14/21	66.7%
✅	dotnet-test	writing-mstest-tests	44/46	95.7%

Uncovered: dotnet-diag/analyzing-dotnet-performance

• [Validation] All critical patterns were checked (from reference files or inline recipes) (line 182)
• [Validation] Topic-specific recipes run only when matching signals detected (line 183)
• [Validation] Each finding includes a concrete code fix (line 184)
• [Validation] Scan execution checklist is complete (all recipes run) (line 185)
• [Validation] Summary table included at end (line 186)
• [Pitfall] Suggesting Span in async methods (line 193)
• [Pitfall] Suggesting ConfigureAwait(false) in app code (line 195)
• [Pitfall] Recommending ValueTask everywhere (line 196)
• [Pitfall] Flagging new HttpClient() in DI services (line 197)
• [Pitfall] Suggesting CollectionsMarshal.AsSpan broadly (line 199)

Uncovered: dotnet-diag/clr-activation-debugging

• [Validation] The entry point for each problematic activation was identified (line 293)
• [Validation] SEM_FAILCRITICALERRORS state was noted for FOD-related issues (line 296)
• [Validation] Multiple activations within a single log were individually traced (line 297)

Uncovered: dotnet-test/scaffold-dotnet-test-project

• [Validation] Framework, runner, target framework, and package style match the repository. (line 165)
• [Validation] Template sample tests were removed. (line 168)
• [Pitfall] Picking a favorite framework (line 178)

Uncovered: dotnet-test/testability-obstacle

• [Validation] The original obstacle was concrete and in the requested path. (line 176)
• [Validation] An existing seam was reused when available. (line 177)
• [Validation] The new abstraction exposes only members required by the target behavior. (line 178)
• [Validation] Time conversions preserve local/UTC and DateTime.Kind semantics. (line 180)
• [Pitfall] Wrapping an entire static API (line 190)
• [Pitfall] Converting UtcNow with .DateTime (line 191)
• [Pitfall] Adding DI to a library with no container (line 193)

Uncovered: dotnet-test/writing-mstest-tests

• [CodePattern] [TestClass] (line 147)
• [CodePattern] [TestMethod] (line 147)

[Copilot]

Pull request overview

This pull request updates skill authoring and execution guidance to prevent silent quality degradation when bundled reference files (e.g., references/*.md) are not readable from the user workspace, by standardizing how skills resolve and fall back when bundled files are unavailable.

Changes:

• Clarify that bundled file paths should resolve relative to the directory containing SKILL.md (not the user workspace).
• Add a bounded fallback: if a direct read fails, list the skill’s references/ directory once and retry only when the expected file is present.
• Require skills to explicitly report reduced “Reference coverage” when bundled files remain unavailable.

Show a summary per file

File	Description
plugins/dotnet-diag/skills/clr-activation-debugging/SKILL.md	Updates Step 1 guidance for loading bundled reference material and reporting reduced reference coverage.
plugins/dotnet-diag/skills/analyzing-dotnet-performance/SKILL.md	Aligns Step 1 reference loading guidance with the new bundled-path and bounded-listing fallback behavior.
.agents/skills/create-skill/SKILL.md	Adds repo-wide authoring guidance and checklist items to preserve the bundled-reference behavior in newly created skills.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 3/3 changed files
• Comments generated: 3
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

plugins/dotnet-diag/skills/analyzing-dotnet-performance/SKILL.md:42

• The phrase “Do not use workspace file or text search to locate the skill installation” is ambiguous in this skill because later steps explicitly instruct using grep to scan the target repository. Clarify that the restriction is only about locating missing bundled reference files / the skill installation directory, not about code analysis searches.

Resolve bundled paths from the directory that contains this `SKILL.md`, not from the user's workspace. Load `references/critical-patterns.md` and the topic-specific reference files listed below. These contain detailed detection recipes and grep commands.

If a direct read fails, list this skill's `references/` directory once without recursion and retry only when the listing shows the expected file. Do not use workspace file or text search to locate the skill installation.

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite

[github-actions]

📊 Skill Evaluation Results

4 model/skill results across 2 skills and 2 models — ✅ 2 improved, ➖ 2 not proven improved, ⚠️ 0 invalid or underpowered, 📉 0 preference losses (report only).

Measurement identity: evaluated commit f5a38199829c341151c903749ac5bc0065d5a325; 2 judge models.

Measurement health: 4 expected / 4 observed / 4 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when the aggregate net win across distinct-stimulus votes is at least 20% and an exact one-sided sign-test result of p ≤ 0.05. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
analyzing-dotnet-performance	claude-sonnet-4.6	✅ Improved	n=11; 8W/2T/1L; d=9; p=0.020; net +63.6%	🟡 0.41	Activation: isolated 10/11; plugin 11/11	Fix activation gaps; Review overfit evidence.
analyzing-dotnet-performance	gpt-5.6-luna	✅ Improved	n=11; 9W/1T/1L; d=10; p=0.011; net +72.7%	✅ 0.12	—	None.
clr-activation-debugging	claude-sonnet-4.6	➖ Not proven improved	n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%	🟡 0.35	—	Inspect tied or lost stimuli; predeclare added breadth before a new experiment.
clr-activation-debugging	gpt-5.6-luna	➖ Not proven improved	n=7; 5W/0T/2L; d=7; p=0.227; net +42.9%	✅ 0.17	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, and net win. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Not proven improved — clr-activation-debugging (claude-sonnet-4.6)

Why: Net win +28.6% (3W/3T/1L over 7 stimulus vote(s), sign test p=0.312), mean preference +28.6% across 7 paired run(s) — not credible — 3 of 7 stimulus vote(s) tied, leaving only 4 discordant stimulus vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: Inspect tied or lost stimuli; predeclare added breadth before a new experiment.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%

Overfit: Moderate (score 0.35)

Repeated-run reliability (not used by the gate): 7 paired runs (3W/3T/1L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
= Analyze healthy managed EXE activation	+0.0%	+0.0%	0/1/0
= Diagnose FOD suppressed but activation still failing	+0.0%	+0.0%	0/1/0
= Diagnose unexpected FOD dialog from native build tool	+0.0%	+0.0%	0/1/0
▼ Identify multiple activation sequences in a single log	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Analyze healthy managed EXE activation: Position-swap inconsistent (forward: tie, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Not proven improved — clr-activation-debugging (gpt-5.6-luna)

Why: Net win +42.9% (5W/0T/2L over 7 stimulus vote(s), sign test p=0.227), mean preference +25.7% across 7 paired run(s) — not credible (sign test p=0.227 > 0.05)

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=7; 5W/0T/2L; d=7; p=0.227; net +42.9%

Overfit: Low (score 0.17)

Repeated-run reliability (not used by the gate): 7 paired runs (5W/0T/2L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
▼ Analyze healthy managed EXE activation	-100.0%	-100.0%	0/0/1
▼ Explain why same binary behaves differently under different launch methods	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• Analyze healthy managed EXE activation: Response A located the available log file (recovering from a failed view call via bash) and gave a correct, complete analysis addressing all rubric points. Response B loaded a relevant skill but then simply asked the user to paste the log, failing to discover the file that was...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

✅ Improved — analyzing-dotnet-performance (claude-sonnet-4.6)

Why: Net win +63.6% (8W/2T/1L over 11 stimulus vote(s), sign test p=0.020), mean preference +20.0% across 11 paired run(s) — credibly better

Next action: Fix activation gaps; Review overfit evidence.

State: VALID_PASS (credible_preference_improvement)

Gate evidence: n=11; 8W/2T/1L; d=9; p=0.020; net +63.6%

Warnings: Activation: isolated 10/11; plugin 11/11

Overfit: Moderate (score 0.41)

Repeated-run reliability (not used by the gate): 11 paired runs (8W/2T/1L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
= Detects CurrentCulture comparer and compiled regex budget in inflection rules	+0.0%	+0.0%	0/1/0
= Finds per-call Dictionary allocation not hoisted to static	+0.0%	+0.0%	0/1/0
▼ Identifies unsealed leaf classes and locale hierarchy patterns	-100.0%	-100.0%	0/0/1

Illustrative judge evidence:

• Detects CurrentCulture comparer and compiled regex budget in inflection rules: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

Routine passing details for 1 result are in Full Results.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1060 in dotnet/skills, download eval artifacts with gh run download 32889892699 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/f5a38199829c341151c903749ac5bc0065d5a325/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

[github-actions]

✅ Evaluation passed for f5a3819. cc @webreidi @AbhitejJohn @dotnet/dotnet-diag @dotnet/skills-diag-reviewers @marklio @ChrisAhna — please review.

[AbhitejJohn]

Same as #1015