Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions .claude/skills/glass-tests/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
name: glass-tests
description: "Use when working with PTVS Glass tests: setup_glass.py, run_glass.py, Glass.TestAdapter, Newtonsoft.Json assembly mismatches, Azure Pipelines Glass cache, TestScript.xml, glass2.exe, and mixed-mode Python debugger tests."
argument-hint: "[setup|run|ci-failure|debug]"
---

# PTVS Glass Tests

## When to Use

Use this skill when setting up, running, debugging, or investigating CI failures for PTVS Glass tests. Glass tests exercise Visual Studio debugger behavior, including mixed-mode Python/native scenarios, and require Microsoft-internal drops for the Glass runner.

Link to existing human-facing documentation instead of duplicating it. Start with [Python/Tests/GlassTests/readme.txt](../../../Python/Tests/GlassTests/readme.txt), which links the internal Concord Glass docs and presentation.

## Key Files

- [Build/setup_glass.py](../../../Build/setup_glass.py): downloads `drop.exe`, Glass, NuGet tools, and vstest drops; extracts VSIX payloads; copies PTVS debugger bits; discovers installed Python versions; generates `.GlassTestProps` files; verifies the test listing.
- [Build/run_glass.py](../../../Build/run_glass.py): calls `copy_ptvs_output()`, verifies the runtime test root, and invokes `vstest.console.exe` with `/Parallel` and TRX logging.
- [Build/templates/run_tests.yml](../../../Build/templates/run_tests.yml): installs Python versions, restores or populates the `GlassTests` cache, acquires the Azure DevOps token, runs setup, runs tests, and publishes TRX results.
- [Python/Tests/GlassTests/PythonTests/PythonConcord.GlassTestRoot](../../../Python/Tests/GlassTests/PythonTests/PythonConcord.GlassTestRoot): root Glass test configuration.
- `Python/Tests/GlassTests/PythonTests/**/TestScript.xml`: individual debugger test scripts with commands, expected events, breakpoints, and source expectations.
- `GlassTests/`: generated/runtime directory for downloaded Glass bits and copied tests; do not treat it as the source of truth for test definitions.

## Local Workflow

1. Build PTVS first so the chosen build output contains `EnvironmentDiscover.exe`, `Microsoft.Python*`, and `DkmDebu*` files.
2. Prefer passing the build output explicitly, especially for VS 18 builds:

```powershell
python Build\setup_glass.py --buildOutput BuildOutput\Debug18.0\raw\binaries
```

3. If setup needs internal drop access, provide an auth token environment variable or allow the interactive credential flow:

```powershell
python Build\setup_glass.py --authTokenVariable SYSTEM_ACCESSTOKEN --buildOutput <path-to-raw-binaries>
```

4. List discovered tests after setup:

```powershell
python Build\setup_glass.py --verifyListing
```

5. Run all tests or a single test:

```powershell
python Build\run_glass.py --buildOutput BuildOutput\Debug18.0\raw\binaries
python Build\run_glass.py --buildOutput BuildOutput\Debug18.0\raw\binaries Repr_Bytes-311-64
```

## CI Failure Workflow

1. Start with the PR log and `TestResults/PythonTests.trx`.
2. Classify the failure:
- Setup/drop/auth failure before `vstest.console.exe` starts.
- Discovery/adapter failure after `vstest.console.exe` starts but before individual tests run.
- `glass2.exe` or `TestScript.xml` failure inside a specific test case.
3. Check [Build/templates/run_tests.yml](../../../Build/templates/run_tests.yml) to see whether `GlassTests` came from cache. Cache hits skip `setup_glass.py`, but `run_glass.py` still calls `copy_ptvs_output()`.
4. For assembly binding errors, inspect every `Newtonsoft.Json.dll` under `GlassTests` and compare assembly versions. `Glass.TestAdapter.dll` expects `Newtonsoft.Json, Version=13.0.0.0`; NuGet/vstest VSIX extraction can place older 9.x or 10.x copies in the runner tree. The compatible copy usually comes from `GlassTests\Glass\Newtonsoft.Json.dll`.
5. Do not assume a PTVS NuGet restore or package update will fix Glass runner dependency conflicts. The failing assembly may come from downloaded Glass/NuGet/vstest VSIX payloads, not from `packages/`.

## Debugging TestScript Failures

1. Reproduce one test at a time with `python Build\run_glass.py --buildOutput <path> <test-name>`.
2. Inspect the failing test's `TestScript.xml` and source files under `Python/Tests/GlassTests/PythonTests/`.
3. Compare the error log against expected events, source file names, line numbers, breakpoints, and function names in the script.
4. Look for generated logs, dumps, and copied debuggee files under the runtime `GlassTests\PythonTests\...\obj\<configuration>\` and `bin\<configuration>\` paths.
5. To debug Glass itself, set `GLASS_DEBUG=1`, rerun one test, attach Visual Studio to `glass2.exe`, and check assertions or event handling.
6. To debug with the real product, follow [Python/Tests/GlassTests/readme.txt](../../../Python/Tests/GlassTests/readme.txt): run PTVS under the debugger, open the generated test folder, mirror the `TestScript.xml` breakpoints, and use `launch.vs.json` with the intended interpreter.

## Common Pitfalls

- Full setup is Microsoft-internal because `drop.exe` downloads from internal DevDiv artifact drops.
- `run_glass.py` without `--buildOutput` historically probes 17.0 build output paths; pass `--buildOutput` for local VS 18 builds.
- `EnvironmentDiscover.exe not found` usually means the build output path is wrong or stale.
- `PythonConcord.GlassTestRoot` or `PythonEngine.regdef` missing means the runtime test copy is incomplete or setup did not run successfully.
- Cache hits can preserve stale `GlassTests` contents. Fixes that must run before test execution should be in the `run_glass.py` path, usually via `copy_ptvs_output()`.
- Runtime files under `GlassTests/` are generated and cacheable; source test changes belong under `Python/Tests/GlassTests/`.
11 changes: 11 additions & 0 deletions .github/instructions/chat-customizations.instructions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
description: "Use when creating or updating PTVS chat customization files, including SKILL.md files, .claude/skills, .github/skills, prompts, instructions, and custom agents."
applyTo: [".claude/skills/**/SKILL.md", ".github/skills/**/SKILL.md", ".agents/skills/**/SKILL.md"]
---
# PTVS Chat Customization Guidelines

- Put project skills under `.claude/skills/<skill-name>/SKILL.md` unless the user explicitly asks for another host location.
- Do not leave duplicate skill copies under `.github/skills` or `.agents/skills` after moving a skill to `.claude/skills`.
- Keep chat customization changes scoped to customization files; do not modify product, build, test, or package files while handling a customization-only request.
- Link to existing repo documentation instead of copying it into customization files.
- After creating or moving a skill, validate that the folder name matches the `name` frontmatter and that relative links still resolve from the new location.
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
"name": "ptvs",
"private": true,
"devDependencies": {
"@pylance/pylance": "2026.1.1"
"@pylance/pylance": "2026.2.1"
}
}
Loading