Skip to content

tests: make the Windows PowerShell tests launch PowerShell less, and faster - #238

Merged
Aaron Li (aaron-jx-li) merged 1 commit into
mainfrom
ci/faster-windows-powershell-tests
Nov 16, 2026
Merged

Aaron Li (aaron-jx-li) merged 1 commit into
mainfrom
ci/faster-windows-powershell-tests

Conversation

@aaron-jx-li

@aaron-jx-li Aaron Li (aaron-jx-li) commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Since #206, each tests (windows-latest, …) job takes about 22 minutes instead of about two, and every PR and push to main waits on it.

Run Windows test jobs
main after #194 (1e16a37) 91 s / 100 s
#206's own PR run (a9ba579) 1091 s / 1088 s
main after #206 (393b9c8) 1369 s / 1344 s

The job log's progress line that holds #206's PowerShell tests went from 17 s to 20 min 57 s. Those tests launch PowerShell about 100 times per job, and each Windows PowerShell 5.1 launch took 12-18 s on the runner. pwsh took about 0.5 s.

Cause

The tests start PowerShell with a minimal environment (WINDOWS_ENV), which leaves out PSModuleAnalysisCachePath. GitHub's Windows image builds PowerShell's module analysis cache ahead of time at that path (C:\PSModuleAnalysisCachePath\ModuleAnalysisCache). Without it, every Windows PowerShell 5.1 launch analyzes each installed module again (the image has large ones, such as Az) before its first cmdlet (Get-Content, Join-Path, ConvertFrom-Json). Running PowerShell itself was never slow: exit 0 takes 0.1 s in any environment.

I found it with a temporary probe on the runner, since dropped from this branch. It timed the Copilot hook under Windows PowerShell 5.1, then took the variables out of the runner's environment in halves while the hook stayed fast:

Hook launch, Windows PowerShell 5.1 Time
tests' environment 18-28 s
+ PSModulePath and the profile folders (USERPROFILE, LOCALAPPDATA, APPDATA, …) 18 s
runner's full environment 0.4-0.6 s
+ PSModuleAnalysisCachePath only (found by the bisect over 141 variables) 0.4 s

This only affects the tests. The Copilot CLI runs the hook with the person's whole environment.

Changes

  1. The cache path. The PowerShell tests pass PSModuleAnalysisCachePath through where it is set (POWERSHELL_ENV, _powershell_env). Nothing in the plugin reads it.
  2. Fewer launches. The slash commands, the piped payload and the no-Python answer don't depend on the hook event, so they now run once per shell and policy (test_powershell_slash_commands_without_git_bash). The hook keeps its shell × event × policy test (test_powershell_hook_without_git_bash), which now also checks that either event leaves bin/te.json. That is about 66 launches per job instead of about 102, and the same checks.
  3. One Python runs them. A powershell marker. The Windows job on Python 3.10 sets PYTEST_ADDOPTS: -m "not powershell", since these tests check the PowerShell text and the 3.13 job covers it.

Results (this PR's CI)

Windows 3.13 job Windows 3.10 job
main now about 1350 s (pytest about 21 min) about 1350 s
changes 2 and 3 only pytest 714 s pytest 49 s
all three 142 s (pytest 121 s) 115 s (pytest 58 s)

On 3.13, 1120 passed, 27 skipped and none failed, and all 16 PowerShell items ran and passed. On 3.10, 1104 passed and those 16 were deselected.

🤖 Generated with Claude Code

@aaron-jx-li
Aaron Li (aaron-jx-li) force-pushed the ci/faster-windows-powershell-tests branch 2 times, most recently from 2c2c298 to c1b70e3 Compare November 16, 2026 03:33
…faster

Since #206 each Windows test job takes about 22 minutes instead of about
two: its PowerShell tests launch PowerShell about 100 times per job, and
each Windows PowerShell 5.1 launch took 12-18 s on the runner (the job
log's progress line that holds them went from 17 s to 21 min).

- The tests gave PowerShell no PSModuleAnalysisCachePath. GitHub's Windows
  image builds the module analysis cache ahead of time at that path, so
  without it every Windows PowerShell launch analyzed each installed module
  again before its first cmdlet. Passed through where set, a launch of the
  Copilot hook takes 0.4 s instead of 18 s (bisected over the runner's
  environment in #238). pwsh was fast either way.
- The slash commands, the piped payload and the no-Python answer ran once
  per hook event as well as per shell and policy, though none of them
  depends on the event. The hook keeps its shell x event x policy test
  (which now also checks that either event leaves bin/te.json); the rest
  runs once per shell and policy.
- The PowerShell tests carry a `powershell` marker, and the Windows job on
  Python 3.10 leaves them out (PYTEST_ADDOPTS): they test the PowerShell
  text, which the 3.13 job covers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aaron-jx-li
Aaron Li (aaron-jx-li) force-pushed the ci/faster-windows-powershell-tests branch from c1b70e3 to 11194a0 Compare November 16, 2026 03:39
@aaron-jx-li
Aaron Li (aaron-jx-li) marked this pull request as ready for review November 16, 2026 03:42
@aaron-jx-li
Aaron Li (aaron-jx-li) merged commit a517532 into main Nov 16, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant