[Bug][Windows]: synchronous PowerShell app-server enumeration blocks /healthz and proxy traffic on v2 requests

[Hylouis233]

Client or integration

Codex App

Area

Proxy and routing / Platform (Windows)

Summary

On Windows, OpenCodex performs synchronous PowerShell/CIM process enumeration on the proxy's main Bun event loop while building multi-agent v2 guidance.

The hot request path is:

1. buildMultiAgentGuidance() handles a v2 surface.
2. It awaits defaultCollectCatalogState().
3. That calls the synchronous collectCodexAppServerCatalogState().
4. listWindowsSnapshots() runs execFileSync(powershell.exe, ..., timeout: 8_000).

Relevant source:

• v2 guidance request path
• per-v2-request catalog collection
• synchronous Windows enumeration

On the affected machine, one enumeration blocks the Bun event loop for 3.8-5.1 seconds. During that interval, even loopback /healthz cannot be serviced. Under repeated v2 requests, these pauses can queue and make the proxy appear permanently hung: the Bun process remains alive and owns port 10100, but health probes time out and connections accumulate in CLOSE_WAIT.

I expected catalog-state discovery to be asynchronous, deduplicated/cached, or outside the request hot path so a slow CIM query cannot block unrelated proxy traffic and health checks.

This remains present in the latest stable package, 2.22.0, and in current main.

Reproduction

With the repository dependencies available on Windows, run the following with Bun:

const { listWindowsSnapshots } =
  await import("./src/codex/app-server-processes.ts");

for (let i = 1; i <= 5; i++) {
  const scheduled = performance.now();
  const timer = new Promise<number>(resolve =>
    setTimeout(() => resolve(performance.now() - scheduled), 10)
  );

  const started = performance.now();
  let count: number | null = null;
  let error: string | null = null;
  try {
    count = listWindowsSnapshots().length;
  } catch (e) {
    error = e instanceof Error ? e.message : String(e);
  }

  const callMs = performance.now() - started;
  const timerMs = await timer;
  console.log({ run: i, callMs, timerMs, count, error });
}

A 10 ms timer is delayed by essentially the full synchronous call duration:

Run	listWindowsSnapshots	10 ms timer fired after	Result
1	3812 ms	3813 ms	3 processes
2	3937 ms	3943 ms	3 processes
3	4080 ms	4082 ms	3 processes
4	5072 ms	5075 ms	windows_enum_incomplete
5	4402 ms	4406 ms	3 processes

The timer evidence makes the event-loop blocking deterministic without requiring an upstream provider request.

For the live proxy symptom:

1. Leave multiAgentGuidanceEnabled at its effective default (true).
2. Use Codex App with the multi-agent v2 surface and issue repeated turns/tool continuations.
3. Probe http://127.0.0.1:10100/healthz concurrently.
4. Observe a PowerShell child under the OpenCodex Bun process running the Get-CimInstance Win32_Process / GetOwner script.
5. Health requests intermittently take several seconds or time out despite the listener and Bun PID remaining alive.
6. Setting multiAgentGuidanceEnabled: false removes this request-path enumeration. In the observed session, two post-change probe batches passed 45/45 and no enumeration child remained.

A live stalled sample before the workaround showed:

• /healthz: 5-second timeout, zero response bytes
• Bun CPU increase over 2 seconds: 0.016 s (waiting, not spinning)
• Bun private memory: about 1.69 GiB
• listener still active on port 10100
• multiple CLOSE_WAIT connections
• a PowerShell child running the catalog process enumeration

These are observations from one affected installation; the deterministic timer reproduction above is the primary evidence.

Root cause

The async wrapper does not make the synchronous collector non-blocking. execFileSync prevents Bun from advancing timers, accepting/servicing health requests, or processing other proxy work until PowerShell exits or reaches the 8-second timeout.

The enumeration is especially expensive because it walks Win32_Process and calls GetOwner for candidates. Issue #1354 already measured multi-second Windows enumeration and fixed a correctness race in the same collector, but did not remove it from the event loop or request hot path.

This is analogous to #612 / #645, where synchronous icacls work blocked /healthz until that high-frequency path was made asynchronous.

Suggested safe direction

1. Use asynchronous process creation for Windows enumeration, or move it to a worker so the proxy event loop remains responsive.
2. Deduplicate concurrent discovery with a single in-flight operation.
3. Cache the last result with a short TTL / stale-while-revalidate policy. Catalog-state guidance is advisory; serving a recent/unknown state is safer than blocking all proxy traffic.
4. Add a regression test that schedules a short timer or health request while the process-enumeration seam is deliberately slow and verifies it is serviced before enumeration completes.
5. Optionally narrow the CIM query before owner lookups, but query optimization alone would not remove the event-loop blocking contract.

Workaround

{
  "multiAgentGuidanceEnabled": false
}

This disables OpenCodex's additional multi-agent guidance/roster injection, not Codex's built-in collaboration tools.

Version

OpenCodex 2.22.0, bundled Bun 1.3.14, Codex CLI 0.147.0.

Operating system

Windows 11 Pro, version 10.0.26200 (build 26200).

Provider and model

Not provider-specific. The blocking occurs before provider work while building v2 guidance.

Logs or error output

GET /healthz -> timeout after 5.0s, 0 bytes
PowerShell child -> Get-CimInstance Win32_Process ... Invoke-CimMethod GetOwner
listWindowsSnapshots() -> 3.8-5.1s synchronous wait

Screenshots and supporting files

Not required; the source links, deterministic timer reproduction, timings, and live probe observations are text-reproducible.

Redacted configuration

{
  "multiAgentGuidanceEnabled": true
}

Related issues

• [Bug]: transient code-mode-host process makes catalog staleness check return "unknown", emitting a false "catalog changed after Codex started" restart prompt (2.11.1, Windows) #1354: correctness/race problem in the same Windows catalog-state collector
• [Bug]: synchronous Windows ACL hardening stalls the proxy on response-state writes #612 / fix(windows): async ACL harden for response-state writes (#612) #645: precedent for moving a synchronous Windows child-process wait off the proxy event loop

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.

[coderabbitai]

🔗 Related PRs

#306 - feat(windows): add tray controls and restart-safety diagnostics [merged]
#359 - fix(codex): stable Codex runtime selection for sync/clamp (#297) [merged]
#576 - fix(codex): warn about stale Codex app-servers after a catalog write [merged]
#770 - fix: research-batch reliability and security-boundary issues [closed]
#801 - test(windows): retry one transient CIM enumeration timeout [merged]
#868 - fix(windows): settle scheduler registration verification [open]
#1154 - fix(probe): walk Windows service definition chains (Task Scheduler + WinSW) for ownership [merged]
#1279 - fix(windows): eliminate console windows from proxy-internal identity & process lookups [open]
#1347 - fix(windows): stop console popups from proxy-internal PowerShell lookups (#1278) [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[Ingwannu]

Confirmed on current dev: the v2 guidance path still awaits collectCodexAppServerCatalogState(), which reaches synchronous Windows PowerShell/CIM enumeration. The async caller does not prevent that child-process wait from blocking Bun's event loop.

I am fixing this with a focused boundary:

• keep the existing synchronous collector for explicit CLI/service operations that rely on it;
• add a non-blocking Windows catalog-state collector for the request path;
• deduplicate concurrent refreshes and serve a short-lived last-known state so repeated v2 turns do not launch duplicate CIM walks;
• fail advisory guidance closed/silent on collection failure rather than blocking unrelated proxy and /healthz traffic;
• add a regression where a deliberately slow enumeration is in flight while the event loop remains responsive.

I will keep the change scoped to catalog-state guidance and add the reason/tradeoff to the relevant architecture documentation.

[Ingwannu]

Implemented in #1876.

The PR keeps the existing synchronous lifecycle collector, moves only the Windows v2 request path to async trusted-System32 PowerShell, deduplicates concurrent CIM probes, caches the advisory result briefly, and generation-guards invalidation so an older in-flight probe cannot restore stale state after a catalog write. Focused tests, typecheck, privacy scan, docs build, and diff check are green; the full-suite baseline exceptions are documented in the PR.

Because this is my own PR, I will not self-approve or self-merge it. It now needs exact-head cross-platform CI and another maintainer review.