Skip to content

[Bug]: PowerShell helpers write -Json output in the console code page, so non-ASCII names and template text arrive as ? #5416

Backlog

Description

@adityajai119

Bug Description

The PowerShell helpers print their -Json output with ConvertTo-Json, which leaves non-ASCII characters as they are. When an agent captures that stdout, Windows PowerShell 5.1 and PowerShell 7 encode it with the console output code page (437 or 1252 on most Western setups, 936 on Chinese ones), not UTF-8. Characters missing from the code page become ?, and the rest arrive as bytes that are not valid UTF-8.

This is the PowerShell side of #5048, which fixed the same problem in the Python helpers by writing their --json output with ensure_ascii=True.

What an agent gets today (stdout captured with a new hidden console, which keeps the default code page, here 437):

  • Since [bug-fix] Fix non-latin-feature-names: preserve Unicode feature names #4780, feature names keep non-Latin letters. create-new-feature.ps1 -Json "添加用户导出" creates specs\001-添加用户导出, but prints "BRANCH_NAME":"001-??????" and a SPEC_FILE under 001-??????. setup-plan.ps1 -Json and check-prerequisites.ps1 -Json then print the same ? paths, so the agent looks for files that do not exist.
  • resolve-template.ps1 constitution-template -Json, which /speckit.constitution runs, cannot be parsed at all with code page 437: the stock constitution template contains →, which code page 437 writes as byte 0x1A, a raw control character inside a JSON string. This happens even when every name and path is ASCII.
  • setup-tasks.ps1 -Json has the same problem with the stock tasks template (emoji), and every helper has it with a non-ASCII project path.

Steps to Reproduce

Windows 11, from Python, capturing stdout the way an agent does:

uvx --from git+https://github.com/github/spec-kit.git@v1.4.0 specify init demo --integration codex --script ps --ignore-agent-tools --non-interactive
import subprocess
r = subprocess.run(
    ["powershell.exe", "-NoProfile", "-File", r".specify\scripts\powershell\create-new-feature.ps1", "-Json", "添加用户导出"],
    cwd="demo", capture_output=True, creationflags=subprocess.CREATE_NO_WINDOW,
)
print(r.stdout)

Expected Behavior

JSON whose values are the real names, paths and template text, as the Python helpers print since #5048.

Actual Behavior

{"BRANCH_NAME":"001-??????","SPEC_FILE":"C:\\...\\demo\\specs\\001-??????\\spec.md","FEATURE_NUM":"001"}

For resolve-template.ps1 constitution-template -Json, json.loads fails with Invalid control character at: line 1 column 690.

Specify CLI Version

v1.4.0. main (2ee2b1e) has the same scripts.

AI Agent

Codex integration for the reproduction; the scripts are agent-independent.

Operating System

Windows 11, Windows PowerShell 5.1.26100 and PowerShell 7.7.0, console code page 437

Python Version

3.13.5

Error Logs

See Actual Behavior.

Additional Context

I have a fix with regression tests and will open a PR for it right away.

AI Disclosure

Found, reproduced and written up by OpenCode with Claude Opus 5.5 in autonomous agent mode, on behalf of @adityajai119. The example above was run locally.

Activity

  1. github-actions commented on Nov 16, 2026

    @github-actions
    Contributor

    Bug assessment — powershell-json-codepage: Valid · severity high


    Bug Assessment: PowerShell JSON output is encoded with console code page

    Report (summarized)

    Issue author @adityajai119 reports that several PowerShell helpers emit -Json output via ConvertTo-Json directly to stdout, which is then encoded by the host code page (commonly cp437/cp1252/cp936 on Windows) when captured by agents.

    Observed failures in the report:

    • Non-ASCII branch/path text is replaced with ? (example: 001-添加用户导出 becomes 001-??????).
    • Some characters in template content can produce bytes that make the captured JSON invalid UTF-8 and unparseable (json.loads failure with an invalid control character).

    Issue context also references an open fix PR: #5417.

    No additional issue comments were present at assessment time.

    Symptom

    When PowerShell helper scripts run in -Json mode under Windows with stdout captured by an agent/process, JSON payloads that contain non-ASCII text (feature names, paths, template content) are emitted in the active console code page instead of transport-safe UTF-8/ASCII-escaped JSON. Consumers parsing UTF-8 JSON receive corrupted data or parse errors.

    Expected behavior is stable, parseable JSON preserving semantic values (paths, names, template text), matching the Python helpers’ guarded JSON behavior.

    Reproduction

    1. Initialize a Spec Kit project and run PowerShell helper scripts in -Json mode with stdout captured by a subprocess (hidden/no-window capture as reported).
    2. Use inputs/content containing non-ASCII text (e.g., feature name 添加用户导出, constitution/tasks template content with Unicode symbols).
    3. Parse captured stdout as JSON in the caller.

    [NEEDS CLARIFICATION: whether every affected host path reproduces identically across all Windows locales/code pages, though code-path evidence indicates broad exposure.]

    Suspected Code Paths

    • scripts/powershell/create-new-feature.ps1:348 — emits JSON via ConvertTo-Json -Compress directly to stdout.
    • scripts/powershell/check-prerequisites.ps1:174 (and :84) — JSON emission paths for regular/paths output use direct ConvertTo-Json -Compress.
    • scripts/powershell/setup-plan.ps1:84 — JSON result emitted directly with ConvertTo-Json -Compress.
    • scripts/powershell/resolve-template.ps1:35 — emits template content JSON directly via ConvertTo-Json -Compress.
    • scripts/powershell/setup-tasks.ps1:79 — emits JSON payload containing template content/path values directly.
    • scripts/powershell/common.ps1:532 (Resolve-TemplateContent) — returns UTF-8 template text that later flows into JSON payload fields; safe read path, but output path above is unsafe.
    • templates/constitution-template.md:14 — contains → in stock template content that can trigger encoding-path failures when serialized into JSON text output.
    • templates/commands/tasks.md:172 — contains Unicode symbols (e.g., ✅) in stock template content, same risk when emitted through affected JSON path.

    Root Cause Hypothesis

    The PowerShell scripts serialize objects with ConvertTo-Json and write the resulting Unicode string to stdout without forcing a transport-safe representation for redirected/captured output. On Windows PowerShell 5.1 and PowerShell 7, captured stdout commonly uses a legacy code page unless explicitly controlled; characters outside that code page are replaced or transformed, yielding lossy bytes or invalid JSON text for UTF-8 parsers. Confidence: high.

    Proposed Remediation

    Preferred: Add a shared JSON output helper in scripts/powershell/common.ps1 that serializes with ConvertTo-Json -Compress, then escapes non-ASCII code points to \uXXXX (including correct surrogate-pair escapes for non-BMP characters), and writes the final ASCII-only JSON line to stdout. Replace direct JSON emission sites in affected scripts (create-new-feature.ps1, check-prerequisites.ps1, setup-plan.ps1, resolve-template.ps1, setup-tasks.ps1) to use this helper.

    This mirrors the robust transport approach already used in Python (ensure_ascii=True) for command JSON output, avoiding dependence on host code page behavior while preserving exact decoded values for consumers.

    Alternatives (optional):

    • Force PowerShell stdout encoding to UTF-8 at process start ([Console]::OutputEncoding = ...). Trade-off: less deterministic across hosts/invocation paths and riskier side effects for mixed text/JSON output flows.
    • Emit JSON bytes through explicit UTF-8 stream writes only in -Json branches. Trade-off: more invasive per-script plumbing and duplication versus one shared helper.

    Files likely to change:

    • scripts/powershell/common.ps1
    • scripts/powershell/create-new-feature.ps1
    • scripts/powershell/check-prerequisites.ps1
    • scripts/powershell/setup-plan.ps1
    • scripts/powershell/resolve-template.ps1
    • scripts/powershell/setup-tasks.ps1
    • tests/test_create_new_feature_python_parity.py
    • tests/test_check_prerequisites_python_parity.py
    • tests/test_setup_plan_python_parity.py
    • tests/test_setup_tasks_python_parity.py
    • tests/test_resolve_template_python_parity.py

    Tests to add or update:

    • Add/extend Windows PowerShell parity tests that capture -Json stdout in a legacy code page context and assert JSON parsing + value fidelity for:
      • non-ASCII feature/branch/path fields,
      • template content with Unicode symbols/arrows,
      • at least one non-BMP character case.
    • Add regression assertions that PowerShell JSON output remains valid UTF-8-consumable JSON in subprocess capture mode (agent-like invocation).

    Risks & Considerations

    • Escaping implementation must handle surrogate pairs correctly; naive character escaping can corrupt non-BMP code points.
    • Centralizing JSON output changes multiple scripts at once; test breadth should include all helpers using -Json.
    • Behavioral parity with bash/python output schema must remain unchanged (only transport encoding should change).

    Open Questions

    • [NEEDS CLARIFICATION: Should the same ASCII-safe JSON helper be applied immediately to extension PowerShell scripts that also emit JSON (e.g., extensions/.../resolve-tasks.ps1, create-new-feature-branch.ps1) for consistency and future-proofing?]
    • [NEEDS CLARIFICATION: Is there any downstream consumer that depends on raw unescaped non-ASCII JSON bytes from PowerShell stdout (unlikely, but worth confirming before rollout)?]

    Generated by 🐛 Assess Bug from Labeled Issue for #5416 · copilot · gpt53codex · 27.8 AIC · ⌖ 0.614 AIC · ⊞ 24.2K · ◷

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions