Skip to content

fix(windows): decode git output as UTF-8, not the process locale - #293

Open
ppcvote wants to merge 1 commit into
openai:mainfrom
ppcvote:fix/git-utf8-decode
Open

fix(windows): decode git output as UTF-8, not the process locale#293
ppcvote wants to merge 1 commit into
openai:mainfrom
ppcvote:fix/git-utf8-decode

Conversation

@ppcvote

@ppcvote ppcvote commented Aug 6, 2026

Copy link
Copy Markdown

Fixes #192.

Git reports paths as UTF-8 on every platform. subprocess.run(..., text=True) decodes with the process locale, which on Windows is the ANSI codepage, so a non-ASCII path is decoded with the wrong codec.

I hit this on a Traditional Chinese host (cp950) and the symptom did not match the filed report, which turned out to be informative rather than a separate bug.

Two shapes, one cause

codepage behaviour what the user sees
single-byte, e.g. cp1251 (as filed) every byte decodes, nothing raises, string is wrong Path.relative_to fails → "Scan target must stay inside its Git working tree."
multi-byte, cp932 / cp936 / cp950 sequence rejected outright UnicodeDecodeError on subprocess's reader thread, so run returns with returncode 0 and stdout None, and git_output calls .strip() on it → AttributeError: 'NoneType' object has no attribute 'strip'

The same bytes, decoded five ways:

git stdout : b'.../\xe6\xb8\xac\xe8\xa9\xa6\xe5\xb0\x88\xe6\xa1\x88-...'
     utf-8 : 測試專案-繁體中文          ← correct
    cp1251 : 測試專案-...          ← silently wrong, never raises
    cp1252 : UnicodeDecodeError
     cp950 : UnicodeDecodeError
     cp936 : UnicodeDecodeError
     cp932 : UnicodeDecodeError

A second call site, outside the issue's scope

Sweeping the plugin for the same pattern found three text=True calls with no encoding=, and no call anywhere that passes one. Two route through git_command; the third is separate:

generate_rank_input.run_git_changed_paths reads names out of git diff --name-status -z. The repository path does not need to contain non-ASCII at all — one tracked file with a non-ASCII name is enough:

repo path (ascii only): ...\ascii-only-repo
tracked file          : 測試檔案.py
git stdout raw        : b'A\x00\xe6\xb8\xac\xe8\xa9\xa6\xe6\xaa\x94\xe6\xa1\x88.py\x00'

as shipped   : stdout None → AttributeError: 'NoneType' object has no attribute 'split'
encoding=utf-8: ['A', '測試檔案.py']

That widens the population from "checkout is in a non-ASCII folder" to "any repository containing a non-ASCII file name, scanned from a non-UTF-8 Windows host".

File I/O in the plugin is already binary ("rb"/"wb") throughout, so this is the whole exposure I could find.

Verification

Windows 10, Python 3.11.6, ANSI codepage cp950, git 2.52.0.windows.1, exercising the patched modules directly rather than a copy of their logic:

                              before                    after
CJK repo directory            AttributeError 'strip'    ok
Cyrillic repo directory       AttributeError 'strip'    ok
CJK file in ASCII repo        AttributeError 'split'    ok
pure ASCII control            ok                        ok
git_bytes returns bytes       ok                        ok

The Cyrillic row is the reporter's own case, confirmed fixed on this host. The ASCII control and git_bytes rows are there to show the text-mode branch did not disturb the binary path.

Notes on the change

  • encoding="utf-8" is exactly what PYTHONUTF8=1 achieves, which Windows: valid Git worktree scans fail for non-ASCII target paths #192 confirms as a workaround, without depending on the environment being set.
  • No errors= handler. On Windows the filesystem is UTF-16 and git converts to UTF-8, so strict decoding is correct here; adding surrogateescape would only move a failure into JSON serialisation later.
  • The text branch is now written out explicitly instead of text=text, because encoding= implies text mode and the two modes should not be conflated. That also let the FileNotFoundError fallback drop its str | bytes union.

Happy to split the second file into its own PR if you would rather keep this scoped strictly to #192.

Git reports paths as UTF-8 on every platform. `subprocess.run(..., text=True)`
decodes with the process locale, which on Windows is the ANSI codepage, so any
non-ASCII path is decoded with the wrong codec.

Fixes openai#192, and covers a second call site the report does not reach.

The failure has two shapes depending on the codepage, which is why the filed
report and this one look different:

- Single-byte (cp1251, the reporter's): every byte decodes, so nothing raises
  and a silently wrong string comes back. `Path.relative_to` then fails and the
  scan is refused with "Scan target must stay inside its Git working tree."
- Multi-byte (cp932/936/950): the sequence is rejected outright. The
  UnicodeDecodeError is raised on subprocess's reader thread, so `run` returns
  with returncode 0 and `stdout` set to None, and `git_output` calls `.strip()`
  on it. That surfaces as `AttributeError: 'NoneType' object has no attribute
  'strip'` rather than a diagnosable message.

Reproduced on Windows 10, Python 3.11.6, ANSI codepage cp950, git 2.52.0:

  before  CJK repo directory          AttributeError ... 'strip'
  before  Cyrillic repo directory     AttributeError ... 'strip'
  before  CJK file in ASCII repo      AttributeError ... 'split'
  after   all three resolve correctly, ASCII control unchanged

The third case is outside openai#192's scope and is the reason for the second file.
`generate_rank_input.run_git_changed_paths` reads names out of `git diff
--name-status -z`, so the repository path does not need to contain non-ASCII at
all: one tracked file with a non-ASCII name is enough to end the scan on any
non-UTF-8 Windows host.

Naming the encoding is what PYTHONUTF8=1 achieves, which the reporter confirmed
as a workaround, without depending on the environment. `git_bytes` is untouched
and still returns undecoded bytes; the text branch is now explicit so the two
modes cannot be conflated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows: valid Git worktree scans fail for non-ASCII target paths

1 participant