fix: tolerate non-UTF-8 output from agent CLIs in text-mode subprocess calls - #71
Open
sjwauto123 wants to merge 1 commit into
Open
fix: tolerate non-UTF-8 output from agent CLIs in text-mode subprocess calls#71sjwauto123 wants to merge 1 commit into
sjwauto123 wants to merge 1 commit into
Conversation
…s calls On zh-CN Windows, agent CLIs (and Windows helpers such as taskkill) can emit GBK/ANSI-encoded output while the harness decodes pipes as UTF-8 (system UTF-8 codepage or PYTHONUTF8=1). With capture_output/text=True this spawns reader threads that die with UnicodeDecodeError, so the --version/--help probe returns nothing and the failure is misattributed to the provider. Add errors="replace" to the five text-mode subprocess call sites that read external process output, so invalid bytes degrade to U+FFFD instead of crashing the reader thread. deepseek_runner already passes errors="replace"; these sites were missed. Fixes AMAP-ML#64
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #64 — text-mode
subprocesscalls crash (or silently drop output) when an agent CLI emits GBK/ANSI bytes on zh-CN Windows.Root cause
probe_agent_cli(and four other call sites) read external process output withtext=Truebut noerrors=handler. On zh-CN Windows, agent CLIs and helpers such astaskkillmay emit GBK-encoded bytes while the harness decodes pipes as UTF-8 (system UTF-8 codepage enabled, orPYTHONUTF8=1). With two captured pipes this spawns two reader threads; both die withUnicodeDecodeError('utf-8' codec can't decode byte 0xb4 in position 0: invalid start byte), the--version/--helpprobe returns nothing, and the run is misreported asprovider_network/provider_model_unavailable.Fix
Add
errors="replace"to the five text-mode subprocess call sites that read external process output, so invalid bytes degrade to U+FFFD instead of crashing the reader thread:src/lh_harness/utils/agent_cli.py—probe_agent_cli(--version)src/lh_harness/agent_registry.py—_cli_help_text(--help)src/lh_harness/model_catalog.py—_codex_app_server_models(app-serverPopen)src/lh_harness/plugins/npm.py—_tool_versionand_run(2 sites)adapters/deepseek_runner.pyalready passeserrors="replace"; these five sites were missed.Reproduction
A fake agent emitting the GBK bytes
b4 f2 d3 a1 1.2.3\n(GBK for "版本", invalid as UTF-8):UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 0: invalid start byte;probe_agent_clireturnsusable=False("no recognisable version").probe_agent_clireturnsusable=True,version="1.2.3".Test
Adds
tests/test_agent_cli.pywith a regression test that probes a shell stub emitting a GBK banner and asserts the version is still parsed.