Skip to content

untracked_files reads ls-files without -z, so a non-ASCII untracked path comes back C-quoted — the rollback safety-net snapshot then fails its own git add and halts the run #783

Description

@yangle94

untracked_files reads ls-files without -z, so a non-ASCII untracked path comes back C-quoted — which halts an unattended run by making the rollback's own safety-net snapshot unable to git add

verify.untracked_files (verify.py:1178) shells out to

rc, out, detail = _git_out(repo, "ls-files", "--others", "--exclude-standard")
return {line.strip() for line in out.splitlines() if line.strip()}

Git's default core.quotepath=true applies C-style quoting to any path containing a non-ASCII or otherwise special byte, so ls-files answers with a quoted, backslash-escaped spelling
— "_bmad-output/…/spec-5-2-\350\203\275…", leading " + \nnn octals + trailing " — rather than the on-disk name. That spelling is returned as a member of the untracked set and every consumer treats it as a repo-relative literal path. It is not one.

This is not an error path. It is the default on any host whose core.quotepath is unset — i.e. most hosts — and it fires on exactly the story keys, spec basenames and artifact filenames this tool generates for non-English projects, because the dispatched spec is itself an untracked artifact file.

Reproduction (git 2.55.0, macOS)

$ git -C repo config --get core.quotepath ; echo "rc=$? (unset ⇒ default true)"
rc=1
$ git -C repo ls-files --others --exclude-standard
"_bmad-output/implementation-artifacts/spec-5-2-\350\203\275\345\212\233\350\276\271\347\225\214\351\227\270\351\227\250-\351\205\215\346\241\243-principal-\350\257\273\350\207\252\345\212\250\347\224\261\346\235\203\351\231\220\346\213\223\346\211\221\344\277\235\350\257\201.md"
src/opsctl/capability.py

Measured through the module itself:

>>> verify.untracked_files(repo)
{'"_bmad-output/implementation-artifacts/spec-5-2-\\350\\203\\275…\\350\\257\\201.md"',
 'src/opsctl/capability.py'}
>>> (repo / that_quoted_member).exists()
False

-z and -c core.quotepath=false each suppress the quoting independently; the shipped reader uses neither:

invocation first record
ls-files --others --exclude-standard "_bmad-output/…\350\203\275…" (quoted)
-c core.quotepath=false ls-files --others --exclude-standard _bmad-output/…能力边界闸门…
ls-files --others --exclude-standard -z _bmad-output/…能力边界闸门…

Symptom 1 (run-halting): the rollback cannot capture its safety-net snapshot, so it refuses to reset and pauses

recovery_flow.preserve_attempt_worktree → verify.snapshot_worktree (verify.py:1781) stages the run-created untracked files into a throwaway index, handing the set members straight back to git as pathspec operands:

new = sorted(untracked_files(repo) - set(baseline_untracked))
if new:
    rc, out = _git_env(repo, "add", "--", *new, env=env)
    if rc != 0:
        raise GitError(f"git add (snapshot untracked) failed in {repo}: {out}")

The quoted spelling is not a valid pathspec:

$ git add -- '"_bmad-output/…\350\203\275…"'
fatal: pathspec '"_bmad-output/…\350\203\275…"' did not match any files
rc=128

Observed journal entries (run paused with the dev attempt's uncommitted work left in place — the #340 refusal behaving exactly as designed):

{"kind": "attempt-worktree-preserve-failed", "story_key": "5-2-能力边界闸门-配档-principal-读自动由权限拓扑保证",
 "error": "git add (snapshot untracked) failed in …: fatal: pathspec '\"_bmad-output/…\\350\\203\\275…\"' did not match any files"}
{"kind": "rollback-manual-required", …}
{"kind": "run-paused", "stage": "escalation", …}

Trigger chain, and none of it is exotic: a dev session times out → the attempt leaves uncommitted tracked edits → the safety-net snapshot runs → the untracked spec (non-ASCII story key) poisons git add → snapshot_worktree raises → _reset_would_destroy is true → the rollback refuses and the run halts for a human. Nothing is lost, but an unattended run cannot continue. Note the untracked leg is the only failure — read-tree/add -u above it succeed, so a single non-ASCII untracked file is enough to disable the whole preservation path for that attempt.

Symptom 2 (silent): _rollback_cleanup_plan resolves targets through the same set, so a non-ASCII untracked file is never actually cleaned up

_rollback_cleanup_plan (verify.py:1881) turns those strings into filesystem paths:

created = untracked_files(repo) - set(baseline_untracked)
for rel in sorted(created):
    path = (repo_root / rel).resolve()
    if path == repo_root or not path.is_relative_to(repo_root):
        continue
    if any(path == root or path.is_relative_to(root) for root in keep_roots):
        continue

A quoted spelling joins to a nonexistent child of the repo root, so it clears both guards — the keep test can never match it, since its first path component is "_bmad-output, not _bmad-output — and lands in the plan as a target that does not exist. Measured:

>>> plan = verify._rollback_cleanup_plan(repo, baseline_untracked=[], keep=(".bmad-loop", "_bmad-output/implementation-artifacts"))
>>> [(t.path.name[:44], t.path.exists()) for t in plan.targets]
[('"_bmad-output/…\\350\\203\\275…"', False), ('capability.py', True)]

safe_rollback's cleanup then runs target.path.unlink(missing_ok=True) over that phantom, which is a no-op. Consequences:

  • A run-created untracked file with a non-ASCII name outside keep (e.g. src/opsctl/能力.py) is never removed by a rollback, contradicting safe_rollback's stated contract of removing exactly the files this run created. The file survives as residue in a "rolled back" worktree.
  • The keep guard that exists to protect _bmad-output/… is matching a phantom rather than the real path. The real artifact file escapes deletion here only accidentally (its quoted twin
    matches no file), not because the guard evaluated it correctly — so the guard is untested for its actual job on non-ASCII projects.

Contrast with the set-arithmetic consumers, which are self-consistent and therefore merely wrong in the harmless direction: attempt_dirty (:925) and has_changes_since (:855) filter with _path_under_any, which also fails to match the quoted spelling against _bmad-output/… — so a non-ASCII artifact reads as dirty rather than excluded (over-reports), and baseline_untracked is captured through this same function (engine.py:2283, sweep.py:995), so the subtraction still cancels. The bug is not in the set logic; it is in every consumer that treats a member as a path.

The fix shape is already in the file

_untracked_paths (verify.py:2659) asks the identical question the correct way:

proc = _run_git(["git", "-C", str(repo), "ls-files", "-z", "--others", "--exclude-standard"], repo)
return frozenset(rel for rel in proc.stdout.split("\0") if rel)

untracked_files is the older of the two and the outlier. Two spellings of one question exist; the NUL-delimited one is right on both counts — -z implies no path quoting, and it removes the #442 line-splitting hazard for a filename containing a newline.

Suggested scope:

  • untracked_files: read ls-files -z --others --exclude-standard through _git_raw / _git_raw_out and split on "\0". Prefer this to -c core.quotepath=false, which fixes the quoting but leaves the newline-in-filename ambiguity and hides the real statement of intent (consumers want the raw name).
  • Consumers of untracked_files inherit the fix for free and were all reading the buggy set: engine.py:2283, engine.py:7179, sweep.py:995, runs.py:4928, workspace.py:234, plus the in-module verify.py:855, :925, :1781, :1881. (verify.py:1316 and :1428 already use -z.)
  • Worth a sweep for the same pattern in siblings: any _git_out(...) whose stdout is parsed as paths rather than as one value, on a command git will quote (ls-files, diff --name-only, status --porcelain without -z). _literal_specs shows the codebase already knows pathspecs need literal handling; this is the symmetric footgun on the output side.
  • Residual, out of scope here: _run_git decodes stdout, so a filename that is not valid UTF-8 still has no correct round-trip (separate concern, cf. run_verify_commands decodes operator command output strictly; a non-UTF-8 byte crashes the run #378 / _run_git's decode fault escapes the GitError taxonomy on -z output #377).

Test angle

All three are cheap and currently red:

  1. A repo with one untracked file named 中文.md — assert untracked_files(repo) == {"中文.md"}, not the quoted spelling.
  2. snapshot_worktree(repo, ref, baseline_untracked=[]) on that same tree — assert it returns the ref instead of raising GitError from the git add leg. This is the run-halting symptom and deserves the direct regression test.
  3. _rollback_cleanup_plan(repo, baseline_untracked=[], keep=()) with an untracked 能力.py outside keep — assert the target actually exists, i.e. rollback would remove it.

Leave core.quotepath unset in the fixture — the default (true) is what real hosts have, and setting it explicitly would test the wrong thing.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Real defect - workaround exists or blast radius is narrowarea:engineOrchestrator engine and run lifecyclebugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions