[antonwolfy]

Summary

Eight sites hash text that can hold a lone surrogate (a surrogate-escaped argv value or file name, or a \udXXX escape in loaded JSON/YAML) with .encode() or .encode("utf-8"), which raises UnicodeEncodeError: ... surrogates not allowed. On main:

Issue	Call	Result on main
#713	cache_path("https://example.com/n\udcff", cache_dir) (what RepoSource.parse("https://example.com/n\udcff@" + "a" * 40) gives)	UnicodeEncodeError
#714	_safe_id("n\udcff?")	UnicodeEncodeError
#715	reach.token("s", "n\udcff", "p") and skill_reach.token("s", "n\udcff", "p")	UnicodeEncodeError
#716	URLClient("n\udcff")._cache_key(), and the topics cache key of EvaluationPipeline._generate_questions for doc_source="url:https://example.com/n\udcff"	UnicodeEncodeError
#717	_paired_plugin_deltas with a matrix cell named n\udcff (or n\ud800)	UnicodeEncodeError
#718	task_id(c) with c.symbols == ("n\udcff",)	UnicodeEncodeError
#719	Bundle(..., files=[BundleFile(path="n\udcff", ...)]).measured_digest()	UnicodeEncodeError
#720	item_digest, assignment_digest, graph_digest, expectations_digest, response_cache_key and _content_key with json.loads('"n\\udcff"') (or "n\\ud800") in the data	UnicodeEncodeError

Now each returns a digest, and the digest of every input that encoded before is unchanged (the tests pin the literals). #713 to #716, #718 and #719 hash .encode("utf-8", "surrogateescape"), which keeps the value of every UTF-8 input, keeps two different surrogate-escaped values apart, and does not depend on the locale (os.fsencode would raise for a non-ASCII value when the file-system encoding is ASCII, e.g. LC_ALL=C PYTHONUTF8=0). #717 and #720 use .encode("utf-8", "surrogatepass") instead: their text comes from a JSON/YAML escape, where "\ud800" is possible and os.fsencode("\ud800") raises (so #717 does not take the surrogateescape encoding the other sites use); errors="replace" would give two inputs one digest, and refusing the escape in each loader would change what a loaded file may hold at many call sites. #718 keeps surrogateescape: the SCIP reader decodes every symbol as strict UTF-8 and refuses the index otherwise, so a symbol read from an index holds no surrogate at all.

For #720, harbor_export._content_key is reached by a terminal-bench task.yaml whose metadata holds such an escape (difficulty: "n\udcff"), so it is fixed and tested here; the export that follows still stops on writing task.toml, which is tracked in #759.

The same search (every .encode( in dokimi/dokimi and scripts) found sites with the same cause that no issue names, tracked in: #754 (run_reach salt from the tree label), #755 (build_fixture writes SKILL.md for a case name with a surrogate), #756 (Context7 and topic-extractor cache keys; this also stops a local: doc source with a non-UTF-8 directory name after #716), #757 (compute_question_set_hash), #758 (Atom id and retention cache key), #760 (process.run input).

Closes #713
Closes #714
Closes #715
Closes #716
Closes #717
Closes #718
Closes #719
Closes #720

Changes

• One commit per issue, repos/source.py cache_path raises UnicodeEncodeError for a remote source URL with a non-UTF-8 byte #713 to Digests of parsed JSON with ensure_ascii=False raise UnicodeEncodeError for a lone-surrogate escape in the input #720, each with its own tests in tests/test_lone_surrogate_digests.py.
• repos/source.py cache_path, harnesses/runner.py _safe_id, repos/reach.py and repos/skill_reach.py token, mcp/url_client.py _cache_key, orchestrator/pipeline.py topics cache key, repos/graph_tasks.py task_id, novelty/bundle.py measured_digest: .encode("utf-8", "surrogateescape") in place of .encode(), so the digest does not depend on the file-system encoding.
• Tests with an ASCII os.fsencode (and one URLClient case in a child process with LC_ALL=C PYTHONUTF8=0 PYTHONCOERCECLOCALE=0) pin that a non-ASCII URL, path or symbol hashes to its UTF-8 digest at each of these sites.
• eval/matrix_rollup.py pair suffix, eval/leakgate.py item_digest and assignment_digest, eval_coverage.py graph_digest, repos/conformance.py expectations_digest, caching.py response_cache_key, harnesses/harbor_export.py _content_key: .encode("utf-8", "surrogatepass").
• Left as they are because their input cannot hold a surrogate: novelty/bundle.py _digest_text (text read as strict UTF-8), repos/reach.py and repos/skill_reach.py salts from the clock (the tree-label one in reach.py can, see run_reach raises UnicodeEncodeError for a checkout whose directory name has a non-UTF-8 byte #754), and the other hashes the search read.
• Changelog fragment (Fixed).

Risk

None for an input that encoded before: the tests pin the digest, token, key or directory name of a UTF-8 input to the literal main gives (ten of them pass on main's code). A value with a lone surrogate used to stop the command with a traceback; it now gets a digest, so a cache directory or file name is made for it (for #713, n_-4d0dae2da255.git under the cache dir). On Windows a surrogate-escaped name is not a file-name form, and the tests that need one are skipped there.

Testing

On a Linux review host, at 4860cdb (on main 1bb8978):

• make ci-local: 33 passed, 0 failed, 14 skipped. 20374 tests passed, 19 skipped; changed-line coverage 100% of 15 lines; mypy, ruff, the duplicate ratchet and the fragment check pass.
• The new tests on main's code: 16 fail with UnicodeEncodeError (each test that feeds a surrogate) and the 10 that pin a UTF-8 digest pass, so the literals are what main gives. On this branch all 26 pass.
• 55 mutants, one break at a time on a copy: for each site, errors="replace" (two surrogates collide), U+DCFE replaced by U+DCFF (two names share a digest), a high surrogate folded into an escaped byte, the pre-fix .encode() back (the raise), the UTF-8 digest changed, and for eval/matrix_rollup.py pair suffix raises UnicodeEncodeError for a matrix cell name with a lone surrogate #717 os.fsencode instead of surrogatepass (raises for n\ud800). Each fails the test of its site.
• The tests of the file-system-encoding independence (8 of 34) on the code before the surrogateescape commit, where the sites used os.fsencode: all 8 fail with UnicodeEncodeError, the 26 others pass.
• Not run: Python 3.11, 3.12 and 3.14 (this run used 3.13), the Windows legs, the gitleaks and semgrep scans, and the docker steps.

Revert

A plain revert restores the UnicodeEncodeError for these inputs. Nothing is stored: a cache directory or file made for a surrogate-escaped name while this was in is not read by the old code.

Checklist

• Tests added/updated and pytest passes
• ruff, mypy, and schema checks pass
• Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
• Self-reviewed per "Before opening a PR" in AGENTS.md
• No ad-hoc run artifacts committed
• New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

[exai-bot]

✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s).

CI at c01b429: 28 passing, 3 skipped.

Review details

• Commit reviewed: c01b429 against merge base 4860cdb
• Coverage: 2 files in 1 groups
• Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
• Usage: 96k tokens (63k from cache), 21s
• Tool calls: 9 (8 file_read, 1 code_search)
• small change set: 2 files

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.72%. Comparing base (1bb8978) to head (c01b429).
⚠️ Report is 11 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #761      +/-   ##
==========================================
+ Coverage   95.71%   95.72%   +0.01%     
==========================================
  Files         413      415       +2     
  Lines       55236    55367     +131     
  Branches     9501     9516      +15     
==========================================
+ Hits        52867    53002     +135     
+ Misses       1466     1463       -3     
+ Partials      903      902       -1     

Files with missing lines	Coverage Δ
dokimi/dokimi/caching.py	98.76% <100.00%> (ø)
dokimi/dokimi/eval/leakgate.py	99.03% <100.00%> (ø)
dokimi/dokimi/eval/matrix_rollup.py	90.00% <100.00%> (ø)
dokimi/dokimi/eval_coverage.py	99.17% <100.00%> (ø)
dokimi/dokimi/harnesses/harbor_export.py	91.22% <100.00%> (ø)
dokimi/dokimi/harnesses/runner.py	99.19% <100.00%> (ø)
dokimi/dokimi/mcp/url_client.py	94.44% <100.00%> (ø)
dokimi/dokimi/novelty/bundle.py	95.12% <100.00%> (ø)
dokimi/dokimi/orchestrator/pipeline.py	91.89% <100.00%> (ø)
dokimi/dokimi/repos/conformance.py	99.21% <100.00%> (ø)
... and 4 more

... and 17 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[antonwolfy]

Review points answered at 4860cdb; 1 checks still pending (Package install smoke test (py3.11)); every other check passed.

[vchamarthi]

Pushed 1 commit to fix/surrogate-digests (4860cdb..c01b429).

• skill_reach.token falls back to surrogatepass when surrogateescape cannot encode.
  Run: the full suite with main merged in, 20521 passed.