Canonicalize wallet JSON unicode keys

[tinyopsstudio]

Bounty #406

Summary:

• align browser wallet.js canonical JSON with server canonical_wallet_json() for Unicode strings and object keys
• ASCII-escape non-ASCII string content before signing, matching Python json.dumps(..., ensure_ascii=True)
• sort object keys by Unicode code point order so non-BMP keys do not diverge from Python sort_keys=True

Evidence:

• current wallet.js uses native JSON.stringify plus default JavaScript .sort(), so non-ASCII values are not escaped like the server and keys such as "\\ue000" and "😀" are ordered by UTF-16 code units instead of Python Unicode code points
• this can make browser-signed wallet transfer payloads fail server signature verification when memos or keys contain Unicode
• this addresses the remaining canonicalization gap called out on PR Refs #406: Canonicalize wallet unicode memos #507 with a focused implementation and regression
• no private keys, cookies, tokens, wallet JSON, browser storage, OAuth state, signatures, private data, price claims, liquidity claims, exchange claims, bridge promises, or off-ramp claims are added

Validation:

• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ./.venv/bin/python -m pytest tests/test_wallet_api.py::test_wallet_js_canonicalizes_unicode_like_server -q -> 1 passed
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ./.venv/bin/python -m pytest tests/test_wallet_api.py tests/test_wallets.py -q -> 44 passed
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ./.venv/bin/python -m pytest -q -> 415 passed
• ./.venv/bin/python -m ruff check tests/test_wallet_api.py -> passed
• ./.venv/bin/python -m ruff format --check tests/test_wallet_api.py -> 1 file already formatted
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ./.venv/bin/python -m mypy app -> success
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ./.venv/bin/python scripts/docs_smoke.py -> docs smoke ok
• git diff --check HEAD -> clean

Summary by CodeRabbit

• Bug Fixes

  • Enhanced payload canonicalization to ensure deterministic and consistent output across the platform.

• Tests

  • Added verification test to confirm wallet payload handling consistency between server and client components.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: eaeb3691-0c98-4ef2-8333-c401913f495f

📥 Commits

Reviewing files that changed from the base of the PR and between d8532d4 and 55f35ee.

📒 Files selected for processing (2)

• app/static/wallet.js
• tests/test_wallet_api.py

---

📝 Walkthrough

Walkthrough

Wallet.js updates stableJson to produce deterministic JSON using Unicode code-point–based key sorting and ASCII escaping. Two new helpers (asciiJson, compareCodePoints) support this. A new test verifies that the JavaScript and Python canonical JSON implementations produce matching output for Unicode-containing payloads.

Changes

Canonical JSON Serialization

Layer / File(s)	Summary
Canonical JSON helpers and stableJson rewrite app/static/wallet.js	asciiJson escapes non-ASCII characters as \uXXXX sequences. compareCodePoints performs lexicographic comparison by Unicode code points. stableJson rewrites object-key sorting to use code-point ordering and produce ASCII-escaped canonical output.
Cross-language canonicalization verification test tests/test_wallet_api.py	Adds shutil and subprocess imports for running Node commands. New test test_wallet_js_canonicalizes_unicode_like_server reads wallet.js, executes it in Node to call stableJson on a Unicode payload, and asserts the output matches Python's canonical_wallet_json implementation.

🚥 Pre-merge checks | ✅ 6

✅ Passed checks (6 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'Canonicalize wallet JSON unicode keys' directly names the changed surface and accurately reflects the main change: updating JSON canonicalization to handle Unicode keys correctly.
Description check	✅ Passed	The description includes all required template sections: Summary with clear bullets, Evidence with problem explanation and context, comprehensive validation results, and Bounty reference. All checklist items show passing status.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Mergework Public Artifact Hygiene	✅ Passed	README correctly describes MRWK as native ledger, prohibits current fiat/USDC/off-ramp claims, qualifies future paths, and avoids fabricated payout claims or security disclosure.
Bounty Pr Focus	✅ Passed	PR references bounty #406. Diff matches stated files: wallet.js with new canonicalization functions, test_wallet_api.py with Unicode test. Test validates Node/Python output alignment. No scope creep.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[xingxi0614-cpu]

Reviewed PR #561 at head 55f35ee9eaf4634879ee221919e1dd5faae6c819 for the wallet JSON Unicode canonicalization slice.

No blocker found in this reviewed scope.

Evidence checked:

• inspected app/static/wallet.js and tests/test_wallet_api.py;
• confirmed stableJson() now sorts object keys by Unicode code point order instead of JavaScript UTF-16 code-unit order;
• confirmed string values and object keys are ASCII-escaped through asciiJson(), aligning browser output with Python json.dumps(..., ensure_ascii=True) for non-ASCII payload content;
• confirmed the regression test executes wallet.js through Node and compares browser-side stableJson() output with server-side canonical_wallet_json() on a payload containing accented text, an emoji, and keys that expose UTF-16 vs Unicode-code-point ordering differences;
• checked CodeRabbit reported no actionable comments on this head.

Validation run locally:

• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ./.venv/Scripts/python.exe -m pytest tests/test_wallet_api.py::test_wallet_js_canonicalizes_unicode_like_server -q -> 1 passed
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ./.venv/Scripts/python.exe -m pytest tests/test_wallet_api.py tests/test_wallets.py -q -> 44 passed
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ./.venv/Scripts/python.exe -m pytest -q -> 415 passed
• ./.venv/Scripts/python.exe -m ruff check tests/test_wallet_api.py -> passed
• ./.venv/Scripts/python.exe -m ruff format --check tests/test_wallet_api.py -> 1 file already formatted
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ./.venv/Scripts/python.exe -m mypy app -> success
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 ./.venv/Scripts/python.exe scripts/docs_smoke.py -> docs smoke ok
• git diff --check origin/main...HEAD -> clean

This looks mergeable from my reviewed slice.

[eliasx45]

Current-head review for PR #561 at 55f35ee9eaf4634879ee221919e1dd5faae6c819.

I do not see a blocker in the wallet JSON canonicalization slice. The implementation changes the browser signing path from native JSON.stringify() key order to a deterministic serializer that sorts object keys by Unicode code point and ASCII-escapes string content before signing.

Additional evidence I checked beyond the existing focused regression:

node --check app\static\wallet.js
# passed

.\.venv\Scripts\python.exe -m pytest tests\test_wallet_api.py::test_wallet_js_canonicalizes_unicode_like_server tests\test_wallet_api.py::test_wallet_transfer_api_rejects_memo_control_characters -q
# 2 passed

.\.venv\Scripts\python.exe -m ruff check tests\test_wallet_api.py
# All checks passed!

.\.venv\Scripts\python.exe -m ruff format --check tests\test_wallet_api.py
# 1 file already formatted

git diff --check origin/main...HEAD
# clean

I also ran an ad hoc Node/Python canonicalization probe with escaped non-BMP keys and values (U+10000, U+1F600, and U+E000). Browser stableJson() matched Python canonical_wallet_json() exactly:

equal True
js_ascii {"memo":"caf\\u00e9 \\ud83d\\ude00","type":"probe","\\ue000":1,"\\ud800\\udc00":3,"\\ud83d\\ude00":2}
py_ascii {"memo":"caf\\u00e9 \\ud83d\\ude00","type":"probe","\\ue000":1,"\\ud800\\udc00":3,"\\ud83d\\ude00":2}

That covers the key UTF-16-vs-code-point ordering edge this PR is meant to fix. No private keys, wallet material, cookies, tokens, signatures, private data, price/liquidity/exchange/off-ramp claims, or fabricated payout claims were used.

[eliasx45]

No blocker found on this head.

I checked the browser/server canonicalization boundary because this is the risky part of the change. The new stableJson() path now sorts object keys by Unicode code point rather than UTF-16 unit order, and asciiJson() emits non-ASCII values/keys in the same escaped form the Python server signs through canonical_wallet_json(). That covers the important astral-key case where default JS sorting would otherwise disagree with Python.

Validated locally:

• git diff --check origin/main...HEAD
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 .\.venv\Scripts\python.exe -m pytest tests\test_wallet_api.py::test_wallet_js_canonicalizes_unicode_like_server tests\test_wallet_api.py::test_wallet_api_register_lookup_and_transfer tests\test_wallet_api.py::test_github_session_can_link_and_claim_wallet -q -> 3 passed
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 .\.venv\Scripts\python.exe -m pytest tests\test_wallet_api.py -q -> 33 passed
• node --check app\static\wallet.js
• .\.venv\Scripts\python.exe -m ruff check tests\test_wallet_api.py
• .\.venv\Scripts\python.exe -m ruff format --check tests\test_wallet_api.py
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 .\.venv\Scripts\python.exe -m mypy app\wallets.py app\ledger\service.py
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 .\.venv\Scripts\python.exe -m pytest tests\test_wallet_api.py tests\test_ledger.py -q -> 55 passed
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 .\.venv\Scripts\python.exe -m pytest -q -> 415 passed
• PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 .\.venv\Scripts\python.exe scripts\docs_smoke.py -> docs smoke ok