fix(translate): re-encode non-ASCII thoughtSignature through base64 at JSON emit

[steventohme]

Summary

Fixes the second-turn 400 INVALID_ARGUMENT that blocks every multi-turn Claude Code session against any Gemini 3.x model. Found 2026-07-21 while onboarding gemini-3.6-flash + gemini-3.5-flash-lite; reproduces identically against the already-deployed gemini-3.1-pro-preview too.

Root cause

Same multi-SDK report class — Google GenAI SDK #711, langchain-google #1570, gemini-cli #8003, #8011 — all about Google's thought_signature bytes field and its base64 wire shape.

Upstream flow is OK end-to-end:

1. Gemini's response carries thought_signature as a base64url-encoded ASCII string in the JSON.
2. embedSignatureInID smugs those bytes into the tool-call id field — base64-encodes the bytes of the (already-ASCII) base64 string, surfaces as __thought__<base64> suffix.
3. On the next request the Anthropic-side tool_use id carries the same suffix unchanged across the boundary.
4. extractSignatureFromID does DecodeString once, returning the original raw bytes of the upstream signature — exactly what Google gave us, just unpacked from the carrier.

The bug is at the JSON emit boundary. Step 4 returns those raw bytes as a Go string. Then sse.WriteJSONString (called by pw.Str) does:

for _, c := range s {
    switch {
    case c < 0x20, c == '
', c == '
':
        writeJSONUnicodeEscape(w, c)
    default:
        w.WriteRune(c)   // Go's range over a non-UTF-8 string converts invalid UTF-8 sequences to U+FFFD
    }
}

A real Gemini thought_signature contains byte sequences that aren't valid UTF-8 — every such sequence silently gets replaced with U+FFFD. On the next turn Google sees a string of � + accented/non-ASCII chars, fails to base64-decode it, and returns Invalid value at 'contents[1].parts[0].thought_signature' (TYPE_BYTES), Base64 decoding failed for ....

Fix

encodeSignatureForJSON re-encodes values containing non-ASCII bytes (c >= 0x80) through base64.RawURLEncoding at the three pw.Str(sig) emit sites. Plain-ASCII values (test fixtures like "ANTHROPIC_SIG", and also the upstream-delivered base64 values themselves which never trip the carrier round-trip) bypass the rewrap since they wire safely as-is.

The non-ASCII round-trip is therefore base64s once for the carrier, then base64s again for the JSON wire: at Google, the JSON-rest layer auto-decodes the outer base64 to bytes — net zero extra encodings on the contract. Same upstream value reaches Google on the second turn that was served on the first.

Tests

• TestEncodeSignatureForJSON_PreservesNonASCIISignatureBytes — table-driven unit test on the helper covering the four cases (ASCII pass-through, base64url pass-through, non-ASCII raw bytes, mixed printable/non-ASCII).
• TestPrepareGemini_ThoughtSignatureCarrierSurvivesMultiTurn — end-to-end repro through PrepareGemini. Builds a non-UTF-8 signature, embeds it in a tool-use id, exercises the full multi-turn translate path, asserts the emitted thought_signature JSON value base64-decodes back to the exact original bytes.
• TestSanityCarrierRoundTripLocked — locks the carrier convention (delimiter string + decode path) so the regression test isn't silently testing the wrong shape.

Existing signature round-trip tests (TestPrepareGemini_FromAnthropic_SystemAndToolUseRoundTripsSignature, TestPrepareGemini_FromAnthropic_ToolUseSignatureSurvivesUnknownFieldStripping, TestSanitizeToolUseID_PreservesLongThoughtSignatureID) all still pass — they use ASCII fixtures that the helper correctly bypasses.

go build ./..., go vet ./..., and the full go test ./... (every package) pass.

Workaround for clients in the meantime

If you can't ship this immediately and need to unblock real multi-turn traffic against Gemini 3.x on your local branch, dropping the body through echo -n "$sig" | xxd -p | xxd -p -r | base64 -w0 (one decode-then-re-encode round-trip) before writing has been used as a manual workaround in test scripts.

🤖 Generated with Claude Code

[claude]

Claude finished @steventohme's task —— View job

---

• Run git diff and analyze added comment lines
• Flag verbose comment blocks (3+ lines)
• Post review suggestions

Posted 4 comment-length suggestions (advisory, won't block merge).

[workweave-bot]

Advisory only — comment-length nits. Won't block merge.

[greptile-apps]

T-Rex Logs

What T-Rex did

• Reproduced the ASCII raw bytes skip encoding issue in a focused carrier test that exercises ParseAnthropic and PrepareGemini, showing thoughtSignature as escaped raw controls instead of base64url and that base64url decoding fails at byte 0.
• Validated that the failure is resolved in HEAD by running the after-state tests, with key tests passing at exit code 0 and consistent results across HEAD^ and HEAD.

_{Ran code and verified through T-Rex}

_{Reviews (1): Last reviewed commit: "fix(translate): re-encode non-ASCII thou..." | Re-trigger Greptile}

[claude]

Claude finished @steventohme's task —— View job

---

• Run git diff and analyze added comment lines
• Flag verbose comment blocks (3+ lines)
• Post review suggestions (or stay silent if nothing to flag)

All added comment blocks are ≤ 5 lines and address genuine non-obvious WHYs. Prior suggestions from the first run were applied in 678345d. Nothing to flag.

View job