fix(server): preserve UTF-8 across stream boundaries

[Amidwestnoob]

Summary

• Preserve valid UTF-8 code points when token output splits a multibyte sequence across stream chunks.
• Reject malformed and impossible UTF-8 prefixes without retaining unbounded state.
• Discard a pending incomplete suffix when a stop sequence matches.
• Flush a genuinely truncated final code point as one replacement character.
• Use one finish-reason policy for streaming responses, non-streaming responses, and completion logs.
• Preserve semantic reasons such as tool_calls; report capped output as length / max_tokens.

Problem

The emitter sanitized each token fragment independently. A valid multibyte code point split across fragments could therefore become replacement characters. Finish-reason derivation also differed between API responses and the final server log.

Validation

• Release dflash_server build passed.
• Server unit suite: 344/344 passed.
• Feature gate: 126 assertions, 0 failures.
• CUDA communicator, Qwen3.5 tensor split, and two-GPU NCCL all-reduce tests passed.
• Live streaming and non-streaming output preserved café — π ∈ ∅ ⊆ and ∀∃∈∅⊆⊇∩≠→ in layer and tensor modes.
• Live capped output returned and logged length.
• Live tool output preserved tool_calls.
• A one-hour mixed runtime test remained healthy with no transport, CUDA, NCCL, fatal, or replacement-character failures.
• Production was restored unchanged after validation.

Patch SHA-256:

1c7b29b1fee3db63e52b696bc7e8812110f396feb1fd2b09ffa614a69a0470db

Scope

This patch changes stream assembly and finish-reason reporting only. It does not change tensor-parallel math, speculative decoding, model loading, draft compatibility, or production configuration.

[cubic-dev-ai]

All reported issues were addressed across 6 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>