2.4.0: streaming Chat tab (SSE) - #35
Conversation
The dashboard Chat tab now renders the assistant reply token-by-token over a text/event-stream response instead of a blank "Thinking..." wait. First token arrives at ~1.6s warm (measured live on FastFlowLM qwen3.5:4b) instead of after the whole completion. - ffp_chat: stream_send() generator (delta -> done/error terminal), _default_llm_stream + _parse_sse_delta SSE parser; send() refactored onto shared _build_send_context/_persist_turn. Persistence re-reads the thread store under the write-lock so a concurrent delete/append during a multi-second stream is never clobbered by a stale snapshot, and a client disconnect (GeneratorExit) still persists the partial reply. - ffp_daemon: _STREAM_ACTIONS + _sse_frame + Handler._stream_action; do_POST streams event-stream frames while keeping the X-FFP-API CSRF + Host gates. - app.js: streamChat() fetch-body reader + parseSseFrame(); sendChat() streams with a transparent fallback to the one-shot chat_send action on any older daemon or open failure. - AHK paste path + grammar/prompt modes untouched (whole-output). Works on FastFlowLM and Ollama (shared OpenAI-compatible SSE). - 25 new tests: SSE parse, stream_send accumulate/persist/notes/partial-on- error, concurrency lost-update guard, client-disconnect persist, daemon end-to-end SSE + CSRF + disconnect. Fixes found by an adversarial review pass. Phase 4 (faster-decode route) deferred: its plan trigger (p50 > 15s) is already resolved by 2.3.0 (3.38s warm) and streaming delivers the perceived speed it targeted. SPEC V37/T27; CHANGELOG Unreleased. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bump version across _version.py, pyproject.toml, installer.iss, README (+ What's new in 2.4), sign.ps1 example, and the version-sync test pin. Move CHANGELOG Unreleased -> 2.4.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3bf3071669
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| idx = _thread_index(threads, str(thread_id or "")) | ||
| if idx < 0: | ||
| thread = {"thread_id": thread_id, "title": "New chat", "updated_at": _now_iso(), "history": []} | ||
| threads.insert(0, thread) | ||
| idx = 0 |
There was a problem hiding this comment.
When a user deletes the same existing thread while its reply is streaming, the delete completes under _write_lock, but this final persistence pass sees the thread missing and recreates it with the new turn. Unlike the previous one-shot action, which held the lock for the entire send, this makes an explicit deletion appear to succeed and then resurrects the conversation; distinguish a new thread from an existing thread that disappeared, or otherwise preserve the deletion.
Useful? React with 👍 / 👎.
| with resp: | ||
| for raw in resp: # http.client yields lines as they flush off the socket | ||
| yield from _parse_sse_delta(raw) |
There was a problem hiding this comment.
If the provider closes the response cleanly after emitting only part of a completion, iteration ends normally here and stream_send emits a done event because no exception was raised. Since _parse_sse_delta discards [DONE] without recording that it was received, a truncated response is persisted and presented as successful; track the terminator (or equivalent completion signal) and treat EOF before it as a stream error.
Useful? React with 👍 / 👎.
| const { value, done } = await reader.read(); | ||
| if (done) break; | ||
| buf += decoder.decode(value, { stream: true }); |
There was a problem hiding this comment.
When the daemon response ends after headers or partial delta frames but before an event: done or event: error frame, reader.read() can return done rather than throw, and this branch returns a result whose error remains empty. sendChat then reports success (and for a new conversation may retain an empty thread id) despite the interrupted request; record whether a terminal event was parsed and set an error when EOF arrives without one.
Useful? React with 👍 / 👎.
2 participants
2.4.0 — Streaming Chat tab
The dashboard Chat tab now renders the assistant reply token-by-token over a Server-Sent-Events stream instead of a blank "Thinking…" wait. Measured live on FastFlowLM
qwen3.5:4b: first token at 1.52–1.58 s vs the full-completion wait (≈2 s short, tens of seconds long).This is Phase 3 of
docs/prompt-mode-speed-quality-plan.md. Phase 4 (faster-decode route) is intentionally deferred — its plan trigger (p50 > 15 s) was already resolved by 2.3.0 (3.38 s warm), and streaming delivers the perceived-speed win it targeted.How it works
ffp_chat.stream_send()generator yieldsdeltaevents then adone/errorterminal;_default_llm_stream+_parse_sse_deltaparse the provider's OpenAI-compatible SSE.send()refactored onto shared_build_send_context/_persist_turn.ffp_daemon:_STREAM_ACTIONS+_sse_frame+Handler._stream_action;do_POSTstreamstext/event-streamwhile keeping theX-FFP-APICSRF + Host gates.app.js:streamChat()fetch-body reader +parseSseFrame();sendChat()streams with a transparent fallback to the one-shotchat_sendon any older daemon or open failure.prompt:modes (whole-output). Works on FastFlowLM and Ollama.Correctness (two bugs found by an adversarial review pass, both fixed + tested)
_persist_turnnow re-reads the thread store under the write-lock (atomic read-modify-write), so a concurrent chat/delete during a multi-second stream can't be clobbered by a stale snapshot.except GeneratorExitpersists the partial reply when the client drops (previously skipped becauseGeneratorExitisn't anException).Verification
--check).done, partial persisted correctly.🤖 Generated with Claude Code