2.4.0: streaming Chat tab (SSE)

CHANGELOG.md

@@ -2,6 +2,18 @@
 
 ## Unreleased
 
+## 2.4.0
+
+**Streaming chat.** The dashboard Chat tab renders replies token-by-token instead of waiting for the whole answer, so the response starts appearing at warm time-to-first-token rather than after the full completion.
+
+### Added
+
+- **The Chat tab streams replies as they generate.** Instead of a blank "Thinking…" wait until the whole answer lands, the assistant's reply now fills in token-by-token over a Server-Sent-Events stream — first text typically appears at ~1.6 s (warm TTFT) rather than after the full completion. Measured live on FastFlowLM `qwen3.5:4b`: first token at **1.52–1.58 s** versus a ~2 s (short) to tens-of-seconds (long) full-completion wait. Prompt-mode/grammar hotkeys and the AHK paste path are unchanged (they still return whole output). A daemon without the streaming endpoint, or any failure to open the stream, transparently falls back to the previous one-shot request, so nothing regresses. Works on both FastFlowLM and Ollama (shared OpenAI-compatible SSE). A mid-stream provider drop or a client disconnect still saves the partial reply.
+
+### Internal
+
+- Streaming persistence re-reads the thread store under the daemon write-lock (atomic read-modify-write), so a concurrent chat write or thread delete during a multi-second stream is never clobbered by a stale snapshot; a client disconnect (`GeneratorExit`) still persists the partial turn. Both were caught by an adversarial review pass and pinned by discriminating tests (SPEC V37 / T27).
+
 ## 2.3.0
 
 **Prompt mode, faster and better grounded.** The default `prompt:` path now emits a compact copy-paste-ready agent prompt in a few seconds without filling gaps with conventional-but-unstated requirements.

README.md

@@ -4,7 +4,11 @@ Flowkey is a Windows desktop assistant that adds local-LLM hotkeys for grammar f
 
 Everything runs locally through [FastFlowLM](https://fastflowlm.com) (AMD Ryzen AI NPU) or, on machines without the NPU, through [Ollama](https://ollama.com) (CPU/GPU) as a secondary provider. No cloud service, analytics, or telemetry is used by the app.
 
-Current version: `2.3.0`
+Current version: `2.4.0`
+
+## What's new in 2.4
+
+- **Chat streams as it thinks.** The dashboard **Chat** tab now fills the reply in token-by-token over a live stream instead of waiting on a blank "Thinking…" until the whole answer is ready — the first words typically appear in ~1.6 s (warm) rather than after the full response. It works on both FastFlowLM and Ollama, and if anything interrupts the stream the partial reply is kept and the error is shown. Grammar, `prompt:`, and the other hotkeys are unchanged (they still paste the finished result).
 
 ## What's new in 2.3
 

SPEC.md

@@ -88,6 +88,7 @@ Caveman-encoded (compression, not amputation). Paths / ids / action names / numb
 - V34: FastFlowLM force restart → old port observed closed before new spawn; ⊥ return `already_running` from dying instance
 - V35: default v2 → exactly 1 short LLM draft call; ⊥ anti-echo/rescue calls; V33 finalizer supplies surfaced 4-section contract
 - V36: pre-2.3 prompt_builder identity cfg (⊥ `prompt_version`, legacy default fields) → migrate v2+concise; any custom field → preserve
+- V37: `chat_send_stream` streams reply deltas as `text/event-stream` (SSE `data:`/`event:` frames), persists the full-or-partial turn exactly once under the daemon write-lock; CSRF `X-FFP-API` + Host gates still apply; dashboard falls back to `chat_send` iff the stream ⊥ opens (never after tokens arrive); `send()`/AHK/grammar/prompt paths unchanged (⊥ streamed)
 
 ## §T tasks
 
@@ -119,6 +120,7 @@ T23|x|prompt-v2 fixed A/B speed+quality harness + tests|V29
 T24|x|prompt-v2 default + v1 rollback selector + 240/320/420 caps|V24,V27,V28,V30,V32
 T25|x|FastFlowLM startup+idle keep-warm + cold/warm measurement support|V31
 T26|x|2.3.0 release evidence + version/docs rebaseline after A/B gate passes|V18,V29,V33,V34,V35,V36
+T27|x|streaming Chat tab (SSE): `ffp_chat.stream_send` + daemon `_STREAM_ACTIONS`/`_sse_frame`/`_stream_action` + `app.js` fetch-reader w/ `chat_send` fallback; live FLM first-token 1.58s|V37
 ```
 
 ## §B bugs

docs/prompt-mode-speed-quality-plan.md

@@ -1,8 +1,8 @@
 # Prompt Mode Speed + Quality Refresh Plan ("prompt v2")
 
-Date: 2026-07-09
-Status: PLAN ONLY — do not implement. Includes measured baseline + the exact
-measurement harness to build.
+Date: 2026-07-09 (Phases 0–2 shipped in 2.3.0; Phase 3 done post-2.3.0; Phase 4 deferred)
+Status: Phases 0–2 SHIPPED (2.3.0). Phase 3 (streaming Chat tab) DONE on
+`feat/streaming-chat`. Phase 4 (faster route) trigger not met → deferred.
 Target release: **2.3.0** (re-baselines the default `prompt:` output — see Risk)
 
 ## Problem
@@ -143,11 +143,24 @@ like the 2.2.0 prompt-builder eval.
   and likely hits the goal alone.
 - **Phase 2 — Warm-model guarantee.** Ensure FLM stays loaded so nobody eats the
   cold-load tax; verify with a cold-vs-warm measurement.
-- **Phase 3 (optional) — Streaming.** SSE from daemon → dashboard Chat/preview
-  for perceived latency; AHK paste path stays whole-output.
-- **Phase 4 (optional) — Faster route.** Only if p50 still > 15 s after 1–2:
-  evaluate an iGPU/LM-Studio decode route behind the provider split, gated on the
-  same quality eval.
+- **Phase 3 (optional) — Streaming. ✅ DONE (post-2.3.0, `feat/streaming-chat`).**
+  The surface that actually benefits is the **Chat tab** (replies up to 1024
+  tokens, formerly a blank wait) — `prompt:` has no live dashboard surface and the
+  AHK paste path pastes at the end, so streaming there is moot. Delivered as a
+  `text/event-stream` response over a `fetch()` POST (keeps the `X-FFP-API` CSRF
+  gate that `EventSource` cannot send): `ffp_chat.stream_send` generator →
+  daemon `_STREAM_ACTIONS`/`_sse_frame`/`_stream_action` → `app.js` fetch-reader,
+  with a transparent fallback to the one-shot `chat_send` on any older daemon.
+  Measured live on FastFlowLM `qwen3.5:4b`: first token **1.58 s** vs **2.10 s**
+  total — text flows in instead of a blank wait. AHK/grammar/prompt paths
+  untouched. See SPEC V37 / T27.
+- **Phase 4 (optional) — Faster route. ⏸ TRIGGER NOT MET — deferred.** The plan
+  gates this on "p50 still > 15 s after phases 1–2," but 2.3.0 put prompt p50 at
+  **3.38 s warm** and Phase 3 streaming already delivers the *perceived* speed
+  this phase was chasing. A second NPU-decode provider (LM Studio/Lemonade) also
+  can't co-serve the NPU with FastFlowLM (both wedge — see the benchmark
+  evidence), so it would add real complexity for no measured need. Revisit only
+  if a concrete slow surface appears.
 
 ## Integration with the existing prompt-builder (2.2.0)
 

installer/installer.iss

@@ -41,7 +41,7 @@
 #define AppURL        "https://github.com/agr77one/Fastflow"
 #define AppExeName    "Flowkey.exe"  ; symbolic — actual launchers below
 ; Keep in lockstep with scripts\_version.py.
-#define AppVersion    "2.3.0"
+#define AppVersion    "2.4.0"
 
 [Setup]
 AppId={{8A4F1E6C-9B3D-4E62-9F7A-FASTFLOW140}}

installer/sign.ps1

@@ -52,7 +52,7 @@
 .EXAMPLE
     # Sign the installer
     $env:FFP_SIGN_PFX_PASSWORD = "ChangeMe!"
-    .\sign.ps1 -FilePath ..\out\Flowkey-Setup-2.3.0.exe
+    .\sign.ps1 -FilePath ..\out\Flowkey-Setup-2.4.0.exe
 #>
 
 [CmdletBinding(DefaultParameterSetName = "Sign")]

pyproject.toml

@@ -17,7 +17,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "fastflowprompt"
-version = "2.3.0"
+version = "2.4.0"
 description = "Local-LLM-powered grammar fix, prompt rewrite, chat, and dashboard for Windows."
 readme = "README.md"
 requires-python = ">=3.11"

scripts/_version.py

@@ -1,3 +1,3 @@
 """Single source of truth for the app version. Read by grammar_fix.py."""
 
-__version__ = "2.3.0"
+__version__ = "2.4.0"

scripts/ffp_chat.py

@@ -10,12 +10,14 @@
 
 from __future__ import annotations
 
+import contextlib
 import json
 import logging
 import time
 import urllib.error
 import urllib.request
 import uuid
+from collections.abc import Iterator
 
 import paths as _paths
 
@@ -240,6 +242,69 @@ def _default_llm_call(messages: list[dict]) -> str:
     return str((choices[0].get("message") or {}).get("content") or "").strip()
 
 
+def _parse_sse_delta(raw_line: str | bytes) -> Iterator[str]:
+    """Yield the assistant text delta(s) carried by one OpenAI-style SSE line.
+
+    The streaming ``/v1/chat/completions`` response is a sequence of
+    ``data: {json}`` lines (plus blank separators and a final ``data: [DONE]``),
+    identical on FastFlowLM and Ollama's OpenAI-compatible endpoint. Non-data
+    lines, the terminator, and malformed JSON yield nothing, so a partial or
+    noisy stream degrades to fewer tokens rather than an exception.
+    """
+    line = raw_line.decode("utf-8", errors="replace") if isinstance(raw_line, bytes) else str(raw_line)
+    line = line.strip()
+    if not line or not line.startswith("data:"):
+        return
+    data = line[len("data:"):].strip()
+    if not data or data == "[DONE]":
+        return
+    try:
+        obj = json.loads(data)
+    except (ValueError, TypeError):
+        return
+    for choice in obj.get("choices") or []:
+        delta = (choice.get("delta") or {}).get("content")
+        if delta:
+            yield str(delta)
+
+
+def _default_llm_stream(messages: list[dict]) -> Iterator[str]:
+    """Yield reply text deltas from the active provider's streaming chat endpoint.
+
+    Mirrors :func:`_default_llm_call` (same provider-resolved endpoint/model/auth
+    from ``grammar_fix``) but sets ``stream: True`` and iterates the SSE body so
+    the daemon can forward tokens to the dashboard as they arrive. Raises
+    RuntimeError on transport failure; a mid-stream drop simply ends iteration.
+    """
+    import grammar_fix
+
+    base_url = str(getattr(grammar_fix, "FLM_BASE_URL", "http://127.0.0.1:52625") or "").rstrip("/")
+    model = str(getattr(grammar_fix, "FLM_MODEL", "qwen3.5:4b") or "qwen3.5:4b")
+    bearer = str(getattr(grammar_fix, "LLM_AUTH_BEARER", "flm") or "")
+    timeout = int(getattr(grammar_fix, "FLM_TIMEOUT_SECONDS", 240) or 240)
+
+    body = json.dumps(
+        {
+            "model": model,
+            "messages": messages,
+            "temperature": TEMPERATURE,
+            "max_tokens": MAX_TOKENS,
+            "stream": True,
+        }
+    ).encode("utf-8")
+    headers = {"Content-Type": "application/json", "Accept": "text/event-stream"}
+    if bearer:
+        headers["Authorization"] = f"Bearer {bearer}"
+    req = urllib.request.Request(base_url + "/v1/chat/completions", data=body, headers=headers, method="POST")
+    try:
+        resp = urllib.request.urlopen(req, timeout=max(2, timeout))
+    except urllib.error.URLError as e:
+        raise RuntimeError(f"LLM unreachable at {base_url}: {getattr(e, 'reason', e)}") from e
+    with resp:
+        for raw in resp:  # http.client yields lines as they flush off the socket
+            yield from _parse_sse_delta(raw)
+
+
 def _derive_title(text: str) -> str:
     text = (text or "").strip()
     if not text:
@@ -250,28 +315,24 @@ def _derive_title(text: str) -> str:
     return (first[:TITLE_MAX_CHARS] + "…") if len(first) > TITLE_MAX_CHARS else first
 
 
-def send(thread_id: str = "", message: str = "", use_notes: bool = False, llm_call=None) -> dict:
-    """Append the user message, call the LLM (optionally grounded in the notes
-    vault), persist, and return ``{thread_id, title, reply, notes_used}``.
+def _build_send_context(thread_id: str, message: str, use_notes: bool) -> tuple[str, list[dict], list[str]]:
+    """Resolve the target thread id and assemble the model request.
 
-    ``llm_call(messages)->str`` is injectable for tests; defaults to the live
-    provider endpoint.
+    Returns ``(thread_id, messages, notes_used)`` — the resolved/new thread id,
+    the system+grounding+window+turn message list, and the grounding titles.
+    Only *reads* the thread store (for the sliding-window history); the write
+    happens later in :func:`_persist_turn`, which re-reads under the lock so the
+    read-here/write-later split can never clobber a concurrent write. Shared by
+    :func:`send` and :func:`stream_send` so both build identical requests.
     """
-    message = str(message or "").strip()
-    if not message:
-        raise ValueError("empty chat message")
-    llm_call = llm_call or _default_llm_call
-
     threads = load_threads()
     idx = _thread_index(threads, str(thread_id or "")) if thread_id else -1
     if idx < 0:
         thread_id = str(thread_id or "") or uuid.uuid4().hex
-        thread = {"thread_id": thread_id, "title": "New chat", "updated_at": _now_iso(), "history": []}
-        threads.insert(0, thread)
-        idx = 0
-    thread = threads[idx]
-    thread_id = thread.get("thread_id")
-    history: list[dict] = list(thread.get("history") or [])
+        history: list[dict] = []
+    else:
+        thread_id = threads[idx].get("thread_id")
+        history = list(threads[idx].get("history") or [])
 
     # Build the request: system + optional notes grounding + sliding window + new turn.
     messages: list[dict] = [{"role": "system", "content": SYSTEM_PROMPT}]
@@ -281,23 +342,126 @@ def send(thread_id: str = "", message: str = "", use_notes: bool = False, llm_ca
         if ctx:
             messages.append({"role": "system", "content": ctx})
             notes_used = titles
-    window = history[-(CONTEXT_WINDOW_TURNS * 2):]
-    for m in window:
+    for m in history[-(CONTEXT_WINDOW_TURNS * 2):]:
         role = str(m.get("role") or "")
         content = str(m.get("content") or "")
         if role in ("user", "assistant") and content:
             messages.append({"role": role, "content": content})
     messages.append({"role": "user", "content": message})
+    return thread_id, messages, notes_used
+
 
+def _persist_turn(thread_id: str, message: str, reply: str, save_lock=None) -> str:
+    """Append the user+assistant turn to ``thread_id`` and save; return its title.
+
+    The whole read-modify-write is wrapped in ``save_lock`` so it is atomic
+    against concurrent writers: it re-loads the current thread list *inside* the
+    lock (rather than trusting a snapshot captured before a multi-second stream),
+    resolves the thread by id, appends the turn, and rewrites. Without the
+    re-read, a stream that started before a concurrent delete/append would
+    overwrite the file with its stale snapshot and silently lose that write. The
+    one-shot ``chat_send`` action passes no lock because ``do_POST`` already holds
+    ``_write_lock`` across its whole handler; streaming passes the daemon's
+    ``_write_lock`` here because it runs outside that wrapper.
+    """
+    with (save_lock or contextlib.nullcontext()):
+        threads = load_threads()
+        idx = _thread_index(threads, str(thread_id or ""))
+        if idx < 0:
+            thread = {"thread_id": thread_id, "title": "New chat", "updated_at": _now_iso(), "history": []}
+            threads.insert(0, thread)
+            idx = 0
+        thread = threads[idx]
+        history: list[dict] = list(thread.get("history") or [])
+        history.append({"role": "user", "content": message})
+        history.append({"role": "assistant", "content": reply})
+        if not thread.get("title") or thread.get("title") == "New chat":
+            thread["title"] = _derive_title(message)
+        thread["history"] = history
+        thread["updated_at"] = _now_iso()
+        threads.pop(idx)
+        threads.insert(0, thread)
+        save_threads(threads[:MAX_THREADS])
+        return thread["title"]
+
+
+def send(thread_id: str = "", message: str = "", use_notes: bool = False, llm_call=None) -> dict:
+    """Append the user message, call the LLM (optionally grounded in the notes
+    vault), persist, and return ``{thread_id, title, reply, notes_used}``.
+
+    ``llm_call(messages)->str`` is injectable for tests; defaults to the live
+    provider endpoint.
+    """
+    message = str(message or "").strip()
+    if not message:
+        raise ValueError("empty chat message")
+    llm_call = llm_call or _default_llm_call
+
+    thread_id, messages, notes_used = _build_send_context(thread_id, message, use_notes)
     reply = str(llm_call(messages) or "").strip()
+    title = _persist_turn(thread_id, message, reply)
+    return {"thread_id": thread_id, "title": title, "reply": reply, "notes_used": notes_used}
+
+
+def stream_send(
+    thread_id: str = "",
+    message: str = "",
+    use_notes: bool = False,
+    llm_stream=None,
+    save_lock=None,
+) -> Iterator[dict]:
+    """Streaming twin of :func:`send`: yield reply deltas, then a terminal event.
+
+    Yields ``{"type": "delta", "delta": text}`` for each token chunk as it
+    arrives, then exactly one terminal event:
+    ``{"type": "done", thread_id, title, notes_used}`` on success, or
+    ``{"type": "error", error, thread_id, title, notes_used}`` if the provider
+    stream failed mid-flight. Whatever text arrived before a failure — a provider
+    drop *or* a client disconnect (``GeneratorExit`` thrown in when the daemon
+    stops iterating) — is still persisted, so the transcript never loses a
+    partial reply.
+
+    ``llm_stream(messages) -> Iterator[str]`` is injectable for tests; defaults
+    to the live provider's streaming endpoint. ``save_lock`` serializes the final
+    write with the daemon's other writers.
+    """
+    message = str(message or "").strip()
+    if not message:
+        raise ValueError("empty chat message")
+    llm_stream = llm_stream or _default_llm_stream
 
-    history.append({"role": "user", "content": message})
-    history.append({"role": "assistant", "content": reply})
-    if not thread.get("title") or thread.get("title") == "New chat":
-        thread["title"] = _derive_title(message)
-    thread["history"] = history
-    thread["updated_at"] = _now_iso()
-    threads.pop(idx)
-    threads.insert(0, thread)
-    save_threads(threads[:MAX_THREADS])
-    return {"thread_id": thread_id, "title": thread["title"], "reply": reply, "notes_used": notes_used}
+    thread_id, messages, notes_used = _build_send_context(thread_id, message, use_notes)
+    parts: list[str] = []
+    err: Exception | None = None
+    try:
+        for delta in llm_stream(messages):
+            delta = str(delta or "")
+            if not delta:
+                continue
+            parts.append(delta)
+            yield {"type": "delta", "delta": delta}
+    except GeneratorExit:
+        # The daemon abandoned the stream (client disconnected). We cannot yield
+        # anymore, but we still persist the partial reply — matching the contract
+        # and the non-streaming path — then let the generator close.
+        partial = "".join(parts).strip()
+        if partial:
+            with contextlib.suppress(Exception):
+                _persist_turn(thread_id, message, partial, save_lock=save_lock)
+        raise
+    except Exception as exc:  # transport drop / provider error mid-stream
+        err = exc
+        log.warning("chat stream failed after %d chunk(s): %s", len(parts), exc)
+
+    reply = "".join(parts).strip()
+    title = _persist_turn(thread_id, message, reply, save_lock=save_lock) if reply else "New chat"
+
+    terminal = {
+        "type": "error" if err else "done",
+        "thread_id": thread_id,
+        "title": title,
+        "notes_used": notes_used,
+    }
+    if err:
+        terminal["error"] = str(err) or "chat stream failed"
+    yield terminal