Chat backend: streaming answers, cited passages, renaming threads

CHANGELOG.md

@@ -4,6 +4,47 @@
 
 ### Added
 
+- Design for the chat redesign
+  (`docs/superpowers/specs/2026-09-26-chat-redesign-design.md`), with the
+  approved mockups in `docs/design/chat-redesign/`. It covers:
+  - streaming answers;
+  - numbered citation seals that open a Sources panel with brief, credited
+    quotes;
+  - renaming and searching threads;
+  - a dark theme;
+  - keyboard shortcuts;
+  - a phone layout.
+
+  The work splits into a backend PR and a frontend PR. The implementation plan
+  is in `docs/superpowers/plans/2026-09-26-chat-redesign.md`.
+
+- `PATCH /api/chat/threads/<id>` renames a thread (1–200 characters, trimmed;
+  30 per minute). It leaves `updated_at`, and so the list order, alone.
+- Each citation now carries its id, and a SHA-256 of the passage text it used
+  (`message_sources.content_sha256`, migration `9c1e4b7a2d10`). Older citations
+  keep NULL.
+- `src/services/passages.py` turns a citation into a brief quote: at most 300
+  characters, from the part of the passage most like the answer, with its kind,
+  title, document, section, a safe `https` link and, for Wikipedia, the
+  registry's CC BY-SA credit. `registry.registered(id)` reads one entry.
+- `GET /api/chat/sources/<id>` returns one of your citations as a brief quote,
+  with status `ok`, `unverified` (older answers, no hash), `changed` or
+  `missing` (60 per minute). It is addressed by citation id, never chunk id;
+  another user's citation is a 404.
+- Generation is an event stream: `answer_events` and `answer_query_events` yield
+  stages, notices, text deltas, `Reset` and a final `Done`. Text streams only
+  once the answer is grounded, and a closed stream closes the model's.
+  `answer()` and `answer_query()` are unchanged wrappers.
+- `POST /api/chat/threads/<id>/messages/stream` streams an answer as
+  Server-Sent Events (`user_message`, `stage`, `notice`, `delta`, `reset`,
+  `done`, `error`). Errors before the stream are ordinary responses; the
+  question is committed first, the answer is saved before `done`, and spend is
+  recorded exactly once, also when the client goes away. It shares one
+  15-per-minute limit with the JSON send route (`scope="chat_send"`).
+- ADR 0027: streaming answers over Server-Sent Events, cited passages
+  (addressed by citation id, hash-checked, quoted briefly with their credit)
+  and renaming threads. It amends ADR 0007 and ADR 0022; sub-project 3's ADR
+  becomes 0028. The README documents the chat API, its limits and the stream.
 - Design and ADR 0026 for California's SBCs, sub-project 2
   (`docs/superpowers/specs/2026-09-25-ca-sbc-design.md`). CMS's California
   file has no SBC links, so each plan's link comes from a hand-built manifest.
@@ -360,6 +401,11 @@
 
 ### Changed
 
+- **HTTP errors are JSON:** 404, 405, 429 and the rest answer with
+  `{"error": "not found"}` and similar instead of Werkzeug's HTML page. The
+  status code and headers (`Allow`, `Retry-After`) are kept.
+- `unsafe_reason` moved to `src/core/urls.py`, so the API can vet links too;
+  `src/ingestion/sbc/fetch.py` still re-exports it.
 - `docs/runbooks/deploy.md` rewritten from what actually worked. The previous
   version would have failed mid-deploy four times over: it set
   `--public-access None` while step 6 restored from a laptop, created the app
@@ -513,6 +559,9 @@
 
 ### Fixed
 
+- **A streamed answer's spend survives a failed save.** Spend is marked as
+  recorded only after its commit succeeds. Before, a rolled-back commit took
+  the spend with it, and it was never recorded again.
 - `make ingest-sbc` counted a recorded failure twice: once as "already
   current" and again as a "recorded failure skipped". So a run over blocked
   documents read as healthier than it was. It now counts it once, as skipped.
@@ -621,6 +670,10 @@
 
 ### Security
 
+- **A streamed round that never reports its usage is counted by estimate:**
+  the whole prompt plus the output cap. This covers a client that
+  disconnects, Stop, or a dropped connection. Without it, ending streams
+  early would spend past the daily token budget (OWASP LLM10).
 - Plan-summary links from CMS render only as `http(s)` URLs, never `javascript:` or
   `data:`, and open with `noopener noreferrer`.
 - Composer discloses that messages and retrieved sources are sent to

README.md

@@ -14,9 +14,10 @@ Because HealthCare.gov is the authoritative consumer source, answers about healt
 
 ## Features
 
-- **Conversational Q&A**, organized into threads, over a curated insurance knowledge base
+- **Conversational Q&A**, organized into threads you can rename, over a curated insurance knowledge base
 - **Hybrid RAG pipeline** — BM25 + pgvector semantic search, cross-encoder reranked and relevance-filtered before reaching the LLM, so weakly-relevant matches never become context
 - **Cited answers** — every reply lists the source passages and their relevance score, and the plans it compared in a side-by-side table; both survive reloading the conversation. Each plan in the table says whether its Summary of Benefits was read, and why not (ADR 0017)
+- **Streamed answers, readable sources** — answers stream in as they are written, with progress along the way; each citation can be opened to a brief quote of the passage it used, checked against a hash of what was cited, with its link and licence credit (ADR 0027)
 - **Real plan comparison** — ask about ACA Marketplace plans; the model searches the ingested catalog and CMS prices the plans live for your age (ADR 0010). In California, premiums are CMS's filed rates for your age and ZIP's rating area, and answers name Covered California (ADR 0024). It compares plans and never recommends one
 - **A profile, kept off the model** — signup takes a ZIP code, date of birth and county, and plan questions use them without their ever being sent to the LLM. Nobody under 13 can sign up (ADR 0012)
 - **JWT-authenticated API** — only a signed-in user can query, and only ever sees their own threads
@@ -55,6 +56,41 @@ User ──► React UI ──► Flask API ──► Retrieval ──► LLM 
 
 Both halves share the same embedding model (`src/core/embedding.py`) so ingestion-time and query-time vectors stay comparable.
 
+### Chat API
+
+Every chat endpoint needs a JWT, and a thread or citation that isn't yours is a
+404, never a 403. Each is rate-limited per user and answers 429 with
+`Retry-After` past its limit.
+
+| Endpoint | Does | Limit |
+| --- | --- | --- |
+| `GET /api/chat/threads` | Your threads, most recently active first | 60/min |
+| `POST /api/chat/threads` | Starts a thread | 30/min |
+| `PATCH /api/chat/threads/<id>` | Renames it (1–200 characters); its place in the list stays | 30/min |
+| `DELETE /api/chat/threads/<id>` | Deletes it with its messages | 30/min |
+| `GET /api/chat/threads/<id>/messages` | The transcript, with citations and plans | 60/min |
+| `POST /api/chat/threads/<id>/messages` | Asks, and returns the answer as JSON | 15/min, shared with the stream |
+| `POST /api/chat/threads/<id>/messages/stream` | Asks, and streams the answer | 15/min, shared with the JSON route |
+| `GET /api/chat/sources/<id>` | One of your citations, as a brief quote | 60/min |
+
+**Streaming** ([ADR 0027](docs/decisions/0027-streaming-answers-and-cited-passages.md)). The stream route sends Server-Sent Events:
+`user_message` (the saved question), then `stage` for progress
+(`understanding`, `searching`, `plans`, `coverage`, `writing`), `notice`,
+`delta` for answer text and `reset` to discard it, and finally `done` with the
+saved answer and thread, or `error`. Errors that can be known up front (401,
+404, 422, 429) come back as ordinary responses before the stream starts. Text
+is streamed only once the answer is grounded, the answer is saved before
+`done` is sent, and a client that goes away saves no answer but is still
+charged for what the model wrote. The frontend reads the stream with `fetch`,
+since `EventSource` can't send a POST with a Bearer header.
+
+**Cited passages.** `GET /api/chat/sources/<id>` takes a citation's own id
+(each source in a message carries one) and returns its kind, title, document,
+section, link, licence and a quote of at most 300 characters, taken from the
+part of the passage most like the answer. Its `status` says whether the quote
+can be trusted: `ok`, `unverified` (an answer from before hashes were stored),
+`changed` or `missing`; the last two carry no quote.
+
 ## Project Structure
 
 ```

docs/decisions/0007-persisting-citations.md

@@ -5,6 +5,9 @@
 Accepted. Amended by ADR 0017: an answer that called `plan_coverage`
 saves only the corpus sources it cites.
 
+Amended by ADR 0027: each citation also stores a SHA-256 of the passage
+text it used, so a rebuilt chunk is never shown as what was cited.
+
 ## Context
 
 Citations were returned once and never stored. `POST /messages` answers with

docs/decisions/0022-the-api-as-a-container.md

@@ -6,6 +6,9 @@ Accepted. Closes the first of ADR 0002's two deferred items ("Docker build +
 image vulnerability scan. There is no Dockerfile for the app."). Depends on
 ADR 0021, without which the image cannot serve a request.
 
+Amended by ADR 0027: an answer stream holds one gthread thread for as long
+as it runs, so the 8 threads also bound concurrent streams.
+
 ## Context
 
 The app has only ever run on the machine that built its corpus. Everything

docs/decisions/0027-streaming-answers-and-cited-passages.md

@@ -0,0 +1,143 @@
+# 0027 — Streaming answers, cited passages, and renaming threads
+
+## Status
+
+Accepted. Part of the chat redesign
+([spec](../superpowers/specs/2026-09-26-chat-redesign-design.md), section 3).
+It amends two earlier ADRs:
+
+- **ADR 0007.** A citation now also stores a SHA-256 of the passage text the
+  answer used (`message_sources.content_sha256`), so a passage can be shown
+  later only while it still says what was cited.
+- **ADR 0022.** An open answer stream holds one of the worker's 8 gthread
+  threads for as long as it runs, as a blocking request already did.
+
+Sub-project 3 of the California expansion, which had planned to take this
+number, takes ADR 0028 instead.
+
+## Context
+
+The chat worked, but three things made it read like a prototype:
+
+- **Answers arrived all at once**, after 5 to 40 seconds behind a spinner:
+  a rewrite, a search, up to two tool rounds and the reply, with nothing shown
+  in between.
+- **Sources couldn't be read.** A citation was a file name and a match score.
+  A user had no way to see the text an answer relied on, and the chunk behind
+  a citation can be rebuilt in place by the next ingest (ADR 0007), so its
+  text today may not be what was cited.
+- **Threads couldn't be renamed.** Every thread kept the first 80 characters
+  of its first question.
+
+## Decision
+
+### Streaming: Server-Sent Events over `fetch`
+
+`POST /api/chat/threads/<id>/messages/stream` takes the same body as
+`POST …/messages` and answers `200 text/event-stream`, with
+`Cache-Control: no-cache` and `X-Accel-Buffering: no`. The browser reads it
+with `fetch` and a stream reader: `EventSource` cannot send a POST or a Bearer
+header. Each event is `event: <name>\ndata: <json>\n\n`:
+
+| Event | Data | Meaning |
+| --- | --- | --- |
+| `user_message` | `{message}` | The saved question |
+| `stage` | `{stage}`: `understanding`, `searching`, `plans`, `coverage` or `writing` | Progress |
+| `notice` | `{text}` | A server-written notice (ADR 0024, 0026), sent as soon as a tool returns it |
+| `delta` | `{text}` | Answer text to append |
+| `reset` | `{}` | Discard the deltas so far |
+| `done` | `{message, thread}` | The saved answer: authoritative, and the client renders it |
+| `error` | `{error: "generation failed"}` | The model failed. The question stays and nothing else is saved |
+
+Every event and field is built by server code. The model supplies only the
+text of a `delta`, which is the text `done` then carries; the client renders
+it as Markdown without raw HTML.
+
+**Generation is one loop with two completion calls.** `answer_events` yields
+`Stage`, `Notice`, `Delta` and `Reset` as they happen and ends with
+`Done(answer)`. `answer()` drains it, so the CLI, the evals and the JSON route
+are unchanged. Blocking and streaming share the tool loop, the fallbacks, the
+coverage-citation filter and the notices; only the model call differs.
+
+**Text streams only once the answer is grounded,** meaning chunks were
+retrieved or a tool has run. Before that it is held back, so the "no chunks
+and no search" refusal (ADR 0010) never has to take back text already shown.
+A round that streamed text and then called a tool is followed by `reset`.
+
+**Every check that can fail comes before the stream,** so its error is an
+ordinary response: 401, 404 for ownership, 422 for the body, and 429 for the
+limiter or the daily token budget. The question is then committed, so no
+transaction is open while the model writes. After `Done` the answer is saved
+through the same helper as the JSON route (`_save_answer`) and committed, and
+only then is `done` sent. `done` always carries exactly what was saved.
+
+**Spend is recorded exactly once, in every ending:** with the answer, on a
+model error, and in a `finally` that also runs when the client goes away
+(`GeneratorExit`). A client that goes away also closes the model's stream, so
+the model stops writing. No partial answer is saved.
+
+**Both send routes share one limit,** `limiter.shared_limit("15 per minute",
+scope="chat_send")`, so a user can't double their rate by using both.
+
+### Passages: by citation id, briefly quoted
+
+`GET /api/chat/sources/<source_id>` (60 per minute) returns one of the
+current user's citations as it can be shown now.
+
+- **Addressed by the citation's own id,** joined through its message to a
+  thread the user owns. Anything else, or a malformed id, is a 404, never a
+  403. Chunk ids are guessable, so there is deliberately no lookup by chunk id.
+- **Status, from the stored hash:**
+
+  | `status` | When | `quote` |
+  | --- | --- | --- |
+  | `ok` | The hash matches the text now | the excerpt |
+  | `unverified` | No hash (an answer from before this ADR), and the chunk exists | the excerpt; the UI says it may have changed |
+  | `changed` | The hash doesn't match | `null` |
+  | `missing` | The chunk is gone | `null` |
+
+- **Brief quotes.** At most 300 characters, taken from the part of the passage
+  most like the answer, not its first 300 characters: segments are scored by
+  the distinct words they share with the answer, and the best one and those
+  after it are kept while they fit. `… ` and ` …` mark what was left out. SBC
+  text stays within ADR 0013's "quote briefly", and the endpoint never returns
+  a passage whole unless it is that short.
+- **Credit and links, all derived by the server.** An SBC shows its plan,
+  plan year, section and document link. A Wikipedia passage carries its
+  CC BY-SA 4.0 credit, read from the source registry's `wikipedia` entry
+  (ADR 0025), and a link to the article. HealthCare.gov text is public domain,
+  and its page URLs are not stored. A link is returned only if it is `https`
+  and passes the ingestion's `unsafe_reason` check (now `src/core/urls.py`),
+  so an IP address or a non-public host is never handed to a browser.
+
+### Renaming leaves `updated_at` alone
+
+`PATCH /api/chat/threads/<id>` (30 per minute) sets a title of 1 to 200
+characters, after trimming. `updated_at` means the thread's last activity and
+orders the list, so a rename must not move a thread to the top. The update
+sets `updated_at` to itself, which stops the column's `onupdate` from firing.
+
+## Consequences
+
+- **The JSON route stays.** Scripts and tests use it, and it is the fallback.
+  The frontend moves to the stream.
+- **Gunicorn's threads bound concurrent streams.** One worker with 8 threads
+  (ADR 0022) serves at most 8 answers at once, streamed or not. More needs
+  more threads, or a shared limiter backend and more workers. With threads,
+  gunicorn runs the gthread worker, whose `--timeout` is a worker heartbeat,
+  not a per-request limit, so a long stream is not killed at 120 s.
+- **Azure's ingress timeout (about 240 s) is above the usual worst case.**
+  That is a rewrite and three answer rounds of up to 30 s each. With every
+  call also retried once (`llm_max_retries = 1`), the theoretical bound comes
+  close to it. A stream cut there ends like a client that went away: the spend
+  is recorded, no answer is saved, and the client shows an error.
+- **A stopped or cut stream saves no answer,** only the question, as a failed
+  answer already did. The model's usage arrives in the stream's last chunk. A
+  round cut short never gets it, so it is counted by estimate, set high: the
+  whole prompt plus the output cap. Otherwise ending streams early would spend
+  past the daily budget (OWASP LLM10).
+- **Citations saved before this ADR are `unverified`,** never shown as if
+  checked.
+- **The Wikipedia link is built from the stored label,** whose title was
+  sanitised at ingest (`Workers'_compensation` became
+  `Workers_compensation`). It relies on Wikipedia's redirects for such titles.