2.4.2: guard benchmarks against out-of-memory; surface provider errors

CHANGELOG.md

@@ -2,6 +2,16 @@
 
 ## Unreleased
 
+## 2.4.2
+
+**Clear failures instead of cryptic ones.** A benchmark that cannot fit in memory is now refused with an explanation, keep-warm no longer fights an in-progress benchmark, and errors the local server reports are passed through instead of being replaced by a generic message.
+
+### Fixed
+
+- **Benchmarks that can't fit are refused up front.** Running `flm bench` on a model too large for the machine failed with a raw driver code — `Failed to submit command to hw queue (0xc01e0200): … the video memory manager could not page-in all of the required allocations` — which said nothing about the cause. The Benchmark tab now checks the model's measured footprint plus room for the 1k–32k context sweep against available memory first, and explains: *"needs about 28.3 GB (~24.3 GB of weights plus room for the 32k-context sweep) but only ~25.6 GB is usable"*. A model with no published footprint is never blocked.
+- **Keep-warm no longer collides with a benchmark.** The background warmup reloaded the active model on its own schedule with no knowledge of benchmarks. Since a benchmark takes 10–20 minutes and the default keepalive is 15, a warmup reliably landed mid-run and competed for the same NPU memory. Warmups are now skipped while a benchmark is running (and logged as such).
+- **Errors from the local server are shown as-is.** FastFlowLM reports some failures as an HTTP 200 response carrying an error message — for example `Failed to load <model> model!` when a model's weights don't fit. Both the hotkey path and chat only looked for a completion, so that message was discarded and replaced with "Local LLM returned no usable text". The real message now reaches you. A warning alongside a valid completion is ignored as before.
+
 ## 2.4.1
 
 **Model picker fixes.** The Models card is one card again, the picker follows the app's styling, no model is hidden from it, and an active model invalidated by a FastFlowLM upgrade is now called out instead of failing silently.

README.md

@@ -4,10 +4,11 @@ Flowkey is a Windows desktop assistant that adds local-LLM hotkeys for grammar f
 
 Everything runs locally through [FastFlowLM](https://fastflowlm.com) (AMD Ryzen AI NPU) or, on machines without the NPU, through [Ollama](https://ollama.com) (CPU/GPU) as a secondary provider. No cloud service, analytics, or telemetry is used by the app.
 
-Current version: `2.4.1`
+Current version: `2.4.2`
 
 ## What's new in 2.4
 
+- **Benchmarks fail clearly, not cryptically.** Starting a benchmark on a model too large for your machine used to end in a raw graphics-driver error code. Flowkey now checks the model's memory footprint (plus room for the context sweep) first and tells you what it needs versus what's available. Background model warmups also pause while a benchmark runs, instead of competing with it for the same memory, and errors reported by the local server are passed through verbatim rather than replaced with a generic message.
 - **The model picker shows every model, sized honestly.** Config → **Models** is now one card, styled like the rest of the dashboard (the old picker used a native browser dropdown that ignored the app's theme). Nothing is hidden from it: models too big for your machine are marked rather than silently dropped, and Mixture-of-Experts models are sized by the parameters actually active per token plus FastFlowLM's own measured memory footprint — so a 35B-total / 3B-active model is no longer mistaken for a 35B one. If a FastFlowLM upgrade invalidates your active model, the card now says so instead of letting the next hotkey fail with an opaque error.
 - **Chat streams as it thinks.** The dashboard **Chat** tab now fills the reply in token-by-token over a live stream instead of waiting on a blank "Thinking…" until the whole answer is ready — the first words typically appear in ~1.6 s (warm) rather than after the full response. It works on both FastFlowLM and Ollama, and if anything interrupts the stream the partial reply is kept and the error is shown. Grammar, `prompt:`, and the other hotkeys are unchanged (they still paste the finished result).
 

SPEC.md

@@ -91,6 +91,9 @@ Caveman-encoded (compression, not amputation). Paths / ids / action names / numb
 - V37: `chat_send_stream` streams reply deltas as `text/event-stream` (SSE `data:`/`event:` frames), persists the full-or-partial turn exactly once under the daemon write-lock; CSRF `X-FFP-API` + Host gates still apply; dashboard falls back to `chat_send` iff the stream ⊥ opens (never after tokens arrive); `send()`/AHK/grammar/prompt paths unchanged (⊥ streamed)
 - V38: `recommend_models` returns EVERY candidate tagged yes/tight/no/unknown + `fit_reason`; UI ⊥ hide any (oversized → marked + confirm-before-pull). Fit signal precedence: provider `footprint_gb` → effective params (MoE active `a<N>b` < total, else total) vs budget
 - V39: configured active model ⊥ installed → `model_recommendations.active_model.status="not_installed"` + dashboard warning naming model/remedy; listing error/exception → `"unknown"` (⊥ claim broken)
+- V41: `flm bench` start → refuse iff catalog `footprint_gb` + 32k-context headroom > usable mem, w/ plain-language reason; unknown footprint ∨ lookup failure → allow (⊥ block on a guess)
+- V42: benchmark running → keep-warm tick ⊥ warms (`skipped_benchmark_running`); ∵ bench 10-20min > 15min keepalive ∴ collision is the norm
+- V43: provider 200-response carrying `error` ∧ ⊥ `choices` → raise that message verbatim (⊥ generic "no usable text"); `error` + `choices` → ignore, return completion
 - V40: model picker + installed list = app-styled elements (⊥ native `<datalist>`/`<select size>`, ∵ browser chrome ignores page CSS); theme-aware + keyboard-navigable (↑↓/Enter/Esc)
 
 ## §T tasks
@@ -124,6 +127,7 @@ T24|x|prompt-v2 default + v1 rollback selector + 240/320/420 caps|V24,V27,V28,V3
 T25|x|FastFlowLM startup+idle keep-warm + cold/warm measurement support|V31
 T26|x|2.3.0 release evidence + version/docs rebaseline after A/B gate passes|V18,V29,V33,V34,V35,V36
 T27|x|streaming Chat tab (SSE): `ffp_chat.stream_send` + daemon `_STREAM_ACTIONS`/`_sse_frame`/`_stream_action` + `app.js` fetch-reader w/ `chat_send` fallback; live FLM first-token 1.58s|V37
+T29|x|bench memory guard + keep-warm/bench mutual exclusion + provider-error surfacing (2.4.2)|V41,V42,V43
 T28|x|model picker 2.4.1: footprint/MoE sizing + never-hide + active-model health + merged single Models card + app-styled combobox|V38,V39,V40
 ```
 
@@ -164,5 +168,8 @@ B30|2026-07-27|FLM 0.9.45 invalidated locally-pulled `qwen3.5:4b` (stamped 0.9.4
 B31|2026-07-27|model picker ⊥ followed app styling ∵ native `<datalist>` popup drawn by browser chrome (page CSS inert) + native `<select size>`|V40; custom app-styled combobox + list
 B32|2026-07-27|`app.js` set `models-title` = "Installed models — <provider>" ∴ overwrote merged card heading at runtime|V40 caught in browser; title → "Models — <provider>"
 B33|2026-07-27|new CSS block referenced undefined `--muted`/`--card`/`--bad` (pre-existing pattern elsewhere in stylesheet: 6+2+3 refs, never defined ∴ inert)|use defined tokens `--text-muted`/`--surface`/`--warn`; pre-existing refs left for separate cleanup
+B35|2026-07-27|`flm bench qwen3.6-moe:35b-a3b` died 4s in w/ driver `0xc01e0200` (page-in failed): 24.3GB weights + 32k-sweep KV > 25.6GB usable; ⊥ preflight ∴ user saw only a hex code|V41; catalog-footprint preflight w/ explanation
+B36|2026-07-27|keep-warm thread ⊥ aware of benchmarks: warms/reloads active model on 15min tick during a 10-20min bench ∴ NPU+mem contention mid-run|V42; `ffp_benchmark.is_running()` gate in `_warm_model_once`
+B37|2026-07-27|FLM returns HTTP **200** + `{"error":"Failed to load <model> model!"}`; `_call_openai_compatible`/`ffp_chat` read only `choices` ∴ real cause discarded → "Local LLM returned no usable text"|V43; surface the error body
 B34|2026-07-27|self-caught: `_active_model_health` tested membership vs `_provider_list("all")`; ollama "all" = installed+suggested (`ffp_provider_runtime:80`) ∴ never-pulled model → false `installed=True` (⊥ warn)|V39; trust `details` (unfiltered, authoritative) else re-list w/ `installed` filter
 ```

installer/installer.iss

@@ -41,7 +41,7 @@
 #define AppURL        "https://github.com/agr77one/Fastflow"
 #define AppExeName    "Flowkey.exe"  ; symbolic — actual launchers below
 ; Keep in lockstep with scripts\_version.py.
-#define AppVersion    "2.4.1"
+#define AppVersion    "2.4.2"
 
 [Setup]
 AppId={{8A4F1E6C-9B3D-4E62-9F7A-FASTFLOW140}}

installer/sign.ps1

@@ -52,7 +52,7 @@
 .EXAMPLE
     # Sign the installer
     $env:FFP_SIGN_PFX_PASSWORD = "ChangeMe!"
-    .\sign.ps1 -FilePath ..\out\Flowkey-Setup-2.4.1.exe
+    .\sign.ps1 -FilePath ..\out\Flowkey-Setup-2.4.2.exe
 #>
 
 [CmdletBinding(DefaultParameterSetName = "Sign")]

pyproject.toml

@@ -17,7 +17,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "fastflowprompt"
-version = "2.4.1"
+version = "2.4.2"
 description = "Local-LLM-powered grammar fix, prompt rewrite, chat, and dashboard for Windows."
 readme = "README.md"
 requires-python = ">=3.11"

scripts/_version.py

@@ -1,3 +1,3 @@
 """Single source of truth for the app version. Read by grammar_fix.py."""
 
-__version__ = "2.4.1"
+__version__ = "2.4.2"

scripts/ffp_benchmark.py

@@ -51,6 +51,17 @@ def status() -> dict:
         return dict(_job)
 
 
+def is_running() -> bool:
+    """True while a benchmark occupies the accelerator.
+
+    The daemon's keep-warm thread checks this before warming: a warmup restarts
+    the serve server and reloads the active model, which on a shared-memory NPU
+    competes with an in-flight `flm bench` (a run takes 10-20 min, longer than
+    the default 15-minute keepalive interval, so a tick WILL land mid-run)."""
+    with _lock:
+        return _job["state"] == "running"
+
+
 def _slug(text: str) -> str:
     return "".join(c if c.isalnum() else "-" for c in str(text)).strip("-") or "model"
 

scripts/ffp_chat.py

@@ -238,6 +238,12 @@ def _default_llm_call(messages: list[dict]) -> str:
 
     choices = payload.get("choices") or []
     if not choices:
+        # FastFlowLM reports load failures as HTTP 200 + {"error": ...} (e.g.
+        # "Failed to load <model> model!"); prefer that over a generic message.
+        provider_error = payload.get("error")
+        if provider_error:
+            detail = provider_error.get("message") if isinstance(provider_error, dict) else provider_error
+            raise RuntimeError(str(detail).strip() or "LLM returned an error.")
         raise RuntimeError("LLM returned no choices.")
     return str((choices[0].get("message") or {}).get("content") or "").strip()
 

scripts/ffp_daemon.py

@@ -524,6 +524,29 @@ def _act_flm_update_check(args: dict) -> dict:
     )
 
 
+def _bench_memory_precheck(model: str, listing: dict | None = None) -> dict:
+    """Refuse a FastFlowLM benchmark that cannot fit in memory.
+
+    `flm bench` loads the weights and then sweeps context to 32k. When the total
+    exceeds what the machine can page in, the driver fails with an opaque
+    `0xc01e0200` ("could not page-in all of the required allocations") that tells
+    the user nothing. Checking the catalog's measured footprint up front turns
+    that into a plain explanation. Unknown footprint -> allow (never block on a
+    guess); any lookup failure -> allow, since refusing a runnable benchmark is
+    worse than letting the driver decide."""
+    import ffp_hardware
+    try:
+        if listing is None:
+            listing = grammar_fix._provider_list("all")
+        details = listing.get("details") or []
+        entry = next((d for d in details if isinstance(d, dict) and d.get("name") == model), None)
+        footprint = entry.get("footprint_gb") if entry else None
+    except Exception as exc:
+        log.warning("bench memory precheck skipped (%s)", exc)
+        return {"ok": True, "error": ""}
+    return ffp_hardware.benchmark_fit(model, footprint)
+
+
 def _act_bench_start(args: dict) -> dict:
     """Kick off a benchmark on a background thread. FastFlowLM: `flm bench`
     (10-20 min; serve stopped for the run and restarted after). Ollama: timed
@@ -541,6 +564,10 @@ def _act_bench_start(args: dict) -> dict:
             provider="ollama",
             base_url=grammar_fix.LLM_BASE_URL,
         )
+    fit = _bench_memory_precheck(model)
+    if not fit["ok"]:
+        log.warning("bench_start refused for %s: %s", model, fit["error"])
+        return {"ok": False, "error": fit["error"], "model": model}
     return ffp_benchmark.start_benchmark(
         model,
         _NO_WINDOW,
@@ -1203,16 +1230,35 @@ def _keep_warm_settings(cfg: dict) -> tuple[bool, int]:
     return warm_on_start, minutes * 60
 
 
+def _benchmark_in_progress() -> bool:
+    """True while a benchmark holds the accelerator (import kept lazy + safe)."""
+    try:
+        import ffp_benchmark
+        return ffp_benchmark.is_running()
+    except Exception:
+        return False
+
+
 def _warm_model_once(
     cfg: dict,
     reason: str,
     *,
     warm_fn: Callable[[], str] | None = None,
+    benchmark_running_fn: Callable[[], bool] | None = None,
 ) -> str:
     llm_cfg = cfg.get("llm") if isinstance(cfg.get("llm"), dict) else {}
     provider = str(llm_cfg.get("provider") or "fastflowlm").strip().lower()
     if provider != "fastflowlm":
         return "skipped_provider"
+    # A benchmark deliberately stops the serve server and drives the NPU on its
+    # own; warming would restart the server and reload the active model
+    # alongside it. On shared NPU/system memory that either wedges the run or
+    # blows past the memory ceiling. A bench takes 10-20 min vs the 15-minute
+    # default keepalive, so this collision is the norm, not an edge case.
+    running = benchmark_running_fn or _benchmark_in_progress
+    if running():
+        log.info("model warmup reason=%s result=skipped_benchmark_running", reason)
+        return "skipped_benchmark_running"
     warm = warm_fn or grammar_fix.warm_configured_fastflowlm
     try:
         result = warm()

scripts/ffp_hardware.py

@@ -254,6 +254,81 @@ def usable_memory_gb(hw: dict | None = None) -> float:
     return max(2.0, round(mem - _OS_HEADROOM_GB, 1))
 
 
+# `flm bench` sweeps 1k-32k context x 8 iterations, so it needs room for a large
+# KV cache ON TOP of the weights. Reserve for the 32k worst case; without this a
+# model that merely *loads* (weights < usable) still dies once the sweep grows
+# the cache, surfacing as an opaque driver page-in failure.
+_BENCH_CONTEXT_HEADROOM_GB = 4.0
+
+
+def available_memory_gb() -> float:
+    """Physical memory currently free, in GB (0.0 when it can't be read)."""
+    if os.name != "nt":
+        return 0.0
+    import ctypes
+
+    class MEMORYSTATUSEX(ctypes.Structure):
+        _fields_ = [
+            ("dwLength", ctypes.c_ulong),
+            ("dwMemoryLoad", ctypes.c_ulong),
+            ("ullTotalPhys", ctypes.c_ulonglong),
+            ("ullAvailPhys", ctypes.c_ulonglong),
+            ("ullTotalPageFile", ctypes.c_ulonglong),
+            ("ullAvailPageFile", ctypes.c_ulonglong),
+            ("ullTotalVirtual", ctypes.c_ulonglong),
+            ("ullAvailVirtual", ctypes.c_ulonglong),
+            ("ullAvailExtendedVirtual", ctypes.c_ulonglong),
+        ]
+
+    stat = MEMORYSTATUSEX()
+    stat.dwLength = ctypes.sizeof(stat)
+    if ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(stat)):
+        return round(stat.ullAvailPhys / 2**30, 1)
+    return 0.0
+
+
+def benchmark_fit(
+    model: str,
+    footprint_gb: float | None,
+    hw: dict | None = None,
+    *,
+    available_gb: float | None = None,
+) -> dict:
+    """Can this model be benchmarked on this machine? -> {ok, error, needed_gb, usable_gb}.
+
+    Benchmarking is stricter than merely running a model: the context sweep adds
+    a large KV cache to the resident weights. Compared against
+    :func:`usable_memory_gb` (stable, matches what the dashboard shows) rather
+    than instantaneous free memory, because the benchmark stops the serve server
+    first and thereby reclaims whatever the active model was holding.
+
+    An unknown footprint never blocks the run — we only refuse on evidence."""
+    hw = hw or detect_hardware()
+    usable = usable_memory_gb(hw)
+    try:
+        footprint = float(footprint_gb) if footprint_gb is not None else None
+    except (TypeError, ValueError):
+        footprint = None
+    if not footprint or footprint <= 0:
+        return {"ok": True, "error": "", "needed_gb": None, "usable_gb": usable}
+    needed = round(footprint + _BENCH_CONTEXT_HEADROOM_GB, 1)
+    if needed <= usable:
+        return {"ok": True, "error": "", "needed_gb": needed, "usable_gb": usable}
+    free = available_memory_gb() if available_gb is None else float(available_gb)
+    detail = f"{free:g} GB free right now" if free else "free memory unknown"
+    return {
+        "ok": False,
+        "needed_gb": needed,
+        "usable_gb": usable,
+        "error": (
+            f"'{model}' is too large to benchmark on this machine: it needs about "
+            f"{needed:g} GB (~{footprint:g} GB of weights plus room for the 32k-context "
+            f"sweep) but only ~{usable:g} GB is usable ({detail}). Benchmark a smaller "
+            f"model, or run this one on a machine with more memory."
+        ),
+    }
+
+
 def _normalize_candidate(candidate: object) -> dict:
     """Accept either a legacy ``(name, params_b)`` tuple or a metadata dict."""
     if isinstance(candidate, dict):

scripts/grammar_fix.py

@@ -564,6 +564,14 @@ def _call_openai_compatible(
     )
     with urllib.request.urlopen(req, timeout=max(2, timeout_seconds)) as resp:
         payload = json.loads(resp.read().decode("utf-8"))
+    # FastFlowLM reports load failures as HTTP 200 with an {"error": ...} body
+    # (e.g. "Failed to load <model> model!" when the weights don't fit in
+    # memory). Surfacing it beats the generic "returned no usable text" that
+    # discards the one message explaining what actually went wrong.
+    provider_error = payload.get("error")
+    if provider_error and not payload.get("choices"):
+        detail = provider_error.get("message") if isinstance(provider_error, dict) else provider_error
+        raise RuntimeError(str(detail).strip() or "local LLM returned an error")
     choices = payload.get("choices") or []
     content = ""
     if choices: