diff --git a/CHANGELOG.md b/CHANGELOG.md index 88a966b..48fb52a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,16 @@ ## Unreleased +## 2.4.2 + +**Clear failures instead of cryptic ones.** A benchmark that cannot fit in memory is now refused with an explanation, keep-warm no longer fights an in-progress benchmark, and errors the local server reports are passed through instead of being replaced by a generic message. + +### Fixed + +- **Benchmarks that can't fit are refused up front.** Running `flm bench` on a model too large for the machine failed with a raw driver code — `Failed to submit command to hw queue (0xc01e0200): … the video memory manager could not page-in all of the required allocations` — which said nothing about the cause. The Benchmark tab now checks the model's measured footprint plus room for the 1k–32k context sweep against available memory first, and explains: *"needs about 28.3 GB (~24.3 GB of weights plus room for the 32k-context sweep) but only ~25.6 GB is usable"*. A model with no published footprint is never blocked. +- **Keep-warm no longer collides with a benchmark.** The background warmup reloaded the active model on its own schedule with no knowledge of benchmarks. Since a benchmark takes 10–20 minutes and the default keepalive is 15, a warmup reliably landed mid-run and competed for the same NPU memory. Warmups are now skipped while a benchmark is running (and logged as such). +- **Errors from the local server are shown as-is.** FastFlowLM reports some failures as an HTTP 200 response carrying an error message — for example `Failed to load model!` when a model's weights don't fit. Both the hotkey path and chat only looked for a completion, so that message was discarded and replaced with "Local LLM returned no usable text". The real message now reaches you. A warning alongside a valid completion is ignored as before. + ## 2.4.1 **Model picker fixes.** The Models card is one card again, the picker follows the app's styling, no model is hidden from it, and an active model invalidated by a FastFlowLM upgrade is now called out instead of failing silently. diff --git a/README.md b/README.md index 6968086..572ac32 100644 --- a/README.md +++ b/README.md @@ -4,10 +4,11 @@ Flowkey is a Windows desktop assistant that adds local-LLM hotkeys for grammar f Everything runs locally through [FastFlowLM](https://fastflowlm.com) (AMD Ryzen AI NPU) or, on machines without the NPU, through [Ollama](https://ollama.com) (CPU/GPU) as a secondary provider. No cloud service, analytics, or telemetry is used by the app. -Current version: `2.4.1` +Current version: `2.4.2` ## What's new in 2.4 +- **Benchmarks fail clearly, not cryptically.** Starting a benchmark on a model too large for your machine used to end in a raw graphics-driver error code. Flowkey now checks the model's memory footprint (plus room for the context sweep) first and tells you what it needs versus what's available. Background model warmups also pause while a benchmark runs, instead of competing with it for the same memory, and errors reported by the local server are passed through verbatim rather than replaced with a generic message. - **The model picker shows every model, sized honestly.** Config → **Models** is now one card, styled like the rest of the dashboard (the old picker used a native browser dropdown that ignored the app's theme). Nothing is hidden from it: models too big for your machine are marked rather than silently dropped, and Mixture-of-Experts models are sized by the parameters actually active per token plus FastFlowLM's own measured memory footprint — so a 35B-total / 3B-active model is no longer mistaken for a 35B one. If a FastFlowLM upgrade invalidates your active model, the card now says so instead of letting the next hotkey fail with an opaque error. - **Chat streams as it thinks.** The dashboard **Chat** tab now fills the reply in token-by-token over a live stream instead of waiting on a blank "Thinking…" until the whole answer is ready — the first words typically appear in ~1.6 s (warm) rather than after the full response. It works on both FastFlowLM and Ollama, and if anything interrupts the stream the partial reply is kept and the error is shown. Grammar, `prompt:`, and the other hotkeys are unchanged (they still paste the finished result). diff --git a/SPEC.md b/SPEC.md index 9a744fe..e99f664 100644 --- a/SPEC.md +++ b/SPEC.md @@ -91,6 +91,9 @@ Caveman-encoded (compression, not amputation). Paths / ids / action names / numb - V37: `chat_send_stream` streams reply deltas as `text/event-stream` (SSE `data:`/`event:` frames), persists the full-or-partial turn exactly once under the daemon write-lock; CSRF `X-FFP-API` + Host gates still apply; dashboard falls back to `chat_send` iff the stream ⊥ opens (never after tokens arrive); `send()`/AHK/grammar/prompt paths unchanged (⊥ streamed) - V38: `recommend_models` returns EVERY candidate tagged yes/tight/no/unknown + `fit_reason`; UI ⊥ hide any (oversized → marked + confirm-before-pull). Fit signal precedence: provider `footprint_gb` → effective params (MoE active `ab` < total, else total) vs budget - V39: configured active model ⊥ installed → `model_recommendations.active_model.status="not_installed"` + dashboard warning naming model/remedy; listing error/exception → `"unknown"` (⊥ claim broken) +- V41: `flm bench` start → refuse iff catalog `footprint_gb` + 32k-context headroom > usable mem, w/ plain-language reason; unknown footprint ∨ lookup failure → allow (⊥ block on a guess) +- V42: benchmark running → keep-warm tick ⊥ warms (`skipped_benchmark_running`); ∵ bench 10-20min > 15min keepalive ∴ collision is the norm +- V43: provider 200-response carrying `error` ∧ ⊥ `choices` → raise that message verbatim (⊥ generic "no usable text"); `error` + `choices` → ignore, return completion - V40: model picker + installed list = app-styled elements (⊥ native ``/``|V40; custom app-styled combobox + list B32|2026-07-27|`app.js` set `models-title` = "Installed models — " ∴ overwrote merged card heading at runtime|V40 caught in browser; title → "Models — " B33|2026-07-27|new CSS block referenced undefined `--muted`/`--card`/`--bad` (pre-existing pattern elsewhere in stylesheet: 6+2+3 refs, never defined ∴ inert)|use defined tokens `--text-muted`/`--surface`/`--warn`; pre-existing refs left for separate cleanup +B35|2026-07-27|`flm bench qwen3.6-moe:35b-a3b` died 4s in w/ driver `0xc01e0200` (page-in failed): 24.3GB weights + 32k-sweep KV > 25.6GB usable; ⊥ preflight ∴ user saw only a hex code|V41; catalog-footprint preflight w/ explanation +B36|2026-07-27|keep-warm thread ⊥ aware of benchmarks: warms/reloads active model on 15min tick during a 10-20min bench ∴ NPU+mem contention mid-run|V42; `ffp_benchmark.is_running()` gate in `_warm_model_once` +B37|2026-07-27|FLM returns HTTP **200** + `{"error":"Failed to load model!"}`; `_call_openai_compatible`/`ffp_chat` read only `choices` ∴ real cause discarded → "Local LLM returned no usable text"|V43; surface the error body B34|2026-07-27|self-caught: `_active_model_health` tested membership vs `_provider_list("all")`; ollama "all" = installed+suggested (`ffp_provider_runtime:80`) ∴ never-pulled model → false `installed=True` (⊥ warn)|V39; trust `details` (unfiltered, authoritative) else re-list w/ `installed` filter ``` diff --git a/installer/installer.iss b/installer/installer.iss index 6c67e32..c0486e2 100644 --- a/installer/installer.iss +++ b/installer/installer.iss @@ -41,7 +41,7 @@ #define AppURL "https://github.com/agr77one/Fastflow" #define AppExeName "Flowkey.exe" ; symbolic — actual launchers below ; Keep in lockstep with scripts\_version.py. -#define AppVersion "2.4.1" +#define AppVersion "2.4.2" [Setup] AppId={{8A4F1E6C-9B3D-4E62-9F7A-FASTFLOW140}} diff --git a/installer/sign.ps1 b/installer/sign.ps1 index f2be445..3427dc0 100644 --- a/installer/sign.ps1 +++ b/installer/sign.ps1 @@ -52,7 +52,7 @@ .EXAMPLE # Sign the installer $env:FFP_SIGN_PFX_PASSWORD = "ChangeMe!" - .\sign.ps1 -FilePath ..\out\Flowkey-Setup-2.4.1.exe + .\sign.ps1 -FilePath ..\out\Flowkey-Setup-2.4.2.exe #> [CmdletBinding(DefaultParameterSetName = "Sign")] diff --git a/pyproject.toml b/pyproject.toml index a1704a4..a646838 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -17,7 +17,7 @@ build-backend = "setuptools.build_meta" [project] name = "fastflowprompt" -version = "2.4.1" +version = "2.4.2" description = "Local-LLM-powered grammar fix, prompt rewrite, chat, and dashboard for Windows." readme = "README.md" requires-python = ">=3.11" diff --git a/scripts/_version.py b/scripts/_version.py index 5d04335..8e3e2ae 100644 --- a/scripts/_version.py +++ b/scripts/_version.py @@ -1,3 +1,3 @@ """Single source of truth for the app version. Read by grammar_fix.py.""" -__version__ = "2.4.1" +__version__ = "2.4.2" diff --git a/scripts/ffp_benchmark.py b/scripts/ffp_benchmark.py index 493293a..858da31 100644 --- a/scripts/ffp_benchmark.py +++ b/scripts/ffp_benchmark.py @@ -51,6 +51,17 @@ def status() -> dict: return dict(_job) +def is_running() -> bool: + """True while a benchmark occupies the accelerator. + + The daemon's keep-warm thread checks this before warming: a warmup restarts + the serve server and reloads the active model, which on a shared-memory NPU + competes with an in-flight `flm bench` (a run takes 10-20 min, longer than + the default 15-minute keepalive interval, so a tick WILL land mid-run).""" + with _lock: + return _job["state"] == "running" + + def _slug(text: str) -> str: return "".join(c if c.isalnum() else "-" for c in str(text)).strip("-") or "model" diff --git a/scripts/ffp_chat.py b/scripts/ffp_chat.py index c0a6843..c15c1d5 100644 --- a/scripts/ffp_chat.py +++ b/scripts/ffp_chat.py @@ -238,6 +238,12 @@ def _default_llm_call(messages: list[dict]) -> str: choices = payload.get("choices") or [] if not choices: + # FastFlowLM reports load failures as HTTP 200 + {"error": ...} (e.g. + # "Failed to load model!"); prefer that over a generic message. + provider_error = payload.get("error") + if provider_error: + detail = provider_error.get("message") if isinstance(provider_error, dict) else provider_error + raise RuntimeError(str(detail).strip() or "LLM returned an error.") raise RuntimeError("LLM returned no choices.") return str((choices[0].get("message") or {}).get("content") or "").strip() diff --git a/scripts/ffp_daemon.py b/scripts/ffp_daemon.py index c4c44d2..b9eaa61 100644 --- a/scripts/ffp_daemon.py +++ b/scripts/ffp_daemon.py @@ -524,6 +524,29 @@ def _act_flm_update_check(args: dict) -> dict: ) +def _bench_memory_precheck(model: str, listing: dict | None = None) -> dict: + """Refuse a FastFlowLM benchmark that cannot fit in memory. + + `flm bench` loads the weights and then sweeps context to 32k. When the total + exceeds what the machine can page in, the driver fails with an opaque + `0xc01e0200` ("could not page-in all of the required allocations") that tells + the user nothing. Checking the catalog's measured footprint up front turns + that into a plain explanation. Unknown footprint -> allow (never block on a + guess); any lookup failure -> allow, since refusing a runnable benchmark is + worse than letting the driver decide.""" + import ffp_hardware + try: + if listing is None: + listing = grammar_fix._provider_list("all") + details = listing.get("details") or [] + entry = next((d for d in details if isinstance(d, dict) and d.get("name") == model), None) + footprint = entry.get("footprint_gb") if entry else None + except Exception as exc: + log.warning("bench memory precheck skipped (%s)", exc) + return {"ok": True, "error": ""} + return ffp_hardware.benchmark_fit(model, footprint) + + def _act_bench_start(args: dict) -> dict: """Kick off a benchmark on a background thread. FastFlowLM: `flm bench` (10-20 min; serve stopped for the run and restarted after). Ollama: timed @@ -541,6 +564,10 @@ def _act_bench_start(args: dict) -> dict: provider="ollama", base_url=grammar_fix.LLM_BASE_URL, ) + fit = _bench_memory_precheck(model) + if not fit["ok"]: + log.warning("bench_start refused for %s: %s", model, fit["error"]) + return {"ok": False, "error": fit["error"], "model": model} return ffp_benchmark.start_benchmark( model, _NO_WINDOW, @@ -1203,16 +1230,35 @@ def _keep_warm_settings(cfg: dict) -> tuple[bool, int]: return warm_on_start, minutes * 60 +def _benchmark_in_progress() -> bool: + """True while a benchmark holds the accelerator (import kept lazy + safe).""" + try: + import ffp_benchmark + return ffp_benchmark.is_running() + except Exception: + return False + + def _warm_model_once( cfg: dict, reason: str, *, warm_fn: Callable[[], str] | None = None, + benchmark_running_fn: Callable[[], bool] | None = None, ) -> str: llm_cfg = cfg.get("llm") if isinstance(cfg.get("llm"), dict) else {} provider = str(llm_cfg.get("provider") or "fastflowlm").strip().lower() if provider != "fastflowlm": return "skipped_provider" + # A benchmark deliberately stops the serve server and drives the NPU on its + # own; warming would restart the server and reload the active model + # alongside it. On shared NPU/system memory that either wedges the run or + # blows past the memory ceiling. A bench takes 10-20 min vs the 15-minute + # default keepalive, so this collision is the norm, not an edge case. + running = benchmark_running_fn or _benchmark_in_progress + if running(): + log.info("model warmup reason=%s result=skipped_benchmark_running", reason) + return "skipped_benchmark_running" warm = warm_fn or grammar_fix.warm_configured_fastflowlm try: result = warm() diff --git a/scripts/ffp_hardware.py b/scripts/ffp_hardware.py index ef67f08..dc1d10c 100644 --- a/scripts/ffp_hardware.py +++ b/scripts/ffp_hardware.py @@ -254,6 +254,81 @@ def usable_memory_gb(hw: dict | None = None) -> float: return max(2.0, round(mem - _OS_HEADROOM_GB, 1)) +# `flm bench` sweeps 1k-32k context x 8 iterations, so it needs room for a large +# KV cache ON TOP of the weights. Reserve for the 32k worst case; without this a +# model that merely *loads* (weights < usable) still dies once the sweep grows +# the cache, surfacing as an opaque driver page-in failure. +_BENCH_CONTEXT_HEADROOM_GB = 4.0 + + +def available_memory_gb() -> float: + """Physical memory currently free, in GB (0.0 when it can't be read).""" + if os.name != "nt": + return 0.0 + import ctypes + + class MEMORYSTATUSEX(ctypes.Structure): + _fields_ = [ + ("dwLength", ctypes.c_ulong), + ("dwMemoryLoad", ctypes.c_ulong), + ("ullTotalPhys", ctypes.c_ulonglong), + ("ullAvailPhys", ctypes.c_ulonglong), + ("ullTotalPageFile", ctypes.c_ulonglong), + ("ullAvailPageFile", ctypes.c_ulonglong), + ("ullTotalVirtual", ctypes.c_ulonglong), + ("ullAvailVirtual", ctypes.c_ulonglong), + ("ullAvailExtendedVirtual", ctypes.c_ulonglong), + ] + + stat = MEMORYSTATUSEX() + stat.dwLength = ctypes.sizeof(stat) + if ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(stat)): + return round(stat.ullAvailPhys / 2**30, 1) + return 0.0 + + +def benchmark_fit( + model: str, + footprint_gb: float | None, + hw: dict | None = None, + *, + available_gb: float | None = None, +) -> dict: + """Can this model be benchmarked on this machine? -> {ok, error, needed_gb, usable_gb}. + + Benchmarking is stricter than merely running a model: the context sweep adds + a large KV cache to the resident weights. Compared against + :func:`usable_memory_gb` (stable, matches what the dashboard shows) rather + than instantaneous free memory, because the benchmark stops the serve server + first and thereby reclaims whatever the active model was holding. + + An unknown footprint never blocks the run — we only refuse on evidence.""" + hw = hw or detect_hardware() + usable = usable_memory_gb(hw) + try: + footprint = float(footprint_gb) if footprint_gb is not None else None + except (TypeError, ValueError): + footprint = None + if not footprint or footprint <= 0: + return {"ok": True, "error": "", "needed_gb": None, "usable_gb": usable} + needed = round(footprint + _BENCH_CONTEXT_HEADROOM_GB, 1) + if needed <= usable: + return {"ok": True, "error": "", "needed_gb": needed, "usable_gb": usable} + free = available_memory_gb() if available_gb is None else float(available_gb) + detail = f"{free:g} GB free right now" if free else "free memory unknown" + return { + "ok": False, + "needed_gb": needed, + "usable_gb": usable, + "error": ( + f"'{model}' is too large to benchmark on this machine: it needs about " + f"{needed:g} GB (~{footprint:g} GB of weights plus room for the 32k-context " + f"sweep) but only ~{usable:g} GB is usable ({detail}). Benchmark a smaller " + f"model, or run this one on a machine with more memory." + ), + } + + def _normalize_candidate(candidate: object) -> dict: """Accept either a legacy ``(name, params_b)`` tuple or a metadata dict.""" if isinstance(candidate, dict): diff --git a/scripts/grammar_fix.py b/scripts/grammar_fix.py index 5ea515d..8c2ef77 100644 --- a/scripts/grammar_fix.py +++ b/scripts/grammar_fix.py @@ -564,6 +564,14 @@ def _call_openai_compatible( ) with urllib.request.urlopen(req, timeout=max(2, timeout_seconds)) as resp: payload = json.loads(resp.read().decode("utf-8")) + # FastFlowLM reports load failures as HTTP 200 with an {"error": ...} body + # (e.g. "Failed to load model!" when the weights don't fit in + # memory). Surfacing it beats the generic "returned no usable text" that + # discards the one message explaining what actually went wrong. + provider_error = payload.get("error") + if provider_error and not payload.get("choices"): + detail = provider_error.get("message") if isinstance(provider_error, dict) else provider_error + raise RuntimeError(str(detail).strip() or "local LLM returned an error") choices = payload.get("choices") or [] content = "" if choices: diff --git a/tests/test_bench_memory_guard.py b/tests/test_bench_memory_guard.py new file mode 100644 index 0000000..abadfe7 --- /dev/null +++ b/tests/test_bench_memory_guard.py @@ -0,0 +1,287 @@ +"""Benchmark memory guard + keep-warm/benchmark mutual exclusion (2.4.2). + +Reproduced from a real failure on 2026-07-27: `flm bench qwen3.6-moe:35b-a3b` +died in 4 seconds with + + Failed to submit command to hw queue (0xc01e0200): ... the video memory + manager could not page-in all of the required allocations ... + +Two causes, both fixed here: + +1. Nothing checked whether the model could fit before launching. The MoE needs + ~24.3 GB of weights plus KV cache for the 1k-32k sweep, against ~25.6 GB + usable on a 32 GB machine — unwinnable, but the user only saw a driver code. +2. The keep-warm thread reloads the active model on its own schedule with no + knowledge of benchmarks. A bench takes 10-20 min and the default keepalive is + 15 min, so a warm tick lands mid-run and competes for the same memory. +""" +from __future__ import annotations + +import ffp_benchmark +import ffp_hardware +import pytest + +HW_32GB = {"ram_gb": 23.6, "vram_gb": 8.0} # usable == 25.6 GB + + +# ---------- benchmark_fit --------------------------------------------------------------- + +def test_rejects_the_model_that_actually_failed(): + fit = ffp_hardware.benchmark_fit("qwen3.6-moe:35b-a3b", 24.3, HW_32GB, available_gb=15.4) + assert fit["ok"] is False + assert fit["needed_gb"] == 28.3 # 24.3 weights + 4.0 context headroom + assert fit["usable_gb"] == 25.6 + # The message must explain, not just refuse. + for fragment in ("too large to benchmark", "24.3", "25.6", "15.4 GB free"): + assert fragment in fit["error"] + + +@pytest.mark.parametrize(("model", "footprint"), [ + ("gpt-oss:20b", 14.0), + ("gemma4-it:e4b", 9.1), + ("qwen3.5:4b", 5.2), + ("llama3.2:3b", 2.7), +]) +def test_allows_models_that_fit(model, footprint): + fit = ffp_hardware.benchmark_fit(model, footprint, HW_32GB) + assert fit["ok"] is True + assert fit["error"] == "" + + +def test_boundary_exactly_at_usable_is_allowed(): + # needed == usable must pass; only strictly-over is refused. + fit = ffp_hardware.benchmark_fit("edge:1b", 21.6, HW_32GB) # 21.6 + 4.0 == 25.6 + assert fit["ok"] is True + over = ffp_hardware.benchmark_fit("edge:1b", 21.7, HW_32GB) # 25.7 > 25.6 + assert over["ok"] is False + + +@pytest.mark.parametrize("footprint", [None, 0, 0.0, "", "abc", -3]) +def test_unknown_or_bogus_footprint_never_blocks(footprint): + # Refusing a runnable benchmark is worse than letting the driver decide. + fit = ffp_hardware.benchmark_fit("mystery:9b", footprint, HW_32GB) + assert fit["ok"] is True + + +def test_bigger_machine_allows_the_moe(): + fit = ffp_hardware.benchmark_fit("qwen3.6-moe:35b-a3b", 24.3, {"ram_gb": 64, "vram_gb": 0}) + assert fit["ok"] is True + + +def test_available_memory_reports_something_sane(): + free = ffp_hardware.available_memory_gb() + assert free >= 0 + assert free <= ffp_hardware.system_memory_gb() + 1 # never more than installed + + +# ---------- benchmark run-state signal --------------------------------------------------- + +def test_is_running_tracks_job_state(monkeypatch): + monkeypatch.setitem(ffp_benchmark._job, "state", "idle") + assert ffp_benchmark.is_running() is False + monkeypatch.setitem(ffp_benchmark._job, "state", "running") + assert ffp_benchmark.is_running() is True + monkeypatch.setitem(ffp_benchmark._job, "state", "done") + assert ffp_benchmark.is_running() is False + monkeypatch.setitem(ffp_benchmark._job, "state", "error") + assert ffp_benchmark.is_running() is False + + +# ---------- keep-warm defers to a running benchmark -------------------------------------- + +def test_warmup_skipped_while_benchmark_runs(fresh_modules): + daemon = fresh_modules("ffp_daemon") + called = [] + result = daemon._warm_model_once( + {"llm": {"provider": "fastflowlm"}}, + "idle_interval", + warm_fn=lambda: called.append("warmed") or "warmed_up", + benchmark_running_fn=lambda: True, + ) + assert result == "skipped_benchmark_running" + assert called == [] # the model was NOT reloaded mid-benchmark + + +def test_warmup_proceeds_when_no_benchmark(fresh_modules): + daemon = fresh_modules("ffp_daemon") + called = [] + result = daemon._warm_model_once( + {"llm": {"provider": "fastflowlm"}}, + "idle_interval", + warm_fn=lambda: called.append("warmed") or "warmed_up", + benchmark_running_fn=lambda: False, + ) + assert result == "warmed_up" + assert called == ["warmed"] + + +def test_benchmark_in_progress_is_false_when_lookup_fails(fresh_modules, monkeypatch): + # A broken/absent benchmark module must not stop keep-warm from working. + daemon = fresh_modules("ffp_daemon") + import ffp_benchmark as b + monkeypatch.setattr(b, "is_running", lambda: (_ for _ in ()).throw(RuntimeError("boom"))) + assert daemon._benchmark_in_progress() is False + + +# ---------- daemon precheck wiring ------------------------------------------------------- + +def test_precheck_refuses_using_catalog_footprint(fresh_modules): + daemon = fresh_modules("ffp_daemon") + listing = {"details": [{"name": "qwen3.6-moe:35b-a3b", "footprint_gb": 24.3, "installed": True}]} + out = daemon._bench_memory_precheck("qwen3.6-moe:35b-a3b", listing) + # Only assert the verdict shape; the exact threshold is covered above and + # depends on the CI machine's real memory. + assert set(out) >= {"ok", "error"} + if not out["ok"]: + assert "too large to benchmark" in out["error"] + + +def test_precheck_allows_model_missing_from_catalog(fresh_modules): + daemon = fresh_modules("ffp_daemon") + out = daemon._bench_memory_precheck("ghost:1b", {"details": [{"name": "other:1b", "footprint_gb": 2.0}]}) + assert out["ok"] is True + + +def test_precheck_allows_when_listing_raises(fresh_modules, monkeypatch): + daemon = fresh_modules("ffp_daemon") + import grammar_fix + + def boom(kind): + raise RuntimeError("flm missing") + + monkeypatch.setattr(grammar_fix, "_provider_list", boom) + out = daemon._bench_memory_precheck("anything:4b") + assert out["ok"] is True # never block on a lookup failure + + +def test_bench_start_refuses_oversized_model_without_launching(fresh_modules, monkeypatch): + daemon = fresh_modules("ffp_daemon") + import grammar_fix + launched = [] + monkeypatch.setattr(grammar_fix, "LLM_PROVIDER", "fastflowlm", raising=False) + monkeypatch.setattr( + daemon, "_bench_memory_precheck", + lambda model, listing=None: {"ok": False, "error": "'x' is too large to benchmark on this machine: ..."}, + ) + import ffp_benchmark as b + monkeypatch.setattr(b, "start_benchmark", lambda *a, **k: launched.append(a) or {"ok": True}) + out = daemon._act_bench_start({"model": "qwen3.6-moe:35b-a3b"}) + assert out["ok"] is False + assert "too large to benchmark" in out["error"] + assert launched == [] # no thread, no flm subprocess, no driver error + + +def test_bench_start_proceeds_when_precheck_passes(fresh_modules, monkeypatch): + daemon = fresh_modules("ffp_daemon") + import grammar_fix + launched = [] + monkeypatch.setattr(grammar_fix, "LLM_PROVIDER", "fastflowlm", raising=False) + monkeypatch.setattr(daemon, "_bench_memory_precheck", lambda model, listing=None: {"ok": True, "error": ""}) + import ffp_benchmark as b + import ffp_flm_server + monkeypatch.setattr(ffp_flm_server, "flm_version", lambda nw: "0.9.45") + monkeypatch.setattr(b, "start_benchmark", lambda *a, **k: launched.append(a[0]) or {"ok": True, "state": "running"}) + out = daemon._act_bench_start({"model": "llama3.2:3b"}) + assert out["ok"] is True + assert launched == ["llama3.2:3b"] + + +# ---------- provider errors returned as HTTP 200 ----------------------------------------- + +def test_openai_call_surfaces_provider_error_body(fresh_modules, monkeypatch): + # FastFlowLM answers 200 with {"error": "Failed to load model!"} when + # the weights don't fit. That message must reach the user instead of the + # generic "Local LLM returned no usable text" that discards the real cause. + import io + import json as _json + grammar_fix = fresh_modules("grammar_fix") + + class FakeResp(io.BytesIO): + def __enter__(self): + return self + + def __exit__(self, *a): + return False + + payload = _json.dumps({"error": "Failed to load qwen3.6-moe:35b-a3b model!"}).encode() + monkeypatch.setattr(grammar_fix.urllib.request, "urlopen", lambda *a, **k: FakeResp(payload)) + with pytest.raises(RuntimeError, match="Failed to load qwen3.6-moe:35b-a3b model!"): + grammar_fix._call_openai_compatible("http://x", "flm", "m", "sys", "user", 32, 5) + + +def test_openai_call_handles_nested_error_object(fresh_modules, monkeypatch): + import io + import json as _json + grammar_fix = fresh_modules("grammar_fix") + + class FakeResp(io.BytesIO): + def __enter__(self): + return self + + def __exit__(self, *a): + return False + + payload = _json.dumps({"error": {"message": "model not found", "type": "invalid"}}).encode() + monkeypatch.setattr(grammar_fix.urllib.request, "urlopen", lambda *a, **k: FakeResp(payload)) + with pytest.raises(RuntimeError, match="model not found"): + grammar_fix._call_openai_compatible("http://x", "flm", "m", "sys", "user", 32, 5) + + +def test_openai_call_ignores_error_field_when_choices_present(fresh_modules, monkeypatch): + # A warning alongside a real completion must not abort the call. + import io + import json as _json + grammar_fix = fresh_modules("grammar_fix") + + class FakeResp(io.BytesIO): + def __enter__(self): + return self + + def __exit__(self, *a): + return False + + payload = _json.dumps({ + "error": "deprecated parameter", + "choices": [{"message": {"content": "real answer"}}], + "model": "m", + }).encode() + monkeypatch.setattr(grammar_fix.urllib.request, "urlopen", lambda *a, **k: FakeResp(payload)) + text, model, _usage = grammar_fix._call_openai_compatible("http://x", "flm", "m", "sys", "user", 32, 5) + assert text == "real answer" + + +def test_chat_surfaces_provider_error_body(monkeypatch): + import ffp_chat + monkeypatch.setattr(ffp_chat, "_default_llm_call", ffp_chat._default_llm_call) + payload = {"error": "Failed to load big:70b model!"} + + class FakeResp: + def __enter__(self): + return self + + def __exit__(self, *a): + return False + + def read(self): + import json as _json + return _json.dumps(payload).encode() + + monkeypatch.setattr(ffp_chat.urllib.request, "urlopen", lambda *a, **k: FakeResp()) + with pytest.raises(RuntimeError, match="Failed to load big:70b model!"): + ffp_chat._default_llm_call([{"role": "user", "content": "hi"}]) + + +def test_ollama_bench_skips_the_flm_memory_precheck(fresh_modules, monkeypatch): + # The guard is about FLM's NPU/weight paging; Ollama benches talk to a + # running server and must not be gated by it. + daemon = fresh_modules("ffp_daemon") + import grammar_fix + called = [] + monkeypatch.setattr(grammar_fix, "LLM_PROVIDER", "ollama", raising=False) + monkeypatch.setattr(grammar_fix, "LLM_BASE_URL", "http://127.0.0.1:11434", raising=False) + monkeypatch.setattr(daemon, "_bench_memory_precheck", lambda *a, **k: called.append(1) or {"ok": False, "error": "no"}) + import ffp_benchmark as b + monkeypatch.setattr(b, "start_benchmark", lambda *a, **k: {"ok": True, "state": "running"}) + out = daemon._act_bench_start({"model": "llama3.2:3b"}) + assert out["ok"] is True + assert called == [] diff --git a/tests/test_version_sync.py b/tests/test_version_sync.py index 2a381e8..50ef3b5 100644 --- a/tests/test_version_sync.py +++ b/tests/test_version_sync.py @@ -32,4 +32,4 @@ def test_v18_release_version_is_synchronized(): ), } - assert set(versions.values()) == {"2.4.1"}, versions + assert set(versions.values()) == {"2.4.2"}, versions