2.4.2: guard benchmarks against out-of-memory; surface provider errors - #37
Conversation
Fixes the reported `flm bench` failure and the class of opaque errors around it.
- Memory preflight. `flm bench qwen3.6-moe:35b-a3b` died 4s in with driver code
0xc01e0200 ("could not page-in all of the required allocations"): 24.3 GB of
weights plus KV cache for the 1k-32k sweep against ~25.6 GB usable. bench_start
now checks the catalog's measured footprint first and refuses with a plain
explanation naming the numbers. Unknown footprint or a lookup failure always
allows the run - never block on a guess.
- Keep-warm defers to a running benchmark. The warm thread reloaded the active
model on its 15-minute tick with no knowledge of benchmarks, while a bench takes
10-20 minutes - so the collision was the norm, not an edge case. _warm_model_once
now returns skipped_benchmark_running while ffp_benchmark.is_running().
- Provider errors are no longer swallowed. FastFlowLM answers HTTP *200* with
{"error": "Failed to load <model> model!"} when weights don't fit; both the
hotkey path and chat read only `choices`, so the real cause was discarded and
users saw "Local LLM returned no usable text". The message is now raised
verbatim. An `error` alongside a valid completion is ignored.
Verified live on the exact failure: bench_start refused the MoE with the readable
message and left the job idle; llama3.2:3b still passes; keep-warm returned
skipped_benchmark_running while running and warmed_up while idle; the broken
active model now reports "Failed to load qwen3.6-moe:35b-a3b model!".
Hotkey regression check on qwen3.5:4b (warm): grammar 2.73s correct; prompt mode
3.77s with 4/4 grounded sections - matching the 2.3.0 baseline.
ruff clean, 462 pytest (+28). SPEC V41/V42/V43, T29, B35-B37.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bump _version.py, pyproject.toml, installer.iss, README (+ What's new in 2.4 bullet), sign.ps1 example, and the version-sync test pin. CHANGELOG 2.4.2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: de5e812331
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # FastFlowLM reports load failures as HTTP 200 + {"error": ...} (e.g. | ||
| # "Failed to load <model> model!"); prefer that over a generic message. | ||
| provider_error = payload.get("error") | ||
| if provider_error: | ||
| detail = provider_error.get("message") if isinstance(provider_error, dict) else provider_error | ||
| raise RuntimeError(str(detail).strip() or "LLM returned an error.") |
There was a problem hiding this comment.
When FastFlowLM returns the HTTP-200 {"error": ...} response for a request with stream: true, the normal dashboard path never reaches this new branch: sendChat() uses chat_send_stream, while _default_llm_stream() feeds the response to _parse_sse_delta(), which ignores both non-data: JSON bodies and SSE objects without choices. stream_send() consequently emits a successful terminal event with no text, so the user still sees (no reply) instead of the provider's load failure. Apply the same error extraction to the streaming response path.
Useful? React with 👍 / 👎.
| hw = hw or detect_hardware() | ||
| usable = usable_memory_gb(hw) |
There was a problem hiding this comment.
On a Ryzen AI machine that also has an NVIDIA GPU, detect_hardware() selects the NVIDIA adapter's VRAM, and usable_memory_gb() adds that value to physical RAM. FastFlowLM's NPU cannot use discrete NVIDIA VRAM for its weights or KV cache, so this comparison can overstate usable benchmark memory by the entire dGPU allocation and allow a model that will still hit the page-in failure the guard is intended to prevent. Benchmark capacity should use system RAM plus only an identified UMA carve-out, rather than the generic vram_gb value.
Useful? React with 👍 / 👎.
2 participants
2.4.2 — Clear failures instead of cryptic ones
Fixes a reported benchmark failure and the class of opaque errors around it.
The reported failure
Root cause:
flm bench qwen3.6-moe:35b-a3bneeds ~24.3 GB of weights plus KV cache for the 1k–32k context sweep, against ~25.6 GB usable on a 32 GB machine. It died 4 s in, at model load. Nothing checked first, so the user saw only a driver code.Fixed
bench_startnow compares the catalog's measuredfootprint_gbplus a 32k-context headroom reserve against usable memory, and refuses with an explanation: "needs about 28.3 GB (~24.3 GB of weights plus room for the 32k-context sweep) but only ~25.6 GB is usable (15.3 GB free right now)". An unknown footprint or any lookup failure always allows the run — never block on a guess. Ollama benches are unaffected (they talk to a running server)._warm_model_oncenow returnsskipped_benchmark_runningwhileffp_benchmark.is_running().{"error": "Failed to load <model> model!"}when weights don't fit. Both the hotkey path and chat read onlychoices, so the real cause was discarded and users got "Local LLM returned no usable text". The message is now raised verbatim; anerroralongside a valid completion is still ignored.Verification
Reproduced and confirmed against the live failure:
bench_starton the 35B MoEidle, no subprocessbench_startonllama3.2:3bskipped_benchmark_runningwarmed_upFailed to load qwen3.6-moe:35b-a3b model!Hotkey regression check on
qwen3.5:4b, warm, real user config: grammar 3.13 s (correct output), prompt 4.35 s with 4/4 grounded sections and no fallback — consistent with the 2.3.0 baseline (3.38 s).ruff clean · 462 pytest (+28 new) · SPEC V41/V42/V43, T29, B35–B37.
🤖 Generated with Claude Code