feat: detect and parse q27 (signalnine/q27 engine) LLM backend - #74
Open
PierpaoloPernici wants to merge 7 commits into
Open
feat: detect and parse q27 (signalnine/q27 engine) LLM backend#74PierpaoloPernici wants to merge 7 commits into
PierpaoloPernici wants to merge 7 commits into
Conversation
- _readHostNetFile fallback now reads /proc/net directly via fs.readFileSync instead of re-dispatching through _readHostFile (which routes /proc/net/* back into this method), fixing 'Maximum call stack size exceeded' on collectors without a mounted host proc netns. - Add 'swap' to the PSEUDO fstype set so the [SWAP] pseudo-mount from lsblk is skipped instead of failing statfs with ENOENT on every poll.
q27 (signalnine/q27) exposes Prometheus /metrics under q27_* series with an api= label (chat/completions/messages/responses). Add it as a first- class backend: - detection: owned_by "q27" on /v1/models, plus a _probeIsQ27 fallback that regex-checks /metrics for q27_ series (mirrors the ds4 path); exclude q27 from the /slots skip list and the vllm fallback branches - _applyQ27Metrics: live tok/s from counter deltas (idle -> 0), exact computed-only prefill (ds4 convention; cache-served tokens never inflate the main tile), cached/uncached prefill split + prefix-cache hit rate, KV cache usage, slots, inflight, spec accept ratio, and TTFT/E2E/ITL p95 from histograms (+Inf == _count enforced, same quantile logic as vLLM) - refactor _parseVllmHistogram into a generic _parseHistogram so q27 (or future backends) reuse the cumulative-bucket parsing - frontend: backend union + label "q27" in the panel and overview; the vLLM-gated tile rows (KV/requests/TTFT/preempts/prefix/E2E/ITL/MTP) render for q27 too, with missing series showing "—"; Requests tile now shows "N run" when the backend has no waiting gauge - tests: LlmProbe.q27.test.js (detection, rates, split, histograms, +Inf mismatch refusal, no vllm mislabel) — full LlmProbe suite green
The earlier edit batch for LlmPanel was rejected as a unit (duplicate gate pattern) and only the gate changes survived; re-apply the two dropped edits: the q27 label in the backend badge map and the Requests tile fallback (show 'N run' when the backend has no waiting gauge, e.g. q27).
…v1/models - Engine tile: q27 keeps weights resident and is ready whenever the server is up (no sleep state), so _applyQ27Metrics reports gpuMemoryUtilization=1 (Active) like the SGLang path - Preempts tile: q27 exposes a constant-0 q27_preemptions_total counter (FIFO admission, never preempts) — reads 0 instead of "—" - Context tile: q27 /v1/models now returns max_model_len, so contextLength populates from the probe - tests: fixture + assertions for the three fields
q27 now exposes unlabeled live counters (q27_*_tokens_processed_total) incremented as tokens are produced (decode per delivered token, prefill at prefill end), so a 2s poll sees generation progress instead of a completion-time step. _applyQ27Metrics prefers the live series and falls back to the completion-based totals for older q27 binaries. Tests cover both paths (88/88).
…ver tok/s, docs - _metricsLookLikeQ27 also matches q27_decode_tokens_processed_total so a new-binary exposition without q27_decode_tokens_total still classifies q27 - readServerGenerationTokens (DecodeBench server-side tok/s) reads the q27 live processed counter with the completion-total fallback (same preference as LlmProbe) - reformat the >100-char q27 guard lines to match file style - fix _applyQ27Metrics docstring: preemptions are a constant-0 counter (not a null tile) and clarify the computed-only prefill convention - backend-neutral tooltip wording (no 'from vLLM's history') now that the p95 tiles render for q27 too - tests: processed-only detection + readServerGenerationTokens q27
…released - feature table: q27 in the auto-detect list and cached/uncached prefill backends; 'vLLM health' row renamed to 'LLM inference health' since the tiles now render for q27 too (FIFO, no wait gauge) - LLM probe backend list: q27 bullet (owned_by / q27_* series, processed counters, p95 histograms, constant-0 preemptions) and the missing EXL3 bullet (doc gap since 1.8.2) - CHANGELOG [Unreleased] Added: q27 LLM backend entry
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds q27 (signalnine/q27 engine) as a first-class LLM backend for the LLM probe, instead of mislabeling it as vLLM. Once detected, the LLM panel gets the full live-telemetry treatment: real-time decode/prefill tok/s, inference-health tiles, and backend-aware labels/badges.
Why
q27 exposes an OpenAI-compatible
/v1/modelsplus Prometheus/metrics, so the probe fell through to the default vLLM classification — the badge read "vLLM" and every vLLM-specific tile stayed empty, because thevllm:*series it looks for don't exist on q27. (The q27/metricsendpoint itself landed upstream in signalnine/q27#40; this PR teaches sparkDash to speak it.)How
Detection (
LlmProbe.js)/v1/modelswithowned_by: "q27", or/metricsexposingq27_decode_tokens_total,q27_decode_tokens_processed_total, orq27_requests_total(regex anchored soq27_requests_errors_totaland friends don't false-positive)/metricsno longer resets the backend to vLLM (same guard as ds4/exl3)Metrics (
_applyQ27Metrics)api=label sets (chat / completions / messages / responses), exactly like the ds4/vLLM pathsq27_*_processedcounter deltas (they move during generation); falls back to the completion-based per-api totals for older binaries (step function: 0 while generating, jump at completion)q27_*_secondshistograms (with the+Inf == _countinvariant check — mismatch refuses the quantile), and MTP/speculative accept ratio0instead of—N runwithout thewaitpartFrontend (
LlmPanel.tsx,OverviewPage.tsx,types.ts)q27badge/label everywhere backend names are renderedvllm || q27; all tiles are null-safeDecode benchmark (
LlmStreaming.js)readServerGenerationTokensreads the q27 live processed counter (completion-total fallback), so bench results include server-side generation tok/s for q27 tooAlso in this branch
a75af15):_readHostNetFile's fallback re-dispatched through_readHostFile, which routes/proc/net/*back into itself →Maximum call stack size exceededon collectors without a host proc-netns bind mount. Now reads/proc/netdirectly. Also addsswapto the pseudo-fstype set so lsblk's[SWAP]pseudo-mount stops failingstatfswith ENOENT on every poll.Docs
[Unreleased] → AddedTesting
LlmProbe.q27.test.js: detection viaowned_byand via/metrics(including processed-only expositions), gauge/counter parsing, idle → 0 tok/s, live processed counters driving real-time rates, cached-jump not inflating prefill,+Inf != _countrefusing the quantile, and the full probe path (no vllm mislabel, context frommax_model_len)showcasePrompts.test.js, fails identically onmain— stale source assertion, unrelated)npm run typecheck✅ ·npm run build✅Screenshots
Overview — q27 auto-detected, live decode tok/s on the Spark card
Spark detail — full q27 LLM panel (tiles populated from
q27_*metrics)Live tok/s split by kind, KV cache, Requests reading
0 runwithout a wait gauge (FIFO), Preempts at0(never preempts), TTFT/E2E/ITL p95, prefix cache, MTP accept, and context from/v1/modelsmax_model_len: