Skip to content

feat: detect and parse q27 (signalnine/q27 engine) LLM backend - #74

Open
PierpaoloPernici wants to merge 7 commits into
MiaAI-Lab:mainfrom
PierpaoloPernici:feat/q27-backend
Open

feat: detect and parse q27 (signalnine/q27 engine) LLM backend#74
PierpaoloPernici wants to merge 7 commits into
MiaAI-Lab:mainfrom
PierpaoloPernici:feat/q27-backend

Conversation

@PierpaoloPernici

Copy link
Copy Markdown

What

Adds q27 (signalnine/q27 engine) as a first-class LLM backend for the LLM probe, instead of mislabeling it as vLLM. Once detected, the LLM panel gets the full live-telemetry treatment: real-time decode/prefill tok/s, inference-health tiles, and backend-aware labels/badges.

Why

q27 exposes an OpenAI-compatible /v1/models plus Prometheus /metrics, so the probe fell through to the default vLLM classification — the badge read "vLLM" and every vLLM-specific tile stayed empty, because the vllm:* series it looks for don't exist on q27. (The q27 /metrics endpoint itself landed upstream in signalnine/q27#40; this PR teaches sparkDash to speak it.)

How

Detection (LlmProbe.js)

  • /v1/models with owned_by: "q27", or
  • Prometheus /metrics exposing q27_decode_tokens_total, q27_decode_tokens_processed_total, or q27_requests_total (regex anchored so q27_requests_errors_total and friends don't false-positive)
  • Once classified q27, a failing /metrics no longer resets the backend to vLLM (same guard as ds4/exl3)

Metrics (_applyQ27Metrics)

  • Sums across api= label sets (chat / completions / messages / responses), exactly like the ds4/vLLM paths
  • Real-time tok/s from q27_*_processed counter deltas (they move during generation); falls back to the completion-based per-api totals for older binaries (step function: 0 while generating, jump at completion)
  • Prefill is exact (per-request accounting, not vLLM estimates) and follows the ds4 convention: the main tile counts computed tokens only, so cache hits don't inflate the "real work" rate; the cached/uncached split doubles as the prefix-cache hit rate
  • KV usage %, slots (inflight / total), TTFT / E2E / ITL p95 from q27_*_seconds histograms (with the +Inf == _count invariant check — mismatch refuses the quantile), and MTP/speculative accept ratio
  • Engine state reports Active whenever the server is reachable (weights always resident, no sleep state), like the SGLang path
  • Preemptions read 0 — q27 FIFO-queues and never preempts; the server exposes a constant-0 counter so the tile shows 0 instead of
  • No wait gauge — the Requests tile renders N run without the wait part

Frontend (LlmPanel.tsx, OverviewPage.tsx, types.ts)

  • q27 badge/label everywhere backend names are rendered
  • The inference-health tile grids render for vllm || q27; all tiles are null-safe
  • Metric tooltips reworded backend-neutral ("from the engine's request history")

Decode benchmark (LlmStreaming.js)

  • readServerGenerationTokens reads the q27 live processed counter (completion-total fallback), so bench results include server-side generation tok/s for q27 too

Also in this branch

  • SystemCollector fix (a75af15): _readHostNetFile's fallback re-dispatched through _readHostFile, which routes /proc/net/* back into itself → Maximum call stack size exceeded on collectors without a host proc-netns bind mount. Now reads /proc/net directly. Also adds swap to the pseudo-fstype set so lsblk's [SWAP] pseudo-mount stops failing statfs with ENOENT on every poll.

Docs

  • README: q27 in the feature table (auto-detect list, cached/uncached prefill backends), "LLM inference health" row (tiles now render for q27 too), and the LLM-probe backend list (also adds the missing EXL3 bullet)
  • CHANGELOG: entry under [Unreleased] → Added

Testing

  • 10 new unit tests in LlmProbe.q27.test.js: detection via owned_by and via /metrics (including processed-only expositions), gauge/counter parsing, idle → 0 tok/s, live processed counters driving real-time rates, cached-jump not inflating prefill, +Inf != _count refusing the quantile, and the full probe path (no vllm mislabel, context from max_model_len)
  • Full suite: 173 tests, 172 passing (the 1 failure, showcasePrompts.test.js, fails identically on main — stale source assertion, unrelated)
  • npm run typecheck ✅ · npm run build

Screenshots

Overview — q27 auto-detected, live decode tok/s on the Spark card

Overview card with the q27 backend badge and live tok/s

Spark detail — full q27 LLM panel (tiles populated from q27_* metrics)

Live tok/s split by kind, KV cache, Requests reading 0 run without a wait gauge (FIFO), Preempts at 0 (never preempts), TTFT/E2E/ITL p95, prefix cache, MTP accept, and context from /v1/models max_model_len:

q27 LLM panel with live tiles

- _readHostNetFile fallback now reads /proc/net directly via fs.readFileSync
  instead of re-dispatching through _readHostFile (which routes /proc/net/*
  back into this method), fixing 'Maximum call stack size exceeded' on
  collectors without a mounted host proc netns.
- Add 'swap' to the PSEUDO fstype set so the [SWAP] pseudo-mount from
  lsblk is skipped instead of failing statfs with ENOENT on every poll.
q27 (signalnine/q27) exposes Prometheus /metrics under q27_* series with
an api= label (chat/completions/messages/responses). Add it as a first-
class backend:

- detection: owned_by "q27" on /v1/models, plus a _probeIsQ27 fallback
  that regex-checks /metrics for q27_ series (mirrors the ds4 path);
  exclude q27 from the /slots skip list and the vllm fallback branches
- _applyQ27Metrics: live tok/s from counter deltas (idle -> 0), exact
  computed-only prefill (ds4 convention; cache-served tokens never
  inflate the main tile), cached/uncached prefill split + prefix-cache
  hit rate, KV cache usage, slots, inflight, spec accept ratio, and
  TTFT/E2E/ITL p95 from histograms (+Inf == _count enforced, same
  quantile logic as vLLM)
- refactor _parseVllmHistogram into a generic _parseHistogram so q27
  (or future backends) reuse the cumulative-bucket parsing
- frontend: backend union + label "q27" in the panel and overview; the
  vLLM-gated tile rows (KV/requests/TTFT/preempts/prefix/E2E/ITL/MTP)
  render for q27 too, with missing series showing "—"; Requests tile now
  shows "N run" when the backend has no waiting gauge
- tests: LlmProbe.q27.test.js (detection, rates, split, histograms,
  +Inf mismatch refusal, no vllm mislabel) — full LlmProbe suite green
The earlier edit batch for LlmPanel was rejected as a unit (duplicate
gate pattern) and only the gate changes survived; re-apply the two
dropped edits: the q27 label in the backend badge map and the
Requests tile fallback (show 'N run' when the backend has no waiting
gauge, e.g. q27).
…v1/models

- Engine tile: q27 keeps weights resident and is ready whenever the
  server is up (no sleep state), so _applyQ27Metrics reports
  gpuMemoryUtilization=1 (Active) like the SGLang path
- Preempts tile: q27 exposes a constant-0 q27_preemptions_total
  counter (FIFO admission, never preempts) — reads 0 instead of "—"
- Context tile: q27 /v1/models now returns max_model_len, so
  contextLength populates from the probe
- tests: fixture + assertions for the three fields
q27 now exposes unlabeled live counters (q27_*_tokens_processed_total)
incremented as tokens are produced (decode per delivered token, prefill
at prefill end), so a 2s poll sees generation progress instead of a
completion-time step. _applyQ27Metrics prefers the live series and falls
back to the completion-based totals for older q27 binaries. Tests cover
both paths (88/88).
…ver tok/s, docs

- _metricsLookLikeQ27 also matches q27_decode_tokens_processed_total so a
  new-binary exposition without q27_decode_tokens_total still classifies q27
- readServerGenerationTokens (DecodeBench server-side tok/s) reads the q27
  live processed counter with the completion-total fallback (same preference
  as LlmProbe)
- reformat the >100-char q27 guard lines to match file style
- fix _applyQ27Metrics docstring: preemptions are a constant-0 counter (not a
  null tile) and clarify the computed-only prefill convention
- backend-neutral tooltip wording (no 'from vLLM's history') now that the
  p95 tiles render for q27 too
- tests: processed-only detection + readServerGenerationTokens q27
…released

- feature table: q27 in the auto-detect list and cached/uncached prefill
  backends; 'vLLM health' row renamed to 'LLM inference health' since the
  tiles now render for q27 too (FIFO, no wait gauge)
- LLM probe backend list: q27 bullet (owned_by / q27_* series, processed
  counters, p95 histograms, constant-0 preemptions) and the missing EXL3
  bullet (doc gap since 1.8.2)
- CHANGELOG [Unreleased] Added: q27 LLM backend entry
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant