Repository navigation
engines: show n/a, not 0.0, for metrics an engine never reported - #146
Merged
Merged
Conversation
An engine that never answered its counters endpoint reads a zero rate, and the harbor draws that zero as measured. Add the field and assert what each reader must set; every true case fails until the readers fill it.
The llama.cpp family, llama-swap upstreams, strata, the Prometheus readers and FreeToken now set metrics_reported on a read that actually got counters, so a zero rate is distinguishable from silence.
An engine that never answered its counters endpoint (llama-swap without --metrics on its llama-server) drew DECODE 0.0 t/s and zero token totals. With metrics_reported false, decode, prefill and the token totals render n/a, model rows show n/a where per-model VRAM is absent, and a reported zero still prints as 0.0. The demo fixture marks the llama-swap card as reporting so the card reads live.
DECODE and PREFILL in the ENGINE table are unknown, not zero, when metrics_reported is false.
Without it the reader gets a 501 per upstream and the card shows n/a. CHANGELOG entry credits @azrael-229 and links #140.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #140, reported by @azrael-229.
Problem
llama-swap starts its llama-server without
--metrics, so each upstream's/metricsanswers501. The reader's counters stayed at zero,rates()reported 0 decode and prefill, and the harbor's engine card drew(eng.decode_tps || 0).toFixed(1). A figure the engine never sent read as a measured zero, while the node's own request log showed 14.5 t/s for the same engine.Change
proto.Engine.MetricsReported(metrics_reported, additive): true only when this read got real counters. The llama.cpp family, llama-swap upstreams, Strata, the Prometheus readers, FreeToken and the custom map set it. Every existing field is unchanged.n/athrough the existingna()helper when the flag is false; a reported zero still prints0.0. A model row without reported VRAM readsready n/a.berth status: DECODE and PREFILL print-when unreported.contrib/INTEGRATION.md: llama-swap's counters need--metricson the model'sllama-server.Proof
TestLlamaSwapWithoutMetrics: a fake llama-swap whose upstream answers the real501body. The engine stays up with its model, and the counters read unreported.ready n/aandloaded 5.0G. The first draft renderedreadyn/a, fixed in c6b443e.go test ./..., Windows build and vet, golangci-lint (0 issues),node --checkon every harbor script.Not proven live: a llama-swap started with
--metrics, because that needs an engine restart.Out of scope, as a follow-up: when an engine reports no counters, fall back to the node's own request log for decode.