Collect metrics from every inference cluster

[dennis-upbound]

Description of your changes

#363 described how a fleet's metrics get off its clusters and onto one surface. Nothing implemented it, so a platform engineer had no way to read an engine, a gateway or a GPU across clusters, and the names they would read differed per engine anyway.

This composes an OpenTelemetry collector onto every InferenceCluster and gives it two kinds to read. A MetricMapping says what a component emits and what Modelplane calls it: from, to, fromUnit where the source is measured in something other than the target name claims, and labels and part for the series that need several of a component's metrics to become one of Modelplane's. A TelemetryDestination says where it goes: name, type, endpoint, secretRef and auth are fields, and everything else an exporter takes passes through unread, because that schema is OpenTelemetry's and moves on its own schedule. Several can exist and their sinks are concatenated, so a second backend is a new object rather than an edit to one somebody else owns, and the credential named by a secretRef is copied from the control plane to each cluster that mounts it.

The function compiles a mapping to OTTL rather than an operator writing it, which keeps the collector's configuration language out of a Modelplane API and leaves room to render something else later. Five built-in mappings carry 32 metrics off vLLM, SGLang, the gateway, the picker and DCGM. Only modelplane_* leaves a cluster. The collector does not count towards a ServingStack's readiness: nothing serving depends on it, and a destination pointing somewhere unreachable should not stop the scheduler placing replicas.

Each replica publishes its own series, told apart by a replica label, and a query over a deployment combines them - sum by (deployment) for anything counted, avg for a ratio. The collector does not combine them, because a scrape of one replica is one batch and adding two cumulative readings taken at different moments is not the traffic that happened. The label is the replica's index rather than its pod, so it is bounded by the replica count and survives a restart and a rolling update.

That is 23 of the 50 metrics #363 describes, which is everything reachable by renaming one metric to one name. #476 lists the rest; most of it needs per-object state from resource-state-metrics rather than anything here.

The endpoint picker, the credential path and multiple destinations are verified on that cluster too: modelplane_route_decision_seconds arrives labelled role=picker once the gateway routes a request, a sink's secretRef Secret written only on the control plane turns up in modelplane-system on the GPU cluster with the collector mounting it, and two TelemetryDestinations render one config carrying both sinks.

It also fixes cross-cluster routing on GKE, because the gateway metrics depend on it. A cluster gateway's name was published as a headless Service and an EndpointSlice, and kube-dns builds its records from the Endpoints API and was never taught the slice one. GKE runs kube-dns, so the name answered NXDOMAIN, Envoy resolved no address for the backend, and every cross-cluster request was refused with no healthy upstream. kind runs CoreDNS, which reads the slice, which is why CI has never seen it. #486 has the detail.

Verification

Run on a GKE cluster with two L4s, vLLM serving Qwen2.5-0.5B, two replicas, loaded to saturation, and later alongside SGLang on the second GPU. Everything here is modelplane_* arriving through the composed collector.

Each layer caught what the one before it couldn't. The collector's own validate caught a value rewrite rendered into a context that cannot reach values. The local two-cluster e2e, which --verify now gates on, caught an engine scrape job matching a label value that never existed, an unnamed engine port, a gateway job asking Envoy for a path it 404s on, a substrate job ignoring prometheus.io/port beside the annotation it honoured, and every series labelled with a generated name rather than the cluster's. Real GPU hardware caught every modelplane_gpu_* series arriving empty, because nothing annotates DCGM for scraping and GKE runs its own. A second replica caught two replicas colliding into one series: the engines held 300 and 4640 prefix-cache lookups and the series read 300. Reading a Grafana dashboard caught Prometheus receiving every series stripped of its identity, since that identity is resource attributes and the remote-write exporter drops those unless asked. Running the collector against a rendered config caught scale_metric refusing an integer factor, which would have stopped it starting on any fleet converting mebibytes; feeding it a histogram in milliseconds confirmed the sum, the bounds and the bucket counts all come out converted, which the previous datapoint rewrite never reached.

Running a second engine found more. SGLang v0.4.9.post2 on the same cluster publishes three of the eight metrics its built-in named and no others: sglang:queue_time_seconds and both retraction counters don't exist at that version, or at v0.4.6 or v0.5.0, so those mappings could never match. Its two token histograms are real but need --collect-tokens-histogram beside --enable-metrics, which the guide didn't say. And a MetricMapping stored before a field became required took the whole ServingStack to ReconcileError: a CRD validates on write, not on what it already holds, so the stale object parses straight into a ValidationError and fails the pipeline step. A telemetry object now gets dropped with a warning rather than stopping the fleet placing replicas.

labels and part have no built-in using them, but both are exercised on a cluster. A mapping folding SGLang's prompt and generation counters onto one name gives modelplane_tokens_total{direction="input"} 398 and {direction="output"} 607; part: Count over its latency histogram gives modelplane_engine_requests_total 12, matching the requests sent; and a values remap turns engine_type="unified" into engine_kind="Unified" and drops the original. They are what the counters in #363 need - tokens by direction, responses by reason, requests by status.

One thing I have not done: the gateway's time-to-first-token and time-per-output-token exist only for streaming requests, and I have not driven one.

I have:

• Read and followed Modelplane's contribution process.
• Run nix flake check (or ./nix.sh flake check) and made sure it passes.
• Added or updated tests covering any composition function changes.
• Signed off every commit with git commit -s.

[github-actions]

Docs preview: https://modelplane-docs-pr-470.vercel.app (ready once the site's Content workflow finishes)

[negz]

Given this introduces new Modelplane APIs, I think it deserves documentation in the 'main' docs - I'm thinking a new 'Monitor the fleet' section (or something like that) under the Platform heading in the sidebar. Ideally we'd add those docs in this PR.

[dennis-upbound]

Moved it to Platform as "Monitor the Fleet" and deleted the Prometheus-stack guide it replaces, in this PR. The two pages that linked there link here now.

[negz]

Can you enable more than one exporter at once? Do we expect other types?

Would an API like this work?

apiVersion: modelplane.ai/v1alpha1
kind: TelemetryDestination
metadata:
  name: platform
spec:
  sinks:
  - name: otel
    protocol: OTLP                # OTLP | PrometheusRemoteWrite
    otlp:
      endpoint: https://otel.acme.example
      transport: HTTP             # HTTP | GRPC
    auth:
      scheme: Bearer              # None | Bearer | Basic
      bearer:
        secretRef:
          name: otel-token
          key: token
  - name: prometheus
    protocol: PrometheusRemoteWrite
    prometheusRemoteWrite:
      endpoint: https://prom.acme.example/api/v1/write
    auth:
      scheme: Basic
      basic:
        secretRef:
          name: prom-credentials
status:
  conditions:
  - type: Ready
    status: "False"
    reason: SecretNotFound
    message: "Sink otel: Secret modelplane-system/otel-token has no key token"

I would imagine potentially allowing multiple TDs like this but rolling them up to form the actual composed config.

The goals of this sketch:

• Use lists of named subobjects, e.g. to allow multiple sinks of the same type
• Associate sinks with their creds
• Use a required discriminator field to clearly discriminate the union

[dennis-upbound]

Took the named list and the per-sink credentials. That second one was a real bug — secretRef was one Secret for the whole object, so two sinks had to share it and tell their keys apart by prefix. Each sink mounts under its own directory now.

Left the body pass-through. The exporter schema is upstream and versioned separately from us, so a closed protocol enum means a Modelplane release per exporter and dropping the tls/retry/sending_queue settings destinations actually need — tls alone is 16 keys, and insecure_skip_verify is what an internal endpoint wants. The OTel Operator hit this twice and passed through both times, most recently typing the pipeline wiring in v1beta1 and leaving exporters/extensions as preserve-unknown-fields.

No discriminator either: Prometheus Operator uses optional siblings validated in the controller, and KEP-1027 never shipped, so there's no APIUnions gate and it'd be hand-written CEL. Say the word if you want one anyway.

[dennis-upbound]

Went further on this — took the middle. A sink now types what's Modelplane's and passes through what's OpenTelemetry's:

sinks:
- name: primary
  type: otlphttp
  endpoint: https://otel.example.internal
  secretRef:
    name: telemetry-credentials
  auth:
    bearerTokenKey: token
  config:            # anything else the exporter takes
    compression: gzip

endpoint is required and typed — every exporter has one and it was the setting most worth catching at the apiserver. auth composes the bearertokenauth extension and wires the reference, since the collector carries no credential on an exporter, only a reference to one; that was previously three hand-written stanzas. Both render after config, so a passed-through key can't redirect a sink or unpick its credential.

One scheme, not an enum of them, per the conventions note about anticipating a union with a single optional field — and bearer reads a file from the sink's own directory, which sidesteps the env-var collision two Secrets would otherwise have. Anything else still works the old way: define the extension under spec.extensions and name it from config.

Still not typing the exporter body. TLS is 16 keys, retry and queue another dozen, and prometheusremotewrite is mid-deprecation of its top-level HTTP settings as of v0.158.0 — typing that means versioning our API through theirs.

[negz]

I know a lot of this feedback is more design shaped. Sorry it's coming in so late.

[negz]

Not super actionable feedback, but I don't love that we leak OTTL here. It'd be nice to abstract it so we have the freedom to switch out to another metric collector if we wanted to in future.

OTOH the only way I can do that is something like this, which feels like it'd have the downfalls of Crossplane P&T:

apiVersion: modelplane.ai/v1alpha1
kind: MetricMapping
metadata:
  name: my-engine
spec:
  metrics:
  # Rename.
  - from: my_engine_queued_requests
    to: modelplane_requests_waiting

  # Rename and convert units. The target's unit is known, so the mapping
  # says what the source is measured in, not a factor.
  # Real cases: SGLang KV transfer in ms, DCGM energy in mJ, FB_USED in MiB,
  # thermal violation in ns.
  - from: my_engine_kv_transfer_ms
    fromUnit: Milliseconds
    to: modelplane_request_kv_transfer_seconds

  # Fold two metrics into one, told apart by a fixed label.
  # Real case: vLLM prompt and generation tokens become
  # modelplane_tokens_total{direction}.
  - from: my_engine_input_tokens_total
    to: modelplane_tokens_total
    labels:
    - name: direction
      value: input
  - from: my_engine_output_tokens_total
    to: modelplane_tokens_total
    labels:
    - name: direction
      value: output

  # Rename a label and map its values onto Modelplane's vocabulary.
  # Real case: each engine's finish vocabulary becomes one reason label.
  - from: my_engine_requests_finished_total
    to: modelplane_responses_total
    labels:
    - name: reason
      from: finish_reason
      values:
        eos: stop
        max_tokens: length
        cancelled: abort

  # Take a histogram's count as a counter.
  # Real case: the gateway's duration histogram becomes
  # modelplane_requests_total.
  - from: my_engine_request_duration_seconds
    part: Count
    to: modelplane_requests_total

[dennis-upbound]

Did it. spec.metrics is from/to/fromUnit and the function compiles to OTTL. Left out labels and part from your sketch — both real, neither has a caller yet.

fromUnit caught a live bug while I was converting: DCGM reports FB_USED in MiB and we were renaming it to modelplane_gpu_memory_used_bytes with no conversion, so the series read a millionth of the memory in use. A field that asks the question catches that; a hand-written statement doesn't.

[negz]

FWIW I was more thinking out loud on this, not actually suggesting we adopt this API. I'm okay proceeding with it, but it's not clear to me that it's a better approach than what you had originally. They both feel imperfect.

[negz]

This is a pretty old version right?

(Also might be worth asking an agent to review commentary in this PR for low value / filler content. This comment reads a bit filler-y.)

[lsviben]

Agree, we should start with a newer version plus probably add a renovate to update as OTEL stuff release quite often

[dennis-upbound]

Bumped to 0.161.0. 0.162.0 is tagged but has no multi-arch manifest on Docker Hub yet. Did a pass over the commentary in this PR and cut three, that one included. No renovate rule yet — worth a follow-up.

[dennis-upbound]

Replaced by the one in the description, which is from a two-replica run under load.

Worth keeping the finding though: taking the first of these is what caught the identity bug. Every legend rendered as /, because Prometheus was receiving each series stripped of its cluster, deployment, engine and role. Those are resource attributes, since that is what the replica merge groups on, and the remote-write exporter drops resource attributes unless asked. The debug exporter prints them, so every check before that one looked right.

[dennis-upbound]

@negz all 29 addressed and replied to in the threads. Seven commits on top; check, render and the label-gated e2e are green.

Nine of them were real bugs rather than doc fixes, and the ones worth knowing about:

• fromUnit never reached a histogram. value_double doesn't touch its sum or bucket bounds, so a histogram was renamed to seconds with buckets still in milliseconds. It's scale_metric in the metric context now — sum 4200 and bounds 100/500/2000 come out 4.2 and 0.1/0.5/2.0, checked against the collector.
• A secretRef could never work. Nothing copied the Secret to the clusters that mount it, so every destination with a credential composed a pod that could not start.
• prometheus_remote_write is the canonical name, as you thought, and it missed the default — so the canonical spelling exported every series stripped of its identity.
• The decode engine and the picker were both unscrapeable. The picker's metrics endpoint also does TokenReview by default against a ServiceAccount that can't hold the ClusterRole for it.

The collector no longer counts towards ServingStack readiness, and destinations concatenate rather than first-sorted-wins.

Two things for you:

• acrossReplicas — you're right that nothing reads it. My reasoning for keeping it is in that thread; happy to drop the field if you'd rather not ship one nothing reads.
• Move the collector off resource_to_telemetry_conversion #489 for resource_to_telemetry_conversion, which is deprecated (it warns at runtime). Left alone here because the current setting is verified against a real backend and getting resource_constant_labels subtly wrong loses identity labels silently.

[lsviben]

Left some comments that can be done as follow-ups. Most importantly as we do not have the collector or RSM at the control plane level, we do not collect some of the metrics planned in the design. We could check if we can get them from the workload clusters directly.

Furthermore, another thing caused by above is as @negz pointed out, we strayed from having the fleet telemetry converge to the control plane cluster Maybe we can document how to set it and give a TelemetryDestination example sending metrics there. It could be nice as an example for users trying it out. Ofc we can add that in a follow up documentation PR.

I think the guide in general could do a rework and making it more concise. Right now its hard to read

[lsviben]

we are not mapping all the planned metrics from the design ATM right?

Can we add a follow-up ticket with whats missing so we can add them later and not forget. I guess for some that we planned RSM on the control plane side we need to rethink how to get them as now no collector lives there.

[dennis-upbound]

Right, 24 of the 50. #476 lists the other 26 grouped by what actually blocks each one, so nothing's lost.

Your control-plane point wasn't captured there though, and it's the important one — #470 composes a collector onto each inference cluster and none onto the control plane, so there's nothing there to scrape resource-state-metrics. Added that to the issue as a thing to decide before that group can be built.

[lsviben]

what metrics are we trying to get with the substrate job atm?

[dennis-upbound]

Nothing, today. It keeps any pod annotated prometheus.io/scrape, minus the gateway, engine and DCGM pods the jobs above already have — but filter/modelplane drops everything that isn't modelplane_*, and no built-in mapping names a substrate metric. So it scrapes the stack's own components and discards all of it.

It's there so an operator can write a MetricMapping for something in the stack — llm-d, ModelExpress, cert-manager — and have it collected without us adding a scrape job per component. That's the only thing it buys today.

Given you and Nic have both leaned toward not shipping surface until there's demand for it, dropping the job until something uses it is a reasonable call. Say the word and I'll pull it — it's a scrape interval on every GPU cluster for no series.

[dennis-upbound]

Checked it on a live cluster rather than just reading the code, and it's worth correcting what I said: modelplane-substrate isn't the only empty one. Series per job right now, on a fleet serving traffic:

modelplane-engines          232
modelplane-gateway-genai     36
modelplane-gpu               16
modelplane-gateway            0
modelplane-substrate          0

modelplane-gateway scrapes the Envoy admin port, which serves only envoy_* names — I confirmed zero gen_ai_* on 19001 and 55 of them on the ext-proc's 1064. All three built-in gateway mappings name gen_ai_*, so the genai job is what actually produces modelplane_frontend_*. The admin-port job collects Envoy's whole stat surface every interval and the filter drops all of it.

So it's two of five jobs, on every cluster, for no series. Both are there for the same reason: they're what lets an operator's own MetricMapping reach the gateway or a stack component without us adding a scrape job per component.

Worth a decision either way. Keeping them means an operator can map envoy_* or a ModelExpress metric and have it just work; dropping them means anyone who tries gets silence until a release adds the job back. I don't think it should hold up this PR — happy to take it as a follow-up.

[lsviben]

Thanks @dennis-upbound for the great work.

The guide looks much better now. One thing I noticed while reading this is the acrossReplica does not really do anything and is just an informative field for users running queries. But that would also require them to check the MetricMapping where it is set. So I am not sure if its really useful, we can have such info in the docs. We can remove it in the next iteration not to block the release (ofc if you agree).

As discussed in the other comment, ill add a ticket to build a custom OTEL image for Modelplane that uses just the components we need and support. That means we will wont support all exporter type or any extension. We will though allow users to use their images if they want something not in the initial set. Another follow-up.

Anyway, excited to get this rolling so LGTM!