Repository navigation
Measured-only placement planner for worker pools (sie_placement) #903
Description
Problem
Operators size worker pools by hand today. A Helm worker pool per machine profile, minReplicas, the pool warm floor (minimum_worker_count) and pinned_models decide which models run on which GPUs and which stay warm. Each choice trades three things: how long a request waits after a model has been idle, what the warm capacity costs, and how fast the model answers once warm. Nothing in the repository states the target for those three or derives a pool layout from measurements.
Proposal
Add a measured-only placement planner as a new workspace package, sie_placement, standard library only, not published to PyPI at first.
- Input documents, versioned JSON. Per served identity (a model id or
model:profile) they hold a come-alive p95, a warm p99 (time to first token and inter-token latency for generation) and a queue-wait p99, with optional named tiers. Receipts are throughput and latency measurements per identity on a machine profile and the GPU product the measurement actually ran on. The other documents are boot evidence, a price list per machine profile, and hourly demand. - Planner core, a pure function. It keeps only measured cells that meet the warm bound, picks the cheapest warmth that meets the come-alive bound, and emits a plan with a per-lane envelope (floor, buffer and scale-down window ranges). Its report gives the cost of each service level, the next cheaper option and the bound that blocks it, and ranked measurements worth taking. Cells without measurements are never chosen.
- Tuner step, a pure function that keeps a lane's floor inside its envelope from live queue signals.
- Optional pieces: a plan-to-Helm-values emitter, a
sie-placement tuneapplier that moves a lane's warm floor inside its envelope through an API logical pool, and an opt-in chart workload for it.
The gateway's pool specification and the chart's KEDA triggers do not change.
Delivery
Eight stacked pull requests: schemas, planner core, tuner step, GPU availability, several models on one GPU, Helm values, the applier, and chart integration.
Activity
krisztian-gajdar commentedon Nov 6, 2026
Stack status. Every PR in the stack, #904 to #911 plus #913, is approved by CodeRabbit on its current head. The trust-boundary PRs have independent adversarial reviews with no blocking findings:
440321d14214cec23361f#913 carries the last non-blocking findings from the #910 and #911 reviews, and #911 should merge together with it.
A follow-up PR, still to open, will carry the six non-blocking findings from #913 review run 2 (#913 (comment)):
values.yaml.krisztian-gajdar commentedon Nov 6, 2026
Deferred from #914, which resolves the six findings of #913 review run 2. The independent reviews of #914 reported no blocking findings in runs 1, 2 and 3. The items below are non-blocking and fail closed. They are accepted and deferred so that #914 can converge. Dispositions: #914 (comment)
sie-cluster.wiredSecretKeysevaluateskube-prometheus-stack.grafana.admin.existingSecretbefore it checks Grafana'senvconditions. A template that Grafana never evaluates can therefore still fail the render. It would be enough to evaluate the template only when Grafana reads the admin Secret, as NOTES now does.0o-7and2001-12-14. The check modelssigs.k8s.io/yaml, which Helm and kubectl use. The API server rejects both rewritten forms..Valuesfield starts with NBSP or U+3000 is trimmed by the check but kept by Grafana. This only causes an extra refusal.krisztian-gajdar commentedon Nov 6, 2026
Deferred from #914, item 4 (adversarial review run 4, #914 (comment)):
SSL_CERT_DIRdirectory that the applier can list but not search, such as mode 0600 or 0400, passes the start-up check. Every gateway call then fails withCERTIFICATE_VERIFY_FAILED, as with httpx's default client. This is pre-existing and fails closed, and the package README already describes it. Suggested fix: refuse a location that failsos.access(path, os.X_OK, effective_ids=...)before scanning it.krisztian-gajdar commentedon Nov 9, 2026
The
sie_placementpackage is inmain: #904 (schemas) landed on its own, and #905 to #911 and #913 landed together through #914 (squash commit c2cf482), because the merge queue squashes each pull request and the stacked branches conflicted with their predecessors' squash commits. The deferred review items listed above stay open here.