Repository navigation
refactor: one Helm chart replaces infra's scripts, manifests and renderer - #54
Merged
Merged
Conversation
These were written, tested, and then stranded on an unmerged branch while the
package was restructured. They are the deployment half of the real-fleet run and
main still had only the old vendored cluster.yaml.
python -m infra.render infra/config/single.yaml --plan
gpu-1 A100 x1 using 1/1
vllm-engine-0 TP=1 PP=1 1 GPU :8000 Qwen/Qwen2.5-7B-Instruct
22GB KV = ~384k tokens (~12 seqs at max_model_len 32768)
max_num_seqs=64 -> KV binds only above ~6.0k tokens/request;
below that the SCHEDULER binds
so a KV policy can fire only if the experiment sends
requests longer than ~6.0k tokens
That last line is the whole value: it is readable before renting anything, and it
is the difference between an hour that produces a finding and an hour that produces
a flat signal for a reason nobody noticed.
python -m infra.render infra/config/single.yaml --serve
Emits the `vllm serve` line from the same config the manifests come from, so the
bare-VM path cannot drift from it. infra/setup/lambda_vllm.sh is the reason this
matters: it hardcodes gpu-memory-utilization 0.85 against the config's 0.90 and
omits --max-num-seqs altogether — which is precisely the flag that decides whether
KV binds before the scheduler, and therefore whether a KV-pressure policy can fire
at all. A test asserts both flags appear.
single.yaml now declares ONE host rather than inheriting four. base.yaml carries
four from when the fleet had four instances, and a plan listing three machines with
"(no pools placed here)" buries the one line that matters under hosts nobody rented.
That changed a test's premise rather than breaking it: the variants no longer share
`hosts`, because a single-GPU config and a four-GPU disaggregated one genuinely need
different fleets. The invariant is now model, engine, overflow, admission and
placement — the things a comparison actually depends on.
Also drops infra/config/cluster.yaml and infra/providers/ (the Modal path), both
superseded.
220 passed, lint clean. infra/ is not part of the package; the wheel is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…erer
infra/ had three things each half-owning deployment: fifteen ordered shell
scripts, static manifests with values baked in, and a Python renderer generating
some of the same resources. They disagreed, and every disagreement produced a
cluster that came up healthy and reported numbers that meant nothing:
- lambda_vllm.sh served at gpu-memory-utilization 0.85 against the config's
0.90 and omitted --max-num-seqs, the one flag that decides whether a KV
policy can fire at all
- mooncake-store deployed onto a one-GPU box with no second engine to hand KV
to, because nothing tied the transport to the topology that needs it
- HAMi and KEDA installed unconditionally while `--components` reported both
skipped, so the plan said one thing and the cluster ran another
- the gateway came up healthy pointing at http://vllm-prefill:8000 on an
aggregated run, a Service that did not exist
- every request returned 400 prompt_too_long: the guardrail's hardcoded 8,000
CHARACTER limit against a ~33,000 character prompt at the KV crossover
- 28 of 32 requests returned 429 tenant_tokens from a 10,000 tokens/min
default, under TWO requests at experiment length. A rate limit is
client-attributable and says nothing about capacity, but in a report it is a
refusal indistinguishable from the policy working
- the node advertised nvidia.com/gpu: 0 with no device plugin, so engines sat
Pending. A runtime grants ACCESS to a card; a plugin ADVERTISES it
A topology is now a values file. One chart loops .Values.pools, and HAMi, KEDA
and kube-prometheus-stack are conditional subcharts, so a config that wants none
installs none. Thirteen combinations were verified from the same templates,
including ones with no values file of their own (eight replicas, autoscaling on
top of slicing).
`helm template` refuses the nine combinations above on a laptop in one second,
and eight guarantees assert the fixes hold. pytest runs that suite, so CI covers
the configs that used to deploy cleanly and prove nothing.
Nothing was dropped. infra/lib is the lab's Python moved byte-for-byte —
gateway, router, KV transports, overflow, kernels, dashboards. All eleven Grafana
dashboards ship as a ConfigMap. The docs came across with a banner on the five
files whose bring-up commands are superseded.
infra/plan.py keeps the one thing Helm cannot do: whether a run can mean
anything. It found two more errors while being written — the slice arithmetic
computed usable memory from the whole card, overstating cache by 3x, and sliced
with a 7B leaves ~1 GB of cache between two engines, which measures eviction
thrash rather than admission. That is why the lab used a 3B there.
tests/test_render_topology.py is replaced by tests/test_plan.py: the arithmetic
and hardware refusals moved to plan.py, the topology refusals to the chart.
KNOWN GAP, stated in the README, in values.yaml and in section 4 of `make plan`:
the gateway does not call AdmitPerf. lib/gateway/admission.py and
lib/router/router.py carry their own tenant bucket and KV_SATURATION check and
import nothing from admitperf, so the policy named in values reaches the pod's
environment and is read by nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces
infra/with a single Helm chart. A topology is a values file, not a script.Supersedes #53 (its commit is included here).
Why
infra/had three things each half-owning deployment: fifteen ordered shell scripts, static manifests with values baked in, and a Python renderer generating some of the same resources. They disagreed, and every disagreement produced a cluster that came up healthy and reported numbers that meant nothing — the expensive kind of bug, because nothing about the run looks wrong.Found and fixed, all of them hit during a real bring-up today:
lambda_vllm.shserved atgpu-memory-utilization 0.85against the config's0.90, and omitted--max-num-seqsmax_num_seqsis the one setting that decides whether a KV policy can firemooncake-storedeployed onto a one-GPU box with no second engine to hand KV to--componentsreported both skippedhttp://vllm-prefill:8000on an aggregated run400 prompt_too_long429 tenant_tokensnvidia.com/gpu: 0, engines stuckPendingWhat replaces it
One chart that loops
.Values.pools, with HAMi, KEDA and kube-prometheus-stack as conditional subcharts — so a config that wants none installs none.13 combinations verified from the same templates, including ones with no values file of their own (8 replicas, autoscaling on top of slicing, disagg without a UI).
helm templatenow refuses the bad combinations in one second on a laptop:pytestruns that suite, so CI covers the configs that used to deploy cleanly and prove nothing.Nothing was dropped
infra/libis the lab's Python moved byte-for-byte — gateway, router, KV transports, overflow, kernels, dashboards. I diffed it. All eleven Grafana dashboards ship as a ConfigMap. The 17 docs came across, with a banner on the five whose bring-up commands are superseded.infra/plan.pyKeeps the one thing Helm cannot do: whether a run can mean anything.
Writing it surfaced two more errors:
slicedwith a 7B leaves ~1 GB of cache between two engines — 14k tokens, under two requests. That measures eviction thrash, not admission. That is why the lab used a 3B there, andsliced.yamlnow does tooTests
tests/test_render_topology.py→tests/test_plan.py. The arithmetic and hardware refusals moved toplan.py; the topology refusals moved into the chart. 160 passing.Known gap
The gateway does not call AdmitPerf.
lib/gateway/admission.pyandlib/router/router.pycarry their own tenant bucket andKV_SATURATIONcheck and import nothing fromadmitperf— so the policy named in values reaches the pod's environment and is read by nothing.Stated in the README, in
values.yaml, and in section 4 ofmake planrather than hidden, because a silent gap means the next run measures the lab's admission logic while the report prints the policy's name. Closing it is the next piece of work.🤖 Generated with Claude Code