Skip to content

refactor: one Helm chart replaces infra's scripts, manifests and renderer - #54

Merged
harshuljain13 merged 2 commits into
mainfrom
refactor/infra-helm-chart
Oct 6, 2026
Merged

harshuljain13 merged 2 commits into
mainfrom
refactor/infra-helm-chart

Conversation

@harshuljain13

Copy link
Copy Markdown
Owner

Replaces infra/ with a single Helm chart. A topology is a values file, not a script.

make plan                      what would deploy, and whether a run on it can mean anything
make up                        deploy it
make up VALUES=disagg          a different topology
make smoke                     does it actually serve?
make down                      remove it

Supersedes #53 (its commit is included here).

Why

infra/ had three things each half-owning deployment: fifteen ordered shell scripts, static manifests with values baked in, and a Python renderer generating some of the same resources. They disagreed, and every disagreement produced a cluster that came up healthy and reported numbers that meant nothing — the expensive kind of bug, because nothing about the run looks wrong.

Found and fixed, all of them hit during a real bring-up today:

what happened why
lambda_vllm.sh served at gpu-memory-utilization 0.85 against the config's 0.90, and omitted --max-num-seqs the flag lived in two files. max_num_seqs is the one setting that decides whether a KV policy can fire
mooncake-store deployed onto a one-GPU box with no second engine to hand KV to nothing tied the transport to the topology needing it
HAMi and KEDA installed unconditionally while --components reported both skipped the bring-up script had no view of the config
the gateway came up healthy pointing at http://vllm-prefill:8000 on an aggregated run the manifest hardcoded a disaggregated topology
every request returned 400 prompt_too_long the guardrail's hardcoded 8,000-character limit, against a ~33,000-character prompt at the KV crossover
28 of 32 requests returned 429 tenant_tokens a 10,000 tokens/min default — under two requests at experiment length. A rate limit is client-attributable and says nothing about capacity, but in a report it is a refusal indistinguishable from the policy working
the node advertised nvidia.com/gpu: 0, engines stuck Pending no device plugin. A container runtime grants access to a card; a plugin advertises it as schedulable — two different things

What replaces it

One chart that loops .Values.pools, with HAMi, KEDA and kube-prometheus-stack as conditional subcharts — so a config that wants none installs none.

13 combinations verified from the same templates, including ones with no values file of their own (8 replicas, autoscaling on top of slicing, disagg without a UI).

helm template now refuses the bad combinations in one second on a laptop:

refusals:
  a pool name the gateway cannot address       refused
  a KV transport with no handoff               refused
  mode and pools disagreeing                   refused
  autoscaling without KEDA                     refused
  slicing without the HAMi scheduler           refused
  HAMi with nothing to slice                   refused
  a chat UI bypassing the gateway              refused
  no device plugin at all                      refused
  two device plugins on one card               refused

guarantees:
  aggregated gateway points at its engine      ok
  disagg gateway separates the pools           ok
  guardrail cannot clip the prompt             ok
  tenant rate limit is off by default          ok
  max-num-seqs reaches the engine              ok
  engines are probed on /v1/models             ok
  prometheus samples fast enough               ok
  no API key is rendered into a manifest       ok

pytest runs that suite, so CI covers the configs that used to deploy cleanly and prove nothing.

Nothing was dropped

infra/lib is the lab's Python moved byte-for-byte — gateway, router, KV transports, overflow, kernels, dashboards. I diffed it. All eleven Grafana dashboards ship as a ConfigMap. The 17 docs came across, with a banner on the five whose bring-up commands are superseded.

infra/plan.py

Keeps the one thing Helm cannot do: whether a run can mean anything.

3  CAN A POLICY FIRE                           the validity gate
     crossover   ~5.7k tokens/request   (363k pool / 64 seqs)

      8192 + 512 =  8704 tok   KV binds          a policy can fire
      2048 + 120 =  2168 tok   scheduler binds   ceiling 0.38 — INERT

Writing it surfaced two more errors:

  • the slice arithmetic computed usable memory from the whole card, overstating cache by 3×
  • sliced with a 7B leaves ~1 GB of cache between two engines — 14k tokens, under two requests. That measures eviction thrash, not admission. That is why the lab used a 3B there, and sliced.yaml now does too

Tests

tests/test_render_topology.py → tests/test_plan.py. The arithmetic and hardware refusals moved to plan.py; the topology refusals moved into the chart. 160 passing.

Known gap

The gateway does not call AdmitPerf. lib/gateway/admission.py and lib/router/router.py carry their own tenant bucket and KV_SATURATION check and import nothing from admitperf — so the policy named in values reaches the pod's environment and is read by nothing.

Stated in the README, in values.yaml, and in section 4 of make plan rather than hidden, because a silent gap means the next run measures the lab's admission logic while the report prints the policy's name. Closing it is the next piece of work.

🤖 Generated with Claude Code

harshuljain13 and others added 2 commits October 5, 2026 20:11
These were written, tested, and then stranded on an unmerged branch while the
package was restructured. They are the deployment half of the real-fleet run and
main still had only the old vendored cluster.yaml.

    python -m infra.render infra/config/single.yaml --plan

    gpu-1  A100 x1  using 1/1
        vllm-engine-0  TP=1 PP=1  1 GPU  :8000  Qwen/Qwen2.5-7B-Instruct
                       22GB KV = ~384k tokens  (~12 seqs at max_model_len 32768)
                       max_num_seqs=64 -> KV binds only above ~6.0k tokens/request;
                       below that the SCHEDULER binds
                       so a KV policy can fire only if the experiment sends
                       requests longer than ~6.0k tokens

That last line is the whole value: it is readable before renting anything, and it
is the difference between an hour that produces a finding and an hour that produces
a flat signal for a reason nobody noticed.

    python -m infra.render infra/config/single.yaml --serve

Emits the `vllm serve` line from the same config the manifests come from, so the
bare-VM path cannot drift from it. infra/setup/lambda_vllm.sh is the reason this
matters: it hardcodes gpu-memory-utilization 0.85 against the config's 0.90 and
omits --max-num-seqs altogether — which is precisely the flag that decides whether
KV binds before the scheduler, and therefore whether a KV-pressure policy can fire
at all. A test asserts both flags appear.

single.yaml now declares ONE host rather than inheriting four. base.yaml carries
four from when the fleet had four instances, and a plan listing three machines with
"(no pools placed here)" buries the one line that matters under hosts nobody rented.

That changed a test's premise rather than breaking it: the variants no longer share
`hosts`, because a single-GPU config and a four-GPU disaggregated one genuinely need
different fleets. The invariant is now model, engine, overflow, admission and
placement — the things a comparison actually depends on.

Also drops infra/config/cluster.yaml and infra/providers/ (the Modal path), both
superseded.

220 passed, lint clean. infra/ is not part of the package; the wheel is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…erer

infra/ had three things each half-owning deployment: fifteen ordered shell
scripts, static manifests with values baked in, and a Python renderer generating
some of the same resources. They disagreed, and every disagreement produced a
cluster that came up healthy and reported numbers that meant nothing:

  - lambda_vllm.sh served at gpu-memory-utilization 0.85 against the config's
    0.90 and omitted --max-num-seqs, the one flag that decides whether a KV
    policy can fire at all
  - mooncake-store deployed onto a one-GPU box with no second engine to hand KV
    to, because nothing tied the transport to the topology that needs it
  - HAMi and KEDA installed unconditionally while `--components` reported both
    skipped, so the plan said one thing and the cluster ran another
  - the gateway came up healthy pointing at http://vllm-prefill:8000 on an
    aggregated run, a Service that did not exist
  - every request returned 400 prompt_too_long: the guardrail's hardcoded 8,000
    CHARACTER limit against a ~33,000 character prompt at the KV crossover
  - 28 of 32 requests returned 429 tenant_tokens from a 10,000 tokens/min
    default, under TWO requests at experiment length. A rate limit is
    client-attributable and says nothing about capacity, but in a report it is a
    refusal indistinguishable from the policy working
  - the node advertised nvidia.com/gpu: 0 with no device plugin, so engines sat
    Pending. A runtime grants ACCESS to a card; a plugin ADVERTISES it

A topology is now a values file. One chart loops .Values.pools, and HAMi, KEDA
and kube-prometheus-stack are conditional subcharts, so a config that wants none
installs none. Thirteen combinations were verified from the same templates,
including ones with no values file of their own (eight replicas, autoscaling on
top of slicing).

`helm template` refuses the nine combinations above on a laptop in one second,
and eight guarantees assert the fixes hold. pytest runs that suite, so CI covers
the configs that used to deploy cleanly and prove nothing.

Nothing was dropped. infra/lib is the lab's Python moved byte-for-byte —
gateway, router, KV transports, overflow, kernels, dashboards. All eleven Grafana
dashboards ship as a ConfigMap. The docs came across with a banner on the five
files whose bring-up commands are superseded.

infra/plan.py keeps the one thing Helm cannot do: whether a run can mean
anything. It found two more errors while being written — the slice arithmetic
computed usable memory from the whole card, overstating cache by 3x, and sliced
with a 7B leaves ~1 GB of cache between two engines, which measures eviction
thrash rather than admission. That is why the lab used a 3B there.

tests/test_render_topology.py is replaced by tests/test_plan.py: the arithmetic
and hardware refusals moved to plan.py, the topology refusals to the chart.

KNOWN GAP, stated in the README, in values.yaml and in section 4 of `make plan`:
the gateway does not call AdmitPerf. lib/gateway/admission.py and
lib/router/router.py carry their own tenant bucket and KV_SATURATION check and
import nothing from admitperf, so the policy named in values reaches the pod's
environment and is read by nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@harshuljain13
harshuljain13 merged commit b7627c5 into main Oct 6, 2026
4 checks passed
@harshuljain13
harshuljain13 deleted the refactor/infra-helm-chart branch October 6, 2026 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant