Skip to content

feat: the cluster configs, and the plan that says whether a run can work - #53

Merged
harshuljain13 merged 1 commit into
mainfrom
feat/cluster-config
Oct 6, 2026
Merged

harshuljain13 merged 1 commit into
mainfrom
feat/cluster-config

Conversation

@harshuljain13

Copy link
Copy Markdown
Owner

These were written and tested, then stranded on an unmerged branch while the package was restructured. main still had only the old vendored cluster.yaml, and this is the deployment half of the real-fleet run.

The plan is the point

$ python -m infra.render infra/config/single.yaml --plan

gpu-1  A100 x1  using 1/1
    vllm-engine-0  TP=1 PP=1  1 GPU  :8000  Qwen/Qwen2.5-7B-Instruct  ~14GB/card of 36GB usable
                   22GB KV = ~384k tokens  (~12 seqs at max_model_len 32768)
                   max_num_seqs=64 -> KV binds only above ~6.0k tokens/request; below that the SCHEDULER binds
                   so a KV policy can fire only if the experiment sends requests longer than ~6.0k tokens

That last line is readable before renting anything, and it's the difference between an hour that produces a finding and an hour that produces a flat signal for a reason nobody noticed.

--serve, so the bring-up can't drift from the config

$ python -m infra.render infra/config/single.yaml --serve
vllm serve Qwen/Qwen2.5-7B-Instruct \
  --max-model-len 32768 --gpu-memory-utilization 0.9 --max-num-seqs 64 ...

infra/setup/lambda_vllm.sh is why this matters: it hardcodes gpu-memory-utilization 0.85 against the config's 0.90 and omits --max-num-seqs entirely — which is exactly the flag that decides whether KV binds before the scheduler, and therefore whether a KV-pressure policy can fire at all. A test asserts both flags appear in the generated command.

One host, not four

single.yaml now declares one host rather than inheriting base.yaml's four. Those four are from when the fleet had four instances, and a plan listing three machines with (no pools placed here) buries the line that matters under hardware nobody rented.

That changed a test's premise rather than breaking it. The variants no longer share hosts, because a single-GPU config and a four-GPU disaggregated one genuinely need different fleets. The invariant is now model, engine, overflow, admission, placement — the things a comparison actually depends on. Forcing hosts to agree was the thing producing the phantom machines.

Also drops infra/config/cluster.yaml and infra/providers/ (the Modal path), both superseded.

220 passed, lint clean. infra/ is not part of the package, so the wheel is unchanged.

These were written, tested, and then stranded on an unmerged branch while the
package was restructured. They are the deployment half of the real-fleet run and
main still had only the old vendored cluster.yaml.

    python -m infra.render infra/config/single.yaml --plan

    gpu-1  A100 x1  using 1/1
        vllm-engine-0  TP=1 PP=1  1 GPU  :8000  Qwen/Qwen2.5-7B-Instruct
                       22GB KV = ~384k tokens  (~12 seqs at max_model_len 32768)
                       max_num_seqs=64 -> KV binds only above ~6.0k tokens/request;
                       below that the SCHEDULER binds
                       so a KV policy can fire only if the experiment sends
                       requests longer than ~6.0k tokens

That last line is the whole value: it is readable before renting anything, and it
is the difference between an hour that produces a finding and an hour that produces
a flat signal for a reason nobody noticed.

    python -m infra.render infra/config/single.yaml --serve

Emits the `vllm serve` line from the same config the manifests come from, so the
bare-VM path cannot drift from it. infra/setup/lambda_vllm.sh is the reason this
matters: it hardcodes gpu-memory-utilization 0.85 against the config's 0.90 and
omits --max-num-seqs altogether — which is precisely the flag that decides whether
KV binds before the scheduler, and therefore whether a KV-pressure policy can fire
at all. A test asserts both flags appear.

single.yaml now declares ONE host rather than inheriting four. base.yaml carries
four from when the fleet had four instances, and a plan listing three machines with
"(no pools placed here)" buries the one line that matters under hosts nobody rented.

That changed a test's premise rather than breaking it: the variants no longer share
`hosts`, because a single-GPU config and a four-GPU disaggregated one genuinely need
different fleets. The invariant is now model, engine, overflow, admission and
placement — the things a comparison actually depends on.

Also drops infra/config/cluster.yaml and infra/providers/ (the Modal path), both
superseded.

220 passed, lint clean. infra/ is not part of the package; the wheel is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@harshuljain13
harshuljain13 merged commit 815208e into main Oct 6, 2026
4 checks passed
@harshuljain13
harshuljain13 deleted the feat/cluster-config branch October 6, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant