Skip to content

refactor: admitperf is the package; the cluster moves out - #55

Merged
harshuljain13 merged 2 commits into
mainfrom
refactor/package-only
Oct 7, 2026
Merged

harshuljain13 merged 2 commits into
mainfrom
refactor/package-only

Conversation

@harshuljain13

Copy link
Copy Markdown
Owner

pip install admitperf should get you Signal, Policy, Decision, Log and the reports — not a Kubernetes cluster.

−10,609 lines. 129 tests, lint clean.

What's removed

why
infra/ — chart, subcharts, bring-up scripts, manifests, renderer a policy library that ships a Helm chart is a test harness pretending to be a library. The cluster that exercises this now lives in llm-inference-experiments/admitperf_testing, which is module 10 — the version that actually deployed on real hardware
src/admitperf/load.py and admitperf load offering traffic has nothing to do with deciding whether to admit it. A load generator inside a policy library implies the library drives the experiment, rather than being called from someone else's gateway
.env.example — 168 lines documented infra/render.py, LAMBDA_HOST_*, PREFILL_URLS, KV_BACKEND, mooncake, overflow. None of it is in this repo, and none of it was ever read — grep -r os.environ src/admitperf returns nothing. The file's own warning applied to itself: "a variable the code does not read is worse than a missing one, because it looks configured."

The CLI now only reads and writes AdmitPerf logs:

report  compare  experiments  signals  policies  dashboard  demo  watch

Two fixes found by using it

compare --experiment X from the wrong directory printed a raw traceback. Once caught, it scanned my entire home directory and offered sixteen unrelated folders as suggestions. Now:

no experiment named 'nope' under /Users/harshuljain/Desktop

  No named experiments here. Logs are found under the current directory,
  and an experiment is the name a policy was given — not a folder name.

    admitperf experiments        list what is here
    admitperf demo               make some, no hardware needed

Report.arm → Report.label. "Arm" is clinical-trial jargon that tells a reader nothing — the thing on screen is a policy and its runs. The rename collided with the existing Report.run (the run's name, not its display label), which would have silently handed callers the wrong string. "Arm" is now gone from the repo except for one comment in policy_runs.py explaining why.

Five layering rules so this cannot drift back

no infra/, chart/ or k8s-config/ directory
no load.py in the package
the package reads NO environment variables
no .env.example
admitperf.core: stdlib only, no sockets, no subprocesses

That third one is the load-bearing one. A policy's behaviour comes from its arguments, so two deployments passing identical parameters decide identically — otherwise a report naming kv_threshold 0.90 describes something the reader cannot reconstruct.

🤖 Generated with Claude Code

harshuljain13 and others added 2 commits October 6, 2026 21:42
A policy library that ships a Helm chart is a test harness pretending to be a
library, and keeping both in one repo let that confusion persist for days.
`pip install admitperf` should get you Signal, Policy, Decision, Log and the
reports — not a Kubernetes cluster.

  removed   infra/ entirely: the chart, the subcharts, the bring-up scripts, the
            manifests, the renderer. The cluster that exercises this now lives in
            llm-inference-experiments/admitperf_testing, which is module 10 — the
            version that actually deployed on real hardware.

  removed   src/admitperf/load.py and its CLI command. Offering traffic has
            nothing to do with deciding whether to admit it, and a load generator
            inside a policy library implies the library drives the experiment
            rather than being called from someone else's gateway.

The CLI now only reads and writes AdmitPerf logs:

    report  compare  experiments  signals  policies  dashboard  demo  watch

Two fixes to that surface, both found by using it:

  `compare --experiment X` from the wrong directory printed a raw traceback, then
  once caught, scanned the whole home directory and listed sixteen unrelated
  folders as suggestions. It now lists only NAMED experiments, caps the list, and
  says what to run instead.

  Report.arm is Report.label. "Arm" is clinical-trial jargon that tells a reader
  nothing; the thing on screen is a policy and its runs. The rename collided with
  the existing Report.run — the run's NAME, not its display label — which would
  have silently handed callers the wrong string.

tests/test_layering.py gains three rules so this cannot drift back: no infra/,
chart/ or k8s-config/ directory, and no load.py in the package.

127 tests, lint clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
168 lines documenting infra/render.py, infra/setup/*.sh, LAMBDA_HOST_*,
PREFILL_URLS, KV_BACKEND, mooncake and overflow — none of which is in this repo
any more, and none of which the package ever read.

`grep -r os.environ src/admitperf` returns nothing. The file's own warning applied
to itself: "a variable the code does not read is worse than a missing one, because
it looks configured."

Cluster credentials belong with the cluster, in
llm-inference-experiments/admitperf_testing, which has its own .env.example.

Two tests keep it that way:

  the package reads no environment variables — a policy's behaviour comes from its
  arguments, so two deployments passing identical parameters decide identically. A
  report naming `kv_threshold 0.90` has to describe something a reader can
  reconstruct.

  no .env.example in the package.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@harshuljain13
harshuljain13 merged commit 5bcb81c into main Oct 7, 2026
4 checks passed
@harshuljain13
harshuljain13 deleted the refactor/package-only branch October 7, 2026 09:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant