Skip to content

feat(measure): add the runner's cache, start-up script, hardware check, tools, service and AgentX client patch - #2156

Draft
zoroyihan7 wants to merge 28 commits into
mainfrom
feature/yihan/runner-basics
Draft

zoroyihan7 wants to merge 28 commits into
mainfrom
feature/yihan/runner-basics

Conversation

@zoroyihan7

@zoroyihan7 zoroyihan7 commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Review only — merges after the multibranch acceptance run (see AMD-AGI board #121); do not merge before.

Based on main. #2155 (the rule pack) was squash-merged into main on 2026-11-02, so the base branch of this PR is main and the diff below is this PR's own. aiperf_patch reads its constants from hyperloom.common.outside_facts, which #2155 added. Stack: this PR -> #2157 (measure leaves, base feature/yihan/measure-leaves-base) -> #2158 (measure middle).

Summary

Adds the first part of the measurement runner, hyperloom.measure.runner: the modules that import nothing of the measurement layer's core, so they can be read, tested and merged before it. Refreshed on 2026-10-30 to multibranch-dev e81281d: every file is byte-identical to that commit except the two test files listed under "What differs from multibranch-dev". The refresh is a merge of the current base (c689388cc) and one sync commit (77ac7c4f9); history is not rewritten.

The pieces:

  1. The runner modules (table below).
  2. The runner dependency group in pyproject.toml, requirements/runner.lock and the test that holds the two together.

hyperloom_common/gpu_identity.py and hyperloom_common/tests/test_gpu_identity.py are no longer part of this PR: the device-id helpers that the runner's hardware imports (GpuBoard.device_ids, normalise_device_id, gpu_type_for_device_id) landed on main with #2326.

The modules (lines), and everything each imports from this repository:

Module Lines What it is Imports
measure/__init__.py, measure/runner/__init__.py 4 + 4 docstrings only, no imports nothing
runner/cache_layout.py 371 the layout of the runner cache of one commit (wheelhouse, requirements, source) and the check that a cache is whole nothing
runner/cache_build.py 524 the builder that fills that cache on a machine with egress, for a named wheel target, with the build tools the lock pins cache_layout
runner/bootstrap.py 294 the start-up script a sandbox runs: installs the runner's software from the cache with no index, then replaces itself with the runner command; plus the template step that starts it in the background cache_layout
runner/aiperf_patch.py 193 patches the AgentX replay client so a reply it found nothing in keeps the output token count the server reported; finds the client's submodule by its name wherever the InferenceX checkout puts it common.outside_facts (#2155)
runner/hardware.py 244 what the machine says about its GPUs (device id, gfx target, model, power cap, partition mode, ECC counts, memory in use), compared with the instruction sheet; reads a busy card again until it is released hyperloom_common.gpu_identity (on main, #2326), common.rocm_smi (on main)
runner/tools.py 97 clean checkouts of Magpie and InferenceX at the commits the sheet names nothing

No module imports hyperloom.measure.spec, socket, local, manifest, signing, sandbox or any other module of the measurement core. bootstrap's last line execs python -m hyperloom.measure.runner run, a command that arrives with the runner core; the bootstrap tests run it against a stand-in runner entry that the test rig writes.

Base update on 2026-11-02

#2155 (the rule pack), #2326 (toolbox small edits, including GpuBoard.device_ids) and #2328 were squash-merged into main on 2026-11-02, so the base of this PR is now main. This branch takes it by two merges of origin/main: 8d43ad49a (main at a026a2e32) and db0db5345 (main at ae51de973, #2336, which touches none of this PR's files); history is not rewritten. The first merge conflicted in three files that came from the merged PRs, and each is resolved to main's version: hyperloom/common/outside_facts/image_check.py and hyperloom/common/tests/test_image_check.py (from #2155), and hyperloom_common/gpu_identity.py (the device_ids change, now on main from #2326). The second merge had no conflicts.

Afterwards git diff origin/main HEAD lists 26 files. They are 26 of the 28 files of this PR's diff before the update (git diff b1a96f3cf 24113f390), with the same changed lines in each. The other two, hyperloom_common/gpu_identity.py and hyperloom_common/tests/test_gpu_identity.py, are identical to main's, which has the same device-id helpers from #2326, so this PR no longer changes them.

Changes in the 2026-10-30 refresh

What differs from multibranch-dev

  1. runner/tests/conftest.py: without dev's _nobody_holds_the_cards fixture and without the two monkeypatch lines of settle_clock that patch card_watch and sandbox.card_occupancy; both modules are later layers (card_watch imports contracts.measurement, measure.failures and sandbox). The rig fixture and settle_clock for hardware are dev's.
  2. runner/tests/test_aiperf_patch.py: without its last test (test_the_reader_counts_empty_and_short_exactly_...), the _classes helper, the two imports it needs (hyperloom.contracts.measurement.RequestClasses, hyperloom.measure.agentx_requests.read_request_classes) and one clause of the docstring; that test reads the patched client's export through measure/agentx_requests.py, which imports the measurement core, and arrives with it. Dev has since moved the pop out of the assert itself (the CodeQL py/side-effect-in-assert fix), so that difference is gone.
  3. plugs/tests/test_architecture.py: as above.
  4. pyproject.toml: only the runner group, identical to multibranch-dev's. The hosted group is not added here: no module in this PR imports e2b, boto3 or cryptography. cryptography is in runner although nothing here imports it, because the extra is the input of the lock, and the lock is the whole runner's (signing included); test_runner_lock pins the two together.

What is not in this PR, and why

Size

1731 production Python lines, all in measure/, against the 1000-line diff budget (size-exception reason here): one slice of the runner, six modules and two package docstrings, taken verbatim from multibranch-dev, none longer than 524 lines (module limit 1200), none depending on anything but its listed imports. If the reviewers prefer, it splits cleanly into the cache trio (cache_layout, cache_build, bootstrap: 1189 lines) and the other three modules (534 lines); the two package docstrings (8 lines) go with either.

CI state

At head db0db5345 (2026-11-02): all checks pass (35 pass, 7 skipped, none pending); update test durations (main) and deploy (github pages) are skipped, as on every PR. 0 unresolved review threads.

Review template

  • Linked issue(s): none.
  • Tests: added, taken from multibranch-dev with the code. Run locally on a tree of the base plus this PR only (no sandbox, no other measure module), Python 3.10 and 3.11, -n 4: pytest src/hyperloom/measure src/hyperloom_common/tests/test_gpu_identity.py src/hyperloom/plugs/tests/test_architecture.py: 176 passed, 1 skipped on both (144 runner tests, of which test_runner_lock has 14; 17 gpu_identity; 16 architecture; the skipped one needs a checkout of the real aiperf client, HYPERLOOM_TEST_AIPERF_SOURCE/HYPERLOOM_TEST_AIPERF_PYTHON). The cache tests call python -m pip, so the interpreter must have pip (22.3 or newer). Both runs repeated at head db0db5345 after the 2026-11-02 base update: 176 passed, 1 skipped on 3.10 and on 3.11. Before that update, at 24113f390, the 40 test files that import gpu_identity or walk the source tree and guard packaging and imports (test_packaging_lint, test_llm_client_architecture, test_dedup_guard, test_common_import_lint, plugs, scripts/tests, ...) gave 1180 passed, 7 skipped on both; they were not repeated locally, and the full suite runs in CI at the current head. Also run: ruff check ., ruff format --check . (clean), scripts/check_broad_suppress.py src scripts (rc 0), lint-imports --no-cache (11 kept, 0 broken).
  • Size/complexity triggers crossed: the 1000-line diff budget, with the reason under "Size".
  • If this refactors: not a refactor.
  • Observable effect: none. Nothing on main imports hyperloom.measure, no CLI, entry point or flag uses it, and the legacy flow behaves as before. What exists afterwards: the new runner extra, requirements/runner.lock (read only by the sealed runner's install). The runner modules import only the standard library and the modules named in the table.
  • Breaking changes: no.
  • PR addresses single concern: yes (the runner's environment: the cache and its builder, the start-up script, the hardware check, the tool checkouts and the client patch).
  • Root cause is upstream: no.

🤖 Generated with Claude Code

zoroyihan7 and others added 6 commits October 23, 2026 20:17
…nds each fact to a name

Facts about projects outside this repository (aiter, SGLang, vLLM, PyTorch, the
HIP runtime, argparse, Transformers, InferenceX) that more than one reader
depends on are stated once, in a YAML rule pack. Each entry carries the project
that owns the fact, the reason it exists and its examples; a pattern that fails
one of its own examples, a repeated key, an alias, an unknown kind or an entry
the code reads but the pack lacks refuses the pack, naming the file and the
entry, and nothing falls back to an empty pack.

hyperloom.common.outside_facts binds each entry to the constant the readers
import. HYPERLOOM_RULE_PACK names the pack a session reads; without it the
reference copy beside the module is read. The pack ships as package data.

Nothing reads the constants yet; the next commit moves the readers that already
repeat these facts onto them.

Co-Authored-By: Claude <noreply@anthropic.com>
…n the readers that repeat them

failure_signature, env_safety, provenance, visible_devices, gpu_power_settings
and agentx_workload spelled out facts that the rule pack now owns: the HIP,
HSA, import-error, argparse and Transformers log markers, the ROCR mask name
and the InferenceX benchmark variable names. Each now imports the constant.

No behaviour change. The compiled patterns, the blocked-name set and the
provenance keys are identical to before: every pattern is built from the same
text, compared pattern by pattern.

Co-Authored-By: Claude <noreply@anthropic.com>
The test package's conftest imports hyperloom.common.gpu_power_settings, which
now reads the rule pack, and the pack is YAML. The job installs only pytest
because the lint itself is stdlib-only, so every test in it failed at setup
with "No module named 'yaml'". The job installs PyYAML beside pytest and stays
seconds long.

Co-Authored-By: Claude <noreply@anthropic.com>
A board's name is whatever the VBIOS says (an MI355X node answers "AMD
Instinct MI355 OAM"); its PCI device id is what the silicon is. gpu_identity
gains the device ids of the MI300X, MI308X, MI325X and MI355X, read from the
amdgpu.ids that ships with the driver, and two helpers: normalise_device_id
spells an id one way and gpu_type_for_device_id names the one board it belongs
to, or None.

An id that names no single board is left out so that the caller falls back to
the gfx target and the name: 0x74a2 and 0x74b6 are an MI308X and, at revision
A1, an MI300X, and 0x7551, the R9700, is also the R9700S and the R9600D. The
R9700 keeps its dispatch identity and gets no device ids, so it is identified
exactly as before.

The runner's hardware check is the first reader (next commit); nothing on main
reads the new names yet.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…k, tools, service and AgentX client patch

The first half of the measurement runner: the modules that import nothing of
the measurement layer's core. They are taken from multibranch-dev as they are.

- cache_layout and cache_build: the runner cache of one commit (wheelhouse,
  requirements, source) and the builder that fills it on a machine with egress.
  The sealed sandbox mounts it read-only and installs from it with no index.
- bootstrap: the start-up script a sandbox runs; it prepares the runner's
  software from that cache, then replaces itself with the runner command.
- aiperf_patch: patches the AgentX replay client so that a reply it found
  nothing in keeps the output token count the server reported.
- hardware: reads what the machine says about its GPUs (device id, gfx target,
  model, power cap, partition mode, ECC counts, memory in use) and compares it
  with what the instruction sheet says.
- tools: clean checkouts of Magpie and InferenceX at the commits the
  instruction sheet names, fetched by hash into the runner's cache.
- service and live_log: what the runner leaves in /out (progress while it
  works, status when it is done) and the last thing a running test said.

plugs/tests/test_architecture.py: hyperloom.measure leaves _DEV_ONLY, as
hyperloom.sandbox and hyperloom.hosted did; the package's own imports of its
modules failed the guard otherwise. The hosted entries of _HOSTED still keep
the rest of main away from hyperloom.measure.runner.

test_aiperf_patch.py is the file from multibranch-dev without its last test,
the helper that test uses and its two imports: that test reads the patched
client's export through measure/agentx_requests, which comes with the
measurement core.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The sealed runner installs requirements/runner.lock, the new runner extra
compiled with hashes, and never resolves the extra itself. The lock is what
pip-compile made of pyproject.toml inside the framework image, with the image's
own packages as constraints, so the framework stack (torch, triton, sglang,
aiter) is not in it. lm_eval[api] is pinned to the PyPI release of the ref
that preflight pins, because GitHub does not keep an archive's checksum
stable and a hash lock must not depend on one.

test_runner_lock holds the file to what the install relies on: every entry is
an exact pin with hashes, the entries pip-compile marked as asked for by
pyproject.toml are exactly the runner extra, and nothing of the framework
stack is in it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@zoroyihan7 zoroyihan7 added type:feature New capability or improvement size-exception Waives the PR diff budget; added by a maintainer with a reason in the PR description labels Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Code metrics gate: PASSED

Judged: the 26 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.

Dimension Limit New Worse Improved Over the limit vs base
Cyclomatic complexity fails > 20 0 0 0 0 +0
Cognitive complexity fails > 30 0 0 0 33 +0
Function length (lines) fails > 60 0 0 0 131 +0
Nested block depth fails > 4 0 0 0 35 +0
Parameter count fails > 5 0 0 0 0 +0
Module length (lines) warns > 800, fails > 1200 0 0 0 40 +0
Duplicated lines per file fails > 0 0 0 0 76 +0
Dead code (vulture) fails > 0 0 0 0 2 +0
Directory size (entries) fails > 50 0 0 0 1 +0
Check Findings Effect
Comments 0
Broad catch without a reason 0
Broad catch that records nothing 0
English only 0
Production code imports test code 0
Repeated literals 0
A1 Private name of another package 0
A8 Flag argument 0
A9 Tuple return 0
A10 Raw vocabulary comparison 0
A11 Hidden global write 0
A12 Pure forwarder 0
A12 Single-implementation abstraction 0

Verdict computed by the gate scripts in /home/runner/work/_temp/code-metrics-gate, not by this tree's copy.

Thresholds and sources: [tool.hyperloom.code_metrics] in pyproject.toml. Tools: complexipy 8.0.1, jscpd 5.4.1, radon 6.0.1, ruff 0.16.2, vulture 2.16.

…hecks it

CodeQL flags py/side-effect-in-assert: the assert held lines.pop("module"), so
under python -O the line would stay in the dict and the callers, which read
every entry as a scenario, would fail on it. The pop now happens first and the
assert tests the path it returned.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
zoroyihan7 and others added 2 commits October 24, 2026 06:09
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
zoroyihan7 and others added 4 commits October 30, 2026 00:07
Conflicts in env_safety.py, failure_signature.py and provenance.py were
resolved by taking multibranch-dev's version of the six readers, which is
today's main plus only the outside_facts swaps (checked with
git diff origin/main origin/multibranch-dev on each reader).

Co-Authored-By: Claude <noreply@anthropic.com>
Every file of the outside_facts package and its tests now equals
origin/multibranch-dev e81281d byte for byte; the six readers already
did after the merge of main.

Moved: nothing (the package layout outside_facts/{__init__,rule_pack}.py
and packs/default.yaml was already the dev layout).

Added, because rule_pack.py on dev imports them (same package, same
layer, not on main and not in another PR of the stack):
- outside_facts/component_layout.py (each component's layout in the pack)
- outside_facts/image_check_spec.py, image_check_eval.py, image_env.py
  (the image_check entries a pack declares, checked against each entry's
  own examples when the pack loads)
- outside_facts/image_check.py (evaluates those checks against the
  environment it runs in; python -m ... image_check), which completes
  the package and is what test_image_check exercises
- tests: test_component_layout.py, test_image_check.py,
  test_image_check_pack.py

Changed: packs/default.yaml grows with the component layouts and the
image_check entries; rule_pack.py validates them; __init__.py binds the
new constants; test_rule_pack.py follows.

Removed: nothing.

Co-Authored-By: Claude <noreply@anthropic.com>
Conflicts: gpu_identity.py and its test take multibranch-dev's text (now main's
table plus the device-id helpers); test_architecture keeps main's _DEV_ONLY minus
hyperloom.measure.

Co-Authored-By: Claude <noreply@anthropic.com>
Every file this PR carries now matches origin/multibranch-dev e81281d, except
the two test files named below.

Moved to dev's text: aiperf_patch (finds the client submodule by its name),
bootstrap (template step, runner start line), cache_build (build target, pinned
build tools), cache_layout, hardware (released_gpus, AMD_GPU_BOARDS),
gpu_identity and its test (dev is now main's table plus the device-id helpers,
r9700 and radeon8060s included), bootstrap_rig, conftest, test_aiperf_patch,
requirements/runner.lock.

Added (dev tests of this PR's modules): test_bootstrap_entry,
test_bootstrap_template_step, test_cache_build_target.

Removed: runner/service.py, runner/tests/test_service.py and measure/live_log.py.
On dev, service imports measure.boot_watch (#2157) and measure.failures (B3), so
it can no longer land below them; live_log has no other reader in this PR. They
go with the runner core.

Kept off dev: conftest drops the card_watch / sandbox.card_occupancy fixtures
(both are later layers); test_aiperf_patch still leaves out the test that reads
the export through measure.agentx_requests.

Co-Authored-By: Claude <noreply@anthropic.com>
zoroyihan7 and others added 4 commits October 30, 2026 01:11
… and its boot log

Five parameters instead of six: the three directories become the one list the
step makes, and the boot log path, formerly derived from the scratch directory,
is its own input. Brings bootstrap.py and its test up to multibranch-dev d8755f4.

Co-Authored-By: Claude <noreply@anthropic.com>
Main's #2308 moved the shared helpers to the top-level hyperloom_common
package. The rule pack stays in hyperloom.common.outside_facts (only the
optimizer reads it, and hl-common-is-leaf allows hyperloom.common to
import hyperloom_common). Resolve the import conflicts in
gpu_power_settings and provenance onto hyperloom_common, and point
outside_facts.image_check at hyperloom_common.io.

Co-Authored-By: Claude <noreply@anthropic.com>
Brings main's hyperloom_common layout (#2308) in through the rule pack.
gpu_identity now lives in hyperloom_common: the runner's hardware probe
imports it from there, and its test moves to
src/hyperloom_common/tests/test_gpu_identity.py beside the module.

Co-Authored-By: Claude <noreply@anthropic.com>
… what was asked

Same change as multibranch-dev 68e612b (CodeQL: non-standard exception in __getattr__).

Co-Authored-By: Claude <noreply@anthropic.com>
…ce_ids

Replace the module-level AMD_GPU_DEVICE_IDS table with a device_ids field
on GpuBoard, matching #2326; gpu_type_for_device_id and normalise_device_id
keep their signatures. The test that pinned the table's keys to the boards
goes with the table. Explain the empty except in the bootstrap test fixture
(CodeQL).

Co-Authored-By: Claude <noreply@anthropic.com>
zoroyihan7 and others added 7 commits November 2, 2026 03:11
#2326 lands the same device_ids design on main; the two copies now agree,
so whichever merges first leaves the other no conflict.

Co-Authored-By: Claude <noreply@anthropic.com>
…nds each fact to a name

Facts about projects outside this repository (aiter, SGLang, vLLM, PyTorch, the
HIP runtime, argparse, Transformers, InferenceX) that more than one reader
depends on are stated once, in a YAML rule pack. Each entry carries the project
that owns the fact, the reason it exists and its examples; a pattern that fails
one of its own examples, a repeated key, an alias, an unknown kind or an entry
the code reads but the pack lacks refuses the pack, naming the file and the
entry, and nothing falls back to an empty pack.

hyperloom.common.outside_facts binds each entry to the constant the readers
import. HYPERLOOM_RULE_PACK names the pack a session reads; without it the
reference copy beside the module is read. The pack ships as package data.

Nothing reads the constants yet; the next commit moves the readers that already
repeat these facts onto them.

Co-Authored-By: Claude <noreply@anthropic.com>
…n the readers that repeat them

failure_signature, env_safety, provenance, visible_devices, gpu_power_settings
and agentx_workload spelled out facts that the rule pack now owns: the HIP,
HSA, import-error, argparse and Transformers log markers, the ROCR mask name
and the InferenceX benchmark variable names. Each now imports the constant.

No behaviour change. The compiled patterns, the blocked-name set and the
provenance keys are identical to before: every pattern is built from the same
text, compared pattern by pattern.

Co-Authored-By: Claude <noreply@anthropic.com>
The test package's conftest imports hyperloom.common.gpu_power_settings, which
now reads the rule pack, and the pack is YAML. The job installs only pytest
because the lint itself is stdlib-only, so every test in it failed at setup
with "No module named 'yaml'". The job installs PyYAML beside pytest and stays
seconds long.

Co-Authored-By: Claude <noreply@anthropic.com>
Every file of the outside_facts package and its tests now equals
origin/multibranch-dev e81281d byte for byte; the six readers already
did after the merge of main.

Moved: nothing (the package layout outside_facts/{__init__,rule_pack}.py
and packs/default.yaml was already the dev layout).

Added, because rule_pack.py on dev imports them (same package, same
layer, not on main and not in another PR of the stack):
- outside_facts/component_layout.py (each component's layout in the pack)
- outside_facts/image_check_spec.py, image_check_eval.py, image_env.py
  (the image_check entries a pack declares, checked against each entry's
  own examples when the pack loads)
- outside_facts/image_check.py (evaluates those checks against the
  environment it runs in; python -m ... image_check), which completes
  the package and is what test_image_check exercises
- tests: test_component_layout.py, test_image_check.py,
  test_image_check_pack.py

Changed: packs/default.yaml grows with the component layouts and the
image_check entries; rule_pack.py validates them; __init__.py binds the
new constants; test_rule_pack.py follows.

Removed: nothing.

Co-Authored-By: Claude <noreply@anthropic.com>
… what was asked

Same change as multibranch-dev 68e612b (CodeQL: non-standard exception in __getattr__).

Co-Authored-By: Claude <noreply@anthropic.com>
@zoroyihan7
zoroyihan7 force-pushed the feature/yihan/rule-pack branch from d9b121b to b1a96f3 Compare November 2, 2026 04:55
Brings the runner basics onto #2155 after it was rebased onto main.

Co-Authored-By: Claude <noreply@anthropic.com>
Base automatically changed from feature/yihan/rule-pack to main November 2, 2026 08:39
zoroyihan7 and others added 2 commits November 2, 2026 08:49
main squash-merged the base PR (#2155, rule pack and outside_facts) and #2326
(GpuBoard.device_ids, normalise_device_id, gpu_type_for_device_id and their
tests). Take main's version of image_check.py, test_image_check.py and
gpu_identity.py, which this PR does not change on top of main.

Co-Authored-By: Claude <noreply@anthropic.com>
main moved by #2336 (function-length refactor); none of its files is one of this
PR's, and the merge has no conflicts.

Co-Authored-By: Claude <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size-exception Waives the PR diff budget; added by a maintainer with a reason in the PR description type:feature New capability or improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants