Repository navigation
feat(measure): add the runner's cache, start-up script, hardware check, tools, service and AgentX client patch - #2156
Draft
zoroyihan7 wants to merge 28 commits into
Draft
zoroyihan7 wants to merge 28 commits into
zoroyihan7 wants to merge 28 commits into
Conversation
…nds each fact to a name Facts about projects outside this repository (aiter, SGLang, vLLM, PyTorch, the HIP runtime, argparse, Transformers, InferenceX) that more than one reader depends on are stated once, in a YAML rule pack. Each entry carries the project that owns the fact, the reason it exists and its examples; a pattern that fails one of its own examples, a repeated key, an alias, an unknown kind or an entry the code reads but the pack lacks refuses the pack, naming the file and the entry, and nothing falls back to an empty pack. hyperloom.common.outside_facts binds each entry to the constant the readers import. HYPERLOOM_RULE_PACK names the pack a session reads; without it the reference copy beside the module is read. The pack ships as package data. Nothing reads the constants yet; the next commit moves the readers that already repeat these facts onto them. Co-Authored-By: Claude <noreply@anthropic.com>
…n the readers that repeat them failure_signature, env_safety, provenance, visible_devices, gpu_power_settings and agentx_workload spelled out facts that the rule pack now owns: the HIP, HSA, import-error, argparse and Transformers log markers, the ROCR mask name and the InferenceX benchmark variable names. Each now imports the constant. No behaviour change. The compiled patterns, the blocked-name set and the provenance keys are identical to before: every pattern is built from the same text, compared pattern by pattern. Co-Authored-By: Claude <noreply@anthropic.com>
The test package's conftest imports hyperloom.common.gpu_power_settings, which now reads the rule pack, and the pack is YAML. The job installs only pytest because the lint itself is stdlib-only, so every test in it failed at setup with "No module named 'yaml'". The job installs PyYAML beside pytest and stays seconds long. Co-Authored-By: Claude <noreply@anthropic.com>
A board's name is whatever the VBIOS says (an MI355X node answers "AMD Instinct MI355 OAM"); its PCI device id is what the silicon is. gpu_identity gains the device ids of the MI300X, MI308X, MI325X and MI355X, read from the amdgpu.ids that ships with the driver, and two helpers: normalise_device_id spells an id one way and gpu_type_for_device_id names the one board it belongs to, or None. An id that names no single board is left out so that the caller falls back to the gfx target and the name: 0x74a2 and 0x74b6 are an MI308X and, at revision A1, an MI300X, and 0x7551, the R9700, is also the R9700S and the R9600D. The R9700 keeps its dispatch identity and gets no device ids, so it is identified exactly as before. The runner's hardware check is the first reader (next commit); nothing on main reads the new names yet. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…k, tools, service and AgentX client patch The first half of the measurement runner: the modules that import nothing of the measurement layer's core. They are taken from multibranch-dev as they are. - cache_layout and cache_build: the runner cache of one commit (wheelhouse, requirements, source) and the builder that fills it on a machine with egress. The sealed sandbox mounts it read-only and installs from it with no index. - bootstrap: the start-up script a sandbox runs; it prepares the runner's software from that cache, then replaces itself with the runner command. - aiperf_patch: patches the AgentX replay client so that a reply it found nothing in keeps the output token count the server reported. - hardware: reads what the machine says about its GPUs (device id, gfx target, model, power cap, partition mode, ECC counts, memory in use) and compares it with what the instruction sheet says. - tools: clean checkouts of Magpie and InferenceX at the commits the instruction sheet names, fetched by hash into the runner's cache. - service and live_log: what the runner leaves in /out (progress while it works, status when it is done) and the last thing a running test said. plugs/tests/test_architecture.py: hyperloom.measure leaves _DEV_ONLY, as hyperloom.sandbox and hyperloom.hosted did; the package's own imports of its modules failed the guard otherwise. The hosted entries of _HOSTED still keep the rest of main away from hyperloom.measure.runner. test_aiperf_patch.py is the file from multibranch-dev without its last test, the helper that test uses and its two imports: that test reads the patched client's export through measure/agentx_requests, which comes with the measurement core. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The sealed runner installs requirements/runner.lock, the new runner extra compiled with hashes, and never resolves the extra itself. The lock is what pip-compile made of pyproject.toml inside the framework image, with the image's own packages as constraints, so the framework stack (torch, triton, sglang, aiter) is not in it. lm_eval[api] is pinned to the PyPI release of the ref that preflight pins, because GitHub does not keep an archive's checksum stable and a hash lock must not depend on one. test_runner_lock holds the file to what the install relies on: every entry is an exact pin with hashes, the entries pip-compile marked as asked for by pyproject.toml are exactly the runner extra, and nothing of the framework stack is in it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Code metrics gate: PASSEDJudged: the 26 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.
Thresholds and sources: |
…hecks it
CodeQL flags py/side-effect-in-assert: the assert held lines.pop("module"), so
under python -O the line would stay in the dict and the callers, which read
every entry as a scenario, would fail on it. The pop now happens first and the
assert tests the path it returned.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
zoroyihan7
force-pushed
the
feature/yihan/runner-basics
branch
from
October 23, 2026 20:59
9709175 to
cb0fa0e
Compare
This was referenced Oct 23, 2026
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Conflicts in env_safety.py, failure_signature.py and provenance.py were resolved by taking multibranch-dev's version of the six readers, which is today's main plus only the outside_facts swaps (checked with git diff origin/main origin/multibranch-dev on each reader). Co-Authored-By: Claude <noreply@anthropic.com>
Every file of the outside_facts package and its tests now equals origin/multibranch-dev e81281d byte for byte; the six readers already did after the merge of main. Moved: nothing (the package layout outside_facts/{__init__,rule_pack}.py and packs/default.yaml was already the dev layout). Added, because rule_pack.py on dev imports them (same package, same layer, not on main and not in another PR of the stack): - outside_facts/component_layout.py (each component's layout in the pack) - outside_facts/image_check_spec.py, image_check_eval.py, image_env.py (the image_check entries a pack declares, checked against each entry's own examples when the pack loads) - outside_facts/image_check.py (evaluates those checks against the environment it runs in; python -m ... image_check), which completes the package and is what test_image_check exercises - tests: test_component_layout.py, test_image_check.py, test_image_check_pack.py Changed: packs/default.yaml grows with the component layouts and the image_check entries; rule_pack.py validates them; __init__.py binds the new constants; test_rule_pack.py follows. Removed: nothing. Co-Authored-By: Claude <noreply@anthropic.com>
Conflicts: gpu_identity.py and its test take multibranch-dev's text (now main's table plus the device-id helpers); test_architecture keeps main's _DEV_ONLY minus hyperloom.measure. Co-Authored-By: Claude <noreply@anthropic.com>
Every file this PR carries now matches origin/multibranch-dev e81281d, except the two test files named below. Moved to dev's text: aiperf_patch (finds the client submodule by its name), bootstrap (template step, runner start line), cache_build (build target, pinned build tools), cache_layout, hardware (released_gpus, AMD_GPU_BOARDS), gpu_identity and its test (dev is now main's table plus the device-id helpers, r9700 and radeon8060s included), bootstrap_rig, conftest, test_aiperf_patch, requirements/runner.lock. Added (dev tests of this PR's modules): test_bootstrap_entry, test_bootstrap_template_step, test_cache_build_target. Removed: runner/service.py, runner/tests/test_service.py and measure/live_log.py. On dev, service imports measure.boot_watch (#2157) and measure.failures (B3), so it can no longer land below them; live_log has no other reader in this PR. They go with the runner core. Kept off dev: conftest drops the card_watch / sandbox.card_occupancy fixtures (both are later layers); test_aiperf_patch still leaves out the test that reads the export through measure.agentx_requests. Co-Authored-By: Claude <noreply@anthropic.com>
… and its boot log Five parameters instead of six: the three directories become the one list the step makes, and the boot log path, formerly derived from the scratch directory, is its own input. Brings bootstrap.py and its test up to multibranch-dev d8755f4. Co-Authored-By: Claude <noreply@anthropic.com>
Main's #2308 moved the shared helpers to the top-level hyperloom_common package. The rule pack stays in hyperloom.common.outside_facts (only the optimizer reads it, and hl-common-is-leaf allows hyperloom.common to import hyperloom_common). Resolve the import conflicts in gpu_power_settings and provenance onto hyperloom_common, and point outside_facts.image_check at hyperloom_common.io. Co-Authored-By: Claude <noreply@anthropic.com>
Brings main's hyperloom_common layout (#2308) in through the rule pack. gpu_identity now lives in hyperloom_common: the runner's hardware probe imports it from there, and its test moves to src/hyperloom_common/tests/test_gpu_identity.py beside the module. Co-Authored-By: Claude <noreply@anthropic.com>
… what was asked Same change as multibranch-dev 68e612b (CodeQL: non-standard exception in __getattr__). Co-Authored-By: Claude <noreply@anthropic.com>
…ce_ids Replace the module-level AMD_GPU_DEVICE_IDS table with a device_ids field on GpuBoard, matching #2326; gpu_type_for_device_id and normalise_device_id keep their signatures. The test that pinned the table's keys to the boards goes with the table. Explain the empty except in the bootstrap test fixture (CodeQL). Co-Authored-By: Claude <noreply@anthropic.com>
#2326 lands the same device_ids design on main; the two copies now agree, so whichever merges first leaves the other no conflict. Co-Authored-By: Claude <noreply@anthropic.com>
…nds each fact to a name Facts about projects outside this repository (aiter, SGLang, vLLM, PyTorch, the HIP runtime, argparse, Transformers, InferenceX) that more than one reader depends on are stated once, in a YAML rule pack. Each entry carries the project that owns the fact, the reason it exists and its examples; a pattern that fails one of its own examples, a repeated key, an alias, an unknown kind or an entry the code reads but the pack lacks refuses the pack, naming the file and the entry, and nothing falls back to an empty pack. hyperloom.common.outside_facts binds each entry to the constant the readers import. HYPERLOOM_RULE_PACK names the pack a session reads; without it the reference copy beside the module is read. The pack ships as package data. Nothing reads the constants yet; the next commit moves the readers that already repeat these facts onto them. Co-Authored-By: Claude <noreply@anthropic.com>
…n the readers that repeat them failure_signature, env_safety, provenance, visible_devices, gpu_power_settings and agentx_workload spelled out facts that the rule pack now owns: the HIP, HSA, import-error, argparse and Transformers log markers, the ROCR mask name and the InferenceX benchmark variable names. Each now imports the constant. No behaviour change. The compiled patterns, the blocked-name set and the provenance keys are identical to before: every pattern is built from the same text, compared pattern by pattern. Co-Authored-By: Claude <noreply@anthropic.com>
The test package's conftest imports hyperloom.common.gpu_power_settings, which now reads the rule pack, and the pack is YAML. The job installs only pytest because the lint itself is stdlib-only, so every test in it failed at setup with "No module named 'yaml'". The job installs PyYAML beside pytest and stays seconds long. Co-Authored-By: Claude <noreply@anthropic.com>
Every file of the outside_facts package and its tests now equals origin/multibranch-dev e81281d byte for byte; the six readers already did after the merge of main. Moved: nothing (the package layout outside_facts/{__init__,rule_pack}.py and packs/default.yaml was already the dev layout). Added, because rule_pack.py on dev imports them (same package, same layer, not on main and not in another PR of the stack): - outside_facts/component_layout.py (each component's layout in the pack) - outside_facts/image_check_spec.py, image_check_eval.py, image_env.py (the image_check entries a pack declares, checked against each entry's own examples when the pack loads) - outside_facts/image_check.py (evaluates those checks against the environment it runs in; python -m ... image_check), which completes the package and is what test_image_check exercises - tests: test_component_layout.py, test_image_check.py, test_image_check_pack.py Changed: packs/default.yaml grows with the component layouts and the image_check entries; rule_pack.py validates them; __init__.py binds the new constants; test_rule_pack.py follows. Removed: nothing. Co-Authored-By: Claude <noreply@anthropic.com>
… what was asked Same change as multibranch-dev 68e612b (CodeQL: non-standard exception in __getattr__). Co-Authored-By: Claude <noreply@anthropic.com>
zoroyihan7
force-pushed
the
feature/yihan/rule-pack
branch
from
November 2, 2026 04:55
d9b121b to
b1a96f3
Compare
Brings the runner basics onto #2155 after it was rebased onto main. Co-Authored-By: Claude <noreply@anthropic.com>
main squash-merged the base PR (#2155, rule pack and outside_facts) and #2326 (GpuBoard.device_ids, normalise_device_id, gpu_type_for_device_id and their tests). Take main's version of image_check.py, test_image_check.py and gpu_identity.py, which this PR does not change on top of main. Co-Authored-By: Claude <noreply@anthropic.com>
main moved by #2336 (function-length refactor); none of its files is one of this PR's, and the merge has no conflicts. Co-Authored-By: Claude <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on
main. #2155 (the rule pack) was squash-merged intomainon 2026-11-02, so the base branch of this PR ismainand the diff below is this PR's own.aiperf_patchreads its constants fromhyperloom.common.outside_facts, which #2155 added. Stack: this PR -> #2157 (measure leaves, basefeature/yihan/measure-leaves-base) -> #2158 (measure middle).Summary
Adds the first part of the measurement runner,
hyperloom.measure.runner: the modules that import nothing of the measurement layer's core, so they can be read, tested and merged before it. Refreshed on 2026-10-30 tomultibranch-deve81281d: every file is byte-identical to that commit except the two test files listed under "What differs frommultibranch-dev". The refresh is a merge of the current base (c689388cc) and one sync commit (77ac7c4f9); history is not rewritten.The pieces:
runnerdependency group inpyproject.toml,requirements/runner.lockand the test that holds the two together.hyperloom_common/gpu_identity.pyandhyperloom_common/tests/test_gpu_identity.pyare no longer part of this PR: the device-id helpers that the runner'shardwareimports (GpuBoard.device_ids,normalise_device_id,gpu_type_for_device_id) landed onmainwith #2326.The modules (lines), and everything each imports from this repository:
measure/__init__.py,measure/runner/__init__.pyrunner/cache_layout.pyrunner/cache_build.pycache_layoutrunner/bootstrap.pycache_layoutrunner/aiperf_patch.pycommon.outside_facts(#2155)runner/hardware.pyhyperloom_common.gpu_identity(onmain, #2326),common.rocm_smi(onmain)runner/tools.pyNo module imports
hyperloom.measure.spec,socket,local,manifest,signing,sandboxor any other module of the measurement core.bootstrap's last lineexecspython -m hyperloom.measure.runner run, a command that arrives with the runner core; the bootstrap tests run it against a stand-in runner entry that the test rig writes.Base update on 2026-11-02
#2155 (the rule pack), #2326 (toolbox small edits, including
GpuBoard.device_ids) and #2328 were squash-merged intomainon 2026-11-02, so the base of this PR is nowmain. This branch takes it by two merges oforigin/main:8d43ad49a(main ata026a2e32) anddb0db5345(main atae51de973, #2336, which touches none of this PR's files); history is not rewritten. The first merge conflicted in three files that came from the merged PRs, and each is resolved tomain's version:hyperloom/common/outside_facts/image_check.pyandhyperloom/common/tests/test_image_check.py(from #2155), andhyperloom_common/gpu_identity.py(thedevice_idschange, now onmainfrom #2326). The second merge had no conflicts.Afterwards
git diff origin/main HEADlists 26 files. They are 26 of the 28 files of this PR's diff before the update (git diff b1a96f3cf 24113f390), with the same changed lines in each. The other two,hyperloom_common/gpu_identity.pyandhyperloom_common/tests/test_gpu_identity.py, are identical tomain's, which has the same device-id helpers from #2326, so this PR no longer changes them.Changes in the 2026-10-30 refresh
aiperf_patch,bootstrap,cache_build,cache_layout,hardware,gpu_identity(+ test),bootstrap_rig,conftest,test_aiperf_patch,requirements/runner.lock.tools.pyand the other tests were already identical.test_bootstrap_entry.py,test_bootstrap_template_step.py,test_cache_build_target.py.runner/service.py,runner/tests/test_service.pyandmeasure/live_log.py. On dev,servicenow importsmeasure.boot_watch(feat(measure): add the leaf modules of the measurement core: instruction sheet, result manifest, signing and what they stand on #2157) andmeasure.failures(B3), so it can no longer land below them;live_loghad no other reader here. They go with the runner core (PR-5c), which lands after B3. Nothing in feat(measure): add the leaf modules of the measurement core: instruction sheet, result manifest, signing and what they stand on #2157 or feat(measure): add the middle layer of the measurement core: recipe compile, framework tables, the jit cache and the result archive #2158 imports them.gpu_identity: dev now carriesmain's board table (withr9700andradeon8060s, both listed with no device id, and the 0x7551 sentence), so the earlier hand-applied version is replaced by dev's text. Then, to match feat(executors, common): the toolbox's small edits to existing main modules #2326, the module-levelAMD_GPU_DEVICE_IDStable became adevice_idsfield on eachGpuBoard(same ids, same exclusions;gpu_type_for_device_idandnormalise_device_idkeep their signatures), and the test that pinned the table's keys to the boards went with the table.plugs/tests/test_architecture.pyconflict with the base:mainnow has_DEV_ONLY = ("hyperloom.measure", "hyperloom.sandbox", "hyperloom_multibranch"); this PR takeshyperloom.measureout of it, so the line is_DEV_ONLY = ("hyperloom.sandbox", "hyperloom_multibranch"). Dev's other change to this file (_ON_MAINand the filter in_main_modules) is not taken: this PR does not need it.What differs from
multibranch-devrunner/tests/conftest.py: without dev's_nobody_holds_the_cardsfixture and without the twomonkeypatchlines ofsettle_clockthat patchcard_watchandsandbox.card_occupancy; both modules are later layers (card_watchimportscontracts.measurement,measure.failuresandsandbox). Therigfixture andsettle_clockforhardwareare dev's.runner/tests/test_aiperf_patch.py: without its last test (test_the_reader_counts_empty_and_short_exactly_...), the_classeshelper, the two imports it needs (hyperloom.contracts.measurement.RequestClasses,hyperloom.measure.agentx_requests.read_request_classes) and one clause of the docstring; that test reads the patched client's export throughmeasure/agentx_requests.py, which imports the measurement core, and arrives with it. Dev has since moved thepopout of theassertitself (the CodeQLpy/side-effect-in-assertfix), so that difference is gone.plugs/tests/test_architecture.py: as above.pyproject.toml: only therunnergroup, identical tomultibranch-dev's. Thehostedgroup is not added here: no module in this PR importse2b,boto3orcryptography.cryptographyis inrunneralthough nothing here imports it, because the extra is the input of the lock, and the lock is the whole runner's (signing included);test_runner_lockpins the two together.What is not in this PR, and why
measure/runner(run,jit,inventory,rewrite,rounds,snapshot,platform_env,card_watch,service,native/,__main__, ...) imports the measurement core and lands after it (PR-5c).measure/live_log.py(see above),measure/agentx_requests.pyand the lasttest_aiperf_patchtest.common/aiter_layout.py(new on dev): nothing in this PR imports it; its readers aremeasure/compile,farm(feat(measure): add the middle layer of the measurement core: recipe compile, framework tables, the jit cache and the result archive #2158) and the runner'sjit.spec,manifest,signing,socket,local,compile,framework,jit, ...): feat(measure): add the leaf modules of the measurement core: instruction sheet, result manifest, signing and what they stand on #2157, feat(measure): add the middle layer of the measurement core: recipe compile, framework tables, the jit cache and the result archive #2158 and B3.hostedextra and.importlinterlines:multibranch-devhas none formeasure, so none are added.Size
1731 production Python lines, all in
measure/, against the 1000-line diff budget (size-exceptionreason here): one slice of the runner, six modules and two package docstrings, taken verbatim frommultibranch-dev, none longer than 524 lines (module limit 1200), none depending on anything but its listed imports. If the reviewers prefer, it splits cleanly into the cache trio (cache_layout,cache_build,bootstrap: 1189 lines) and the other three modules (534 lines); the two package docstrings (8 lines) go with either.CI state
At head
db0db5345(2026-11-02): all checks pass (35 pass, 7 skipped, none pending);update test durations (main)anddeploy (github pages)are skipped, as on every PR. 0 unresolved review threads.Review template
multibranch-devwith the code. Run locally on a tree of the base plus this PR only (nosandbox, no othermeasuremodule), Python 3.10 and 3.11,-n 4:pytest src/hyperloom/measure src/hyperloom_common/tests/test_gpu_identity.py src/hyperloom/plugs/tests/test_architecture.py: 176 passed, 1 skipped on both (144 runner tests, of whichtest_runner_lockhas 14; 17gpu_identity; 16 architecture; the skipped one needs a checkout of the real aiperf client,HYPERLOOM_TEST_AIPERF_SOURCE/HYPERLOOM_TEST_AIPERF_PYTHON). The cache tests callpython -m pip, so the interpreter must have pip (22.3 or newer). Both runs repeated at headdb0db5345after the 2026-11-02 base update: 176 passed, 1 skipped on 3.10 and on 3.11. Before that update, at24113f390, the 40 test files that importgpu_identityor walk the source tree and guard packaging and imports (test_packaging_lint,test_llm_client_architecture,test_dedup_guard,test_common_import_lint,plugs,scripts/tests, ...) gave 1180 passed, 7 skipped on both; they were not repeated locally, and the full suite runs in CI at the current head. Also run:ruff check .,ruff format --check .(clean),scripts/check_broad_suppress.py src scripts(rc 0),lint-imports --no-cache(11 kept, 0 broken).mainimportshyperloom.measure, no CLI, entry point or flag uses it, and the legacy flow behaves as before. What exists afterwards: the newrunnerextra,requirements/runner.lock(read only by the sealed runner's install). The runner modules import only the standard library and the modules named in the table.🤖 Generated with Claude Code