Repository navigation
feat(contracts): add the measurement request and result contract and the recipe it builds on - #2153
Conversation
…d how they apply and differ The recipe contract is the notebook's: a frozen, validated document that names the image, the source commits, the patches, the kernel artifacts and the server arguments of one configuration, and the delta that turns one recipe into another. The measurement contract in the next commit builds on RecipeDelta, so the two land together; this commit is the recipe half, taken as written on the multi-branch development branch. The contracts package starts here with an empty export list; the next commit adds the measurement names. Co-authored-by: siliangchen-amd <SiLiang.Chen@amd.com> Co-Authored-By: Claude <noreply@anthropic.com>
… the two JSON Schemas The test request a multi-branch module hands to the toolbox and the test result it gets back, as frozen dataclasses that validate on construction and round-trip through to_document() and from_document(). The two JSON Schemas describe the documents of the hosted measurement: the spec the application writes into a sandbox and the result manifest the runner signs. The schemas ship as package data. hyperloom.contracts joins the top-level layer contract between agents and inference_optimizer: it builds on the optimizer's server-argument helpers (through the recipe contract) and nothing below imports it. Nothing on main imports the package yet; it is the vocabulary the sandbox, the measurement runner and the toolbox share. Co-Authored-By: Claude <noreply@anthropic.com>
Code metrics gate: PASSEDJudged: the 15 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.
Module length warning: over 800 lines, within the 1200-line failure limit (not failing): 2
Thresholds and sources: |
Co-Authored-By: Claude <noreply@anthropic.com>
Conflict in src/hyperloom/contracts/__init__.py (main added the package with a docstring only): resolved to the multibranch-dev version, which keeps main's package docstring and adds the measurement exports. Co-Authored-By: Claude <noreply@anthropic.com>
Updated to multibranch-dev e81281d: contracts/measurement.py, the measurement spec and result JSON Schemas, test_measurement_contract.py (recipe.py and test_recipe_contract.py were already identical). Added, because measurement.py now imports them (same layer, nothing else imported): contracts/_validate.py, consistency.py, long_context.py and their tests (test_consistency_contract.py, test_long_context_contract.py, test_long_context_bar.py, long_context_checks.py). Removed: the "contracts/*.schema.json" package-data line in pyproject.toml; main's "contracts/*.json" already ships the schemas. Co-Authored-By: Claude <noreply@anthropic.com>
chaojhou
left a comment
There was a problem hiding this comment.
PR #2153 -- feat(contracts): add the measurement request and result contract and the recipe it builds on
What it does: adds the measurement vocabulary of the multibranch toolbox to hyperloom.contracts:
- the declarative recipe contract (
recipe.py:Recipe,RecipeDelta,apply,delta_between); - the measurement request and result types (
measurement.py:TestRequest,TestResult,Attemptand their parts), with shared validators (_validate.py), the consistency bar (consistency.py), the long-context bar (long_context.py), and two JSON Schemas for the sandbox spec and the runner manifest; - an
.importlinterline placinghyperloom.contractsbetweenagentsandinference_optimizer.
Nothing on main imports these yet, and the files match multibranch-dev e81281d.
Risk: high -- .importlinter changes the layer contract
Decisions for a human:
- .importlinter, src/hyperloom/contracts/ -- a contract layer of 2695 production lines lands with no reader on main (its users arrive in #2326 and #2328), and
hyperloom.contractsis placed betweenagentsandinference_optimizer-- accept landing the layer ahead of its users, and its place in the layers - src/hyperloom/contracts/ -- the PR body says "Review only -- merges after the multibranch acceptance run (board #121); do not merge before", while the review request asks to merge -- decide when this merges
Blocking issues: 1
-
[free:delta-between-ref-move]
apply(first, delta_between(first, second))raises for two valid recipes when a kernel ref moves to another slot [verified]
Problem:delta_betweencomputesremove_kernelsas the set of refs infirstthat are not insecond(src/hyperloom/contracts/recipe.py:624-627). A ref that both recipes use is therefore never removed, even when it moves to another(kind, target)slot._apply_kernels(:596-603) keepsfirst's artifact at its old slot and addssecond's at the new slot.Recipe.__post_init__(:359) then refuses the result as duplicate refs. Run on the head tree:first = [GEMM X -> kernel/k.csv],second = [GEMM Y -> kernel/k.csv]: the delta isadd=[Y], remove=[], andapplyraisesRecipeError: duplicate kernel artifacts: kernel/k.csv;first = [X -> kernel/a.csv, Z -> kernel/c.csv],second = [X -> kernel/c.csv]: the delta isadd=[X:c], remove=[kernel/a.csv], andapplyraisesduplicate kernel artifacts: kernel/c.csv.
The docstring of
delta_between(:607-613) promises thatapply(first, delta)loadssecond's kernel artifacts.
Impact: the toolbox cannot build the change that moves one branch's recipe to another whenever a kernel file is retargeted or reused at another slot. Its intended caller onmultibranch-dev(warm_startWarmReplay.on,apply(head, delta_between(head, wanted))) raises instead of replaying. Path-like refs filled with different targets are what the multibranch tests use.
Action: make the delta remove every artifact offirstthatseconddoes not keep at the same slot with the same content. For example, computeremove_kernelsfrom the artifacts not insecond.kernelsrather than from refs, or have_apply_kernelsdrop a kept artifact whose ref an added one takes. Add a round-trip test for both cases.recipe.pyis siliang.chen's code; the PR body requests that reviewer for it.
Checked: recipe.py (apply, delta_between, _apply_kernels, _config_delta, component patches, env set/unset, schema_version refusal) and its four grid_server_args imports, which exist on main with the signatures used; measurement.py and _validate.py (construction-time validation, to_document/from_document round trips for every type) against the two schemas, which describe the sandbox spec and the runner manifest rather than these documents and agree on the overlapping fields; consistency.py and long_context.py boundary decisions and their tests; readers on main (none outside three tests that read the existing schema JSONs); the contracts/*.json package-data entry; the byte-identical claim against origin/multibranch-dev; the body; AGENTS.md. Independent readers: four lenses (measurement and schemas; recipe; the two bars; repository contract), with two adversarial verifiers per finding. Four other candidates did not survive both verifiers | Ran: the two delta cases on an extracted head tree; the contracts tests on the extracted tree (277 pass); git merge-tree against main a646a9c (clean); code-metrics PASSED | Skill: 3eec7f38c6e3 | Model: claude-opus-5-5 | Base: a646a9c | Head: 6879545
… recipe does not keep A ref both recipes load at different slots was never removed, so apply() kept it at its old slot, added it at the new one, and refused the result as duplicate kernel artifacts. Co-Authored-By: Claude <noreply@anthropic.com>
|
@chaojhou thanks — the blocking finding is fixed in 2122aae (also on
|
Superseded by the re-review at 2122aae: the delta_between finding is fixed.
PR #2153 -- feat(contracts): add the measurement request and result contract and the recipe it builds onWhat it does: adds the measurement vocabulary of the multibranch toolbox to
Nothing on main imports these yet. Risk: high -- .importlinter changes the layer contract
Blocking issues: none Checked: [free:delta-between-ref-move] fixed at 2122aae: LGTM |
The measurement contract (#2153) is on main now; the contracts take main's version. Co-Authored-By: Claude <noreply@anthropic.com>
Summary
Adds the
hyperloom.contractsmeasurement vocabulary: the types the sandbox layer, the measurement runner and the multi-branch toolbox share.mainalready has thehyperloom.contractspackage (campaign spec and events schemas, #2131); this PR adds modules to it. Commits and owners:feat(contracts): add the declarative recipe contract, ...:contracts/recipe.pyandtest_recipe_contract.py. This code is siliang.chen's (the notebook's recipe contract); it is copied as written onmultibranch-devand carries aCo-authored-bytrailer. @siliangchen-amd is requested as reviewer for this commit. It is in this PR only becausemeasurement.pyimportsRecipeDeltaandRecipeErrorfrom it.feat(contracts): add the measurement request and result contract with the two JSON Schemas:contracts/measurement.py,measurement_spec.v1.schema.json,measurement_result.v1.schema.json,test_measurement_contract.py, the export lines incontracts/__init__.pyand thehyperloom.contractslayer in.importlinter.main(the only conflict:contracts/__init__.py, whichmainadded with a docstring; resolved to themultibranch-devversion, which keeps that docstring and adds the exports).chore(measure): bring the contracts up to multibranch-dev e81281d60: every file brought tomultibranch-deve81281d60, plus the three same-layer modulesmeasurement.pynow imports:contracts/_validate.py(shared field validators),contracts/consistency.py(consistency bar of a result) andcontracts/long_context.py(long-context bar), with their tests.TestRequestis what a multi-branch module hands to the toolbox;TestResultis the one row it gets back per request, with the tries it needed asAttemptrecords. Every type is a frozen dataclass that validates on construction and round-trips throughto_document()/from_document(), and refuses with aMeasurementErrorthat names the field./in/spec.json) and the result manifest the runner signs. They ship as package data throughmain's existingcontracts/*.jsonentry (nopyproject.tomlchange is needed any more).hyperloom.contractsjoins thehl-top-layerscontract betweenagentsandinference_optimizer: it builds on the optimizer's server-argument helpers (through the recipe contract) and nothing below imports it.fix(contracts): delta_between removes every kernel artifact the other recipe does not keep(review finding):delta_betweenremoved only refs absent from the other recipe, so a ref moved to another slot stayed at its old slot andapplyrefused the result as duplicate kernel artifacts. It now removes every artifact the other recipe does not keep unchanged; a three-case round-trip test (ref moves slot, ref takes a slot another ref leaves, same slot new content) fails on the old code and passes now. Landed onmultibranch-devfirst asd2f3a406d.Every file under
src/hyperloom/contracts/is byte-identical tomultibranch-devd2f3a406d(git diff HEAD origin/multibranch-dev -- src/hyperloom/contractsis empty). The only shared-file edit is the.importlinterlayer line.recipe.pyimports four functions fromhyperloom.inference_optimizer.grid_server_args; all four are onmain.Size against
main: 15 files, +5570/-2; 2695 production Python lines (measurement.py1111,recipe.py817,long_context.py327,consistency.py303,_validate.py105,__init__.pyexports), 1096 lines of JSON Schema, 1777 test lines.Why now
This is the bottom of the toolbox layer stack (the sandbox PR, the measure leaves #2157 and the runner come after). It is opened now so it can be read; it merges only after the 8-card multibranch acceptance run on the synced head.
CI state
The earlier red on
main(theIntegrateRecoveryIncompletecollection error) was fixed by #2149, which this branch now contains. See the checks on the current head.Review template
multibranch-devwith the code.pytest -n 4 src/hyperloom/contracts/tests src/hyperloom/plugs/tests/test_architecture.py src/hyperloom/inference_optimizer/tests/test_packaging_lint.py: 305 passed on Python 3.10 and on Python 3.11 (144 measurement-contract, 54 recipe-contract, 40 consistency, 19 long-context contract, 23 long-context bar, the 16-test architecture guard and the 9-test packaging lint), with onlymainplus this PR's files on the path. Also run:ruff check .andruff format --checkon the changed files (clean),lint-imports --no-cache(11 kept, 0 broken).size-exceptionlabel, for this reason: it is one cohesive contract layer taken verbatim (measurement.py1111 lines andrecipe.py817 lines, each under the 1200-line module limit, over the 800 warning; the three new helper modules exist becausemeasurement.pyimports them), and splitting a frozen-dataclass contract between PRs would leave neither half importable.mainimports these modules yet, no CLI, entry point or flag uses them, and the legacy flow behaves exactly as before. The two new JSON Schema files ship as package data.🤖 Generated with Claude Code