Skip to content

feat(contracts): add the measurement request and result contract and the recipe it builds on - #2153

Merged
chaojhou merged 8 commits into
mainfrom
feature/yihan/measurement-contract
Nov 2, 2026
Merged

chaojhou merged 8 commits into
mainfrom
feature/yihan/measurement-contract

Conversation

@zoroyihan7

@zoroyihan7 zoroyihan7 commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds the hyperloom.contracts measurement vocabulary: the types the sandbox layer, the measurement runner and the multi-branch toolbox share. main already has the hyperloom.contracts package (campaign spec and events schemas, #2131); this PR adds modules to it. Commits and owners:

  1. feat(contracts): add the declarative recipe contract, ...: contracts/recipe.py and test_recipe_contract.py. This code is siliang.chen's (the notebook's recipe contract); it is copied as written on multibranch-dev and carries a Co-authored-by trailer. @siliangchen-amd is requested as reviewer for this commit. It is in this PR only because measurement.py imports RecipeDelta and RecipeError from it.
  2. feat(contracts): add the measurement request and result contract with the two JSON Schemas: contracts/measurement.py, measurement_spec.v1.schema.json, measurement_result.v1.schema.json, test_measurement_contract.py, the export lines in contracts/__init__.py and the hyperloom.contracts layer in .importlinter.
  3. A merge of main (the only conflict: contracts/__init__.py, which main added with a docstring; resolved to the multibranch-dev version, which keeps that docstring and adds the exports).
  4. chore(measure): bring the contracts up to multibranch-dev e81281d60: every file brought to multibranch-dev e81281d60, plus the three same-layer modules measurement.py now imports: contracts/_validate.py (shared field validators), contracts/consistency.py (consistency bar of a result) and contracts/long_context.py (long-context bar), with their tests.
  • TestRequest is what a multi-branch module hands to the toolbox; TestResult is the one row it gets back per request, with the tries it needed as Attempt records. Every type is a frozen dataclass that validates on construction and round-trips through to_document() / from_document(), and refuses with a MeasurementError that names the field.
  • The two JSON Schemas describe the documents of the hosted measurement: the spec the application writes into a sandbox (/in/spec.json) and the result manifest the runner signs. They ship as package data through main's existing contracts/*.json entry (no pyproject.toml change is needed any more).
  • hyperloom.contracts joins the hl-top-layers contract between agents and inference_optimizer: it builds on the optimizer's server-argument helpers (through the recipe contract) and nothing below imports it.
  1. fix(contracts): delta_between removes every kernel artifact the other recipe does not keep (review finding): delta_between removed only refs absent from the other recipe, so a ref moved to another slot stayed at its old slot and apply refused the result as duplicate kernel artifacts. It now removes every artifact the other recipe does not keep unchanged; a three-case round-trip test (ref moves slot, ref takes a slot another ref leaves, same slot new content) fails on the old code and passes now. Landed on multibranch-dev first as d2f3a406d.

Every file under src/hyperloom/contracts/ is byte-identical to multibranch-dev d2f3a406d (git diff HEAD origin/multibranch-dev -- src/hyperloom/contracts is empty). The only shared-file edit is the .importlinter layer line. recipe.py imports four functions from hyperloom.inference_optimizer.grid_server_args; all four are on main.

Size against main: 15 files, +5570/-2; 2695 production Python lines (measurement.py 1111, recipe.py 817, long_context.py 327, consistency.py 303, _validate.py 105, __init__.py exports), 1096 lines of JSON Schema, 1777 test lines.

Why now

This is the bottom of the toolbox layer stack (the sandbox PR, the measure leaves #2157 and the runner come after). It is opened now so it can be read; it merges only after the 8-card multibranch acceptance run on the synced head.

CI state

The earlier red on main (the IntegrateRecoveryIncomplete collection error) was fixed by #2149, which this branch now contains. See the checks on the current head.

Review template

  • Linked issue(s): none.
  • Tests: added, taken from multibranch-dev with the code. pytest -n 4 src/hyperloom/contracts/tests src/hyperloom/plugs/tests/test_architecture.py src/hyperloom/inference_optimizer/tests/test_packaging_lint.py: 305 passed on Python 3.10 and on Python 3.11 (144 measurement-contract, 54 recipe-contract, 40 consistency, 19 long-context contract, 23 long-context bar, the 16-test architecture guard and the 9-test packaging lint), with only main plus this PR's files on the path. Also run: ruff check . and ruff format --check on the changed files (clean), lint-imports --no-cache (11 kept, 0 broken).
  • Size/complexity triggers crossed: the diff is 2695 production Python lines against the 1000-line budget, so the PR carries the size-exception label, for this reason: it is one cohesive contract layer taken verbatim (measurement.py 1111 lines and recipe.py 817 lines, each under the 1200-line module limit, over the 800 warning; the three new helper modules exist because measurement.py imports them), and splitting a frozen-dataclass contract between PRs would leave neither half importable.
  • Observable effect: none. Nothing on main imports these modules yet, no CLI, entry point or flag uses them, and the legacy flow behaves exactly as before. The two new JSON Schema files ship as package data.
  • Breaking changes: no.
  • PR addresses single concern: yes (the measurement contracts: the recipe it depends on, the request/result types and their validators).
  • Root cause is upstream: no.

🤖 Generated with Claude Code

zoroyihan7 and others added 2 commits October 23, 2026 19:35
…d how they apply and differ

The recipe contract is the notebook's: a frozen, validated document that names
the image, the source commits, the patches, the kernel artifacts and the server
arguments of one configuration, and the delta that turns one recipe into
another. The measurement contract in the next commit builds on RecipeDelta, so
the two land together; this commit is the recipe half, taken as written on the
multi-branch development branch.

The contracts package starts here with an empty export list; the next commit
adds the measurement names.

Co-authored-by: siliangchen-amd <SiLiang.Chen@amd.com>
Co-Authored-By: Claude <noreply@anthropic.com>
… the two JSON Schemas

The test request a multi-branch module hands to the toolbox and the test result
it gets back, as frozen dataclasses that validate on construction and round-trip
through to_document() and from_document(). The two JSON Schemas describe the
documents of the hosted measurement: the spec the application writes into a
sandbox and the result manifest the runner signs.

The schemas ship as package data. hyperloom.contracts joins the top-level layer
contract between agents and inference_optimizer: it builds on the optimizer's
server-argument helpers (through the recipe contract) and nothing below imports
it.

Nothing on main imports the package yet; it is the vocabulary the sandbox, the
measurement runner and the toolbox share.

Co-Authored-By: Claude <noreply@anthropic.com>
@zoroyihan7 zoroyihan7 added type:feature New capability or improvement size-exception Waives the PR diff budget; added by a maintainer with a reason in the PR description labels Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Code metrics gate: PASSED

Judged: the 15 files this change touches. Files it does not touch never fail it; anything found there is listed under Outside this change for information.

Dimension Limit New Worse Improved Over the limit vs base
Cyclomatic complexity fails > 20 0 0 0 0 +0
Cognitive complexity fails > 30 0 0 0 34 +0
Function length (lines) fails > 60 0 0 0 249 +0
Nested block depth fails > 4 0 0 0 37 +0
Parameter count fails > 5 0 0 0 0 +0
Module length (lines) warns > 800, fails > 1200 0 0 0 42 +0
Duplicated lines per file fails > 0 0 0 0 76 +0
Dead code (vulture) fails > 0 0 0 0 2 +0
Directory size (entries) fails > 50 0 0 0 1 +0
Check Findings Effect
Comments 0
Broad catch without a reason 0
Broad catch that records nothing 0
English only 0
Production code imports test code 0
Repeated literals 0
A1 Private name of another package 0
A8 Flag argument 0
A9 Tuple return 0
A10 Raw vocabulary comparison 0
A11 Hidden global write 0
A12 Pure forwarder 0
A12 Single-implementation abstraction 0

Module length warning: over 800 lines, within the 1200-line failure limit (not failing): 2

Module Lines Limit
src/hyperloom/contracts/measurement.py 1111 warns > 800
src/hyperloom/contracts/recipe.py 815 warns > 800

Verdict computed by the gate scripts in /home/runner/work/_temp/code-metrics-gate, not by this tree's copy.

Thresholds and sources: [tool.hyperloom.code_metrics] in pyproject.toml. Tools: complexipy 8.0.1, jscpd 5.4.1, radon 6.0.1, ruff 0.16.2, vulture 2.16.

zoroyihan7 and others added 4 commits October 24, 2026 06:09
Co-Authored-By: Claude <noreply@anthropic.com>
Conflict in src/hyperloom/contracts/__init__.py (main added the package
with a docstring only): resolved to the multibranch-dev version, which
keeps main's package docstring and adds the measurement exports.

Co-Authored-By: Claude <noreply@anthropic.com>
Updated to multibranch-dev e81281d: contracts/measurement.py, the
measurement spec and result JSON Schemas, test_measurement_contract.py
(recipe.py and test_recipe_contract.py were already identical).

Added, because measurement.py now imports them (same layer, nothing
else imported): contracts/_validate.py, consistency.py, long_context.py
and their tests (test_consistency_contract.py,
test_long_context_contract.py, test_long_context_bar.py,
long_context_checks.py).

Removed: the "contracts/*.schema.json" package-data line in
pyproject.toml; main's "contracts/*.json" already ships the schemas.

Co-Authored-By: Claude <noreply@anthropic.com>
@zoroyihan7
zoroyihan7 marked this pull request as ready for review October 30, 2026 01:29
@zoroyihan7
zoroyihan7 requested a review from a team as a code owner October 30, 2026 01:29

@chaojhou chaojhou left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #2153 -- feat(contracts): add the measurement request and result contract and the recipe it builds on

What it does: adds the measurement vocabulary of the multibranch toolbox to hyperloom.contracts:

  • the declarative recipe contract (recipe.py: Recipe, RecipeDelta, apply, delta_between);
  • the measurement request and result types (measurement.py: TestRequest, TestResult, Attempt and their parts), with shared validators (_validate.py), the consistency bar (consistency.py), the long-context bar (long_context.py), and two JSON Schemas for the sandbox spec and the runner manifest;
  • an .importlinter line placing hyperloom.contracts between agents and inference_optimizer.

Nothing on main imports these yet, and the files match multibranch-dev e81281d.

Risk: high -- .importlinter changes the layer contract
Decisions for a human:

  • .importlinter, src/hyperloom/contracts/ -- a contract layer of 2695 production lines lands with no reader on main (its users arrive in #2326 and #2328), and hyperloom.contracts is placed between agents and inference_optimizer -- accept landing the layer ahead of its users, and its place in the layers
  • src/hyperloom/contracts/ -- the PR body says "Review only -- merges after the multibranch acceptance run (board #121); do not merge before", while the review request asks to merge -- decide when this merges

Blocking issues: 1

  1. [free:delta-between-ref-move] apply(first, delta_between(first, second)) raises for two valid recipes when a kernel ref moves to another slot [verified]
    Problem: delta_between computes remove_kernels as the set of refs in first that are not in second (src/hyperloom/contracts/recipe.py:624-627). A ref that both recipes use is therefore never removed, even when it moves to another (kind, target) slot. _apply_kernels (:596-603) keeps first's artifact at its old slot and adds second's at the new slot. Recipe.__post_init__ (:359) then refuses the result as duplicate refs. Run on the head tree:

    • first = [GEMM X -> kernel/k.csv], second = [GEMM Y -> kernel/k.csv]: the delta is add=[Y], remove=[], and apply raises RecipeError: duplicate kernel artifacts: kernel/k.csv;
    • first = [X -> kernel/a.csv, Z -> kernel/c.csv], second = [X -> kernel/c.csv]: the delta is add=[X:c], remove=[kernel/a.csv], and apply raises duplicate kernel artifacts: kernel/c.csv.

    The docstring of delta_between (:607-613) promises that apply(first, delta) loads second's kernel artifacts.
    Impact: the toolbox cannot build the change that moves one branch's recipe to another whenever a kernel file is retargeted or reused at another slot. Its intended caller on multibranch-dev (warm_start WarmReplay.on, apply(head, delta_between(head, wanted))) raises instead of replaying. Path-like refs filled with different targets are what the multibranch tests use.
    Action: make the delta remove every artifact of first that second does not keep at the same slot with the same content. For example, compute remove_kernels from the artifacts not in second.kernels rather than from refs, or have _apply_kernels drop a kept artifact whose ref an added one takes. Add a round-trip test for both cases. recipe.py is siliang.chen's code; the PR body requests that reviewer for it.

Checked: recipe.py (apply, delta_between, _apply_kernels, _config_delta, component patches, env set/unset, schema_version refusal) and its four grid_server_args imports, which exist on main with the signatures used; measurement.py and _validate.py (construction-time validation, to_document/from_document round trips for every type) against the two schemas, which describe the sandbox spec and the runner manifest rather than these documents and agree on the overlapping fields; consistency.py and long_context.py boundary decisions and their tests; readers on main (none outside three tests that read the existing schema JSONs); the contracts/*.json package-data entry; the byte-identical claim against origin/multibranch-dev; the body; AGENTS.md. Independent readers: four lenses (measurement and schemas; recipe; the two bars; repository contract), with two adversarial verifiers per finding. Four other candidates did not survive both verifiers | Ran: the two delta cases on an extracted head tree; the contracts tests on the extracted tree (277 pass); git merge-tree against main a646a9c (clean); code-metrics PASSED | Skill: 3eec7f38c6e3 | Model: claude-opus-5-5 | Base: a646a9c | Head: 6879545

… recipe does not keep

A ref both recipes load at different slots was never removed, so apply()
kept it at its old slot, added it at the new one, and refused the result
as duplicate kernel artifacts.

Co-Authored-By: Claude <noreply@anthropic.com>
@zoroyihan7

Copy link
Copy Markdown
Collaborator Author

@chaojhou thanks — the blocking finding is fixed in 2122aae (also on multibranch-dev as d2f3a40).

  • delta-between-ref-move: remove_kernels is now every artifact of first that second does not keep unchanged (same ref, slot and content), instead of refs second lacks. New parametrized round-trip test test_delta_between_moves_a_kernel_ref_the_two_recipes_load_at_different_slots covers your two cases plus same-slot-new-content; the first two fail on 6879545 with duplicate kernel artifacts and all pass now. Contracts tests: 280 passed; the warm_start / blackboard callers: 271 passed.
  • When it merges: the "Review only — do not merge before" line was stale; the workstreams agreed each PR merges on its own, so it is removed from the body.
  • Landing the layer ahead of its users and its place in .importlinter: left for the human reviewer as you flagged.

@chaojhou
chaojhou dismissed their stale review November 2, 2026 00:28

Superseded by the re-review at 2122aae: the delta_between finding is fixed.

@chaojhou

chaojhou commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

PR #2153 -- feat(contracts): add the measurement request and result contract and the recipe it builds on

What it does: adds the measurement vocabulary of the multibranch toolbox to hyperloom.contracts:

  • the declarative recipe contract (recipe.py: Recipe, RecipeDelta, apply, delta_between);
  • the measurement request and result types (measurement.py), with shared validators, the consistency bar and the long-context bar;
  • two JSON Schemas for the sandbox spec and the runner manifest;
  • an .importlinter line placing hyperloom.contracts between agents and inference_optimizer.

Nothing on main imports these yet.

Risk: high -- .importlinter changes the layer contract
Decisions for a human:

Blocking issues: none

Checked: [free:delta-between-ref-move] fixed at 2122aae: delta_between now removes every artifact of first that second does not keep unchanged (src/hyperloom/contracts/recipe.py:625), and test_delta_between_moves_a_kernel_ref_the_two_recipes_load_at_different_slots pins it; apply(first, delta_between(first, second)) reproduces second for a retargeted ref, a ref reused at another slot, same slot with new content, two slots swapping refs, and an unchanged recipe; the body no longer says the PR must wait for the acceptance run. The rest of the PR as reviewed at 6879545 holds: the grid_server_args imports exist on main, measurement round trips, the two bars' boundary decisions, package data, no reader on main, files matching multibranch-dev | Ran: the five kernel cases on an extracted head tree; gh pr checks 2153 at 2122aae: 35 pass; git merge-tree against main a646a9c (clean) | Skill: 3eec7f38c6e3 | Model: claude-opus-5-5 | Prior: free:delta-between-ref-move fixed | Base: a646a9c | Head: 2122aae

LGTM

@chaojhou chaojhou left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Clean review at 2122aae; merges clean on latest main. Merging with the maintainer's consent to the decisions on the card.

@chaojhou
chaojhou merged commit 36d0dec into main Nov 2, 2026
42 of 44 checks passed
@chaojhou
chaojhou deleted the feature/yihan/measurement-contract branch November 2, 2026 01:51
zoroyihan7 added a commit that referenced this pull request Nov 2, 2026
The measurement contract (#2153) is on main now; the contracts take main's
version.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size-exception Waives the PR diff budget; added by a maintainer with a reason in the PR description type:feature New capability or improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants