diff --git a/CHANGELOG.md b/CHANGELOG.md index 7ce03ea..cf9e351 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,33 @@ All notable changes to this project are documented here, following ## [Unreleased] ### Added +- **The run contract (`loop.yaml`) — a reviewable order ticket for a loop** — run inputs + (target, goal, lane, ten budget knobs) existed only as CLI flags: un-diffable, + unreviewable, and impossible to attach to the proof pack the run produced. Ships + `src/loopeng/contracts/` + `loop-anything run --contract`, `contract check`, and + `contract evidence`. Built on three rules that keep it from becoming decoration: + (1) **it compiles, it doesn't extend** — every key becomes a `config.Budget`/`Lane` the + loop already reads, so there is no new controller state and no knob the engine ignores; + (2) **an unenforced declaration is a parse error, not a no-op** — a typo'd + `max_iteratons:`, a `safety:` block nothing consumes, or maker-authored + `evaluation.dimensions` (maker ≠ checker laundering) all fail the parse, and the error + message names what *is* accepted; (3) **the gate is monotonic** — + `gate.require_human_confirm: false` is rejected, because a caller-authored file must + never hand back the bypass `VerificationGate` deliberately withholds. `evidence.required` + names real `ProofPack` fields and is verified against the recorded run (exit 1 on a gap), + so declaring evidence is a claim the run has to satisfy. Conflicting flags alongside + `--contract` fail closed rather than silently taking precedence. 57 tests in + `tests/test_run_contract.py`; design in `docs/solutions/run-contract.md`; annotated + example (pinned by a test) in `docs/examples/loop.yaml`. +- **`docs/solutions/external-eval-2026-08-12.md` — the repo's answer to an external + architecture review** — an item-by-item verdict, checked against the code rather than + the README: 6 real gaps, 5 partials, and 1 proposal (`artifact_type` as a closed enum) + **rejected as a regression** of the existing `Domain` registry seam, plus a + counter-scorecard that scores human-gate and generality *higher* than the review and + safety, observability, and enterprise-readiness *lower*. Records the meta-finding: the + review's every citation resolves to the README, so "the architecture is too CLI-shaped" + is evidence the **README** is CLI-shaped, not the engine. Names the best idea in the + review (a **false-green rate** benchmark) as the top remaining P0. - **`ToolingSkillFactory` — generate tooling from a repo's KG (U4)** — the tooling half's generator, closing the loop with U3. Given a repo's knowledge graph (`kg-summary.json`), it emits a **skill** — `SKILL.md` + a real, compiling diff --git a/README.md b/README.md index 0b90242..c75593a 100644 --- a/README.md +++ b/README.md @@ -233,6 +233,42 @@ same sanitize-on-write path as recorded learnings (`MemoryStore.record_learning` hostile line in a corpus file can't forge prompt structure — and re-importing the same file inserts nothing. +### 📝 Run it from a contract, not a shell history + +A run used to exist only as CLI flags — unreviewable, un-diffable, and impossible to +attach to the proof pack it produced. Now the ask is a file you commit next to the thing +it converges: + +```bash +loop-anything contract check loop.yaml # validate; print the compiled plan +loop-anything run --contract loop.yaml # run it +loop-anything contract evidence loop.yaml --run 12 # verify it proved what it promised +``` + +```yaml +version: 1 +name: qms-agent-native +target: ./qms-kbp +goal: Make the factory QMS inspection workflow operable by AI agents. +budget: { target_grade: A, max_iterations: 8, plateau_patience: 2, token_budget: 250000 } +evidence: + required: [grade_trajectory, dimension_diff, regression_tests] +``` + +Three rules keep it from becoming decoration +([design](docs/solutions/run-contract.md), [annotated example](docs/examples/loop.yaml)): + +- **It compiles, it doesn't extend.** Every key becomes a `config.Budget` / `Lane` the + loop already reads. No new controller state, no knob the engine ignores. +- **An unenforced declaration is an error, not a no-op.** A typo'd `max_iteratons:` — or a + hopeful `safety:` block nothing consumes — fails the parse. A governance field that + quietly does nothing is the false-green this engine exists to prevent. +- **The gate only tightens.** `require_human_confirm: false` is rejected. The contract is + caller-authored, and a caller can never pre-confirm its own run. + +`evidence.required` is checked against the run's **real** proof pack and exits non-zero on +a gap — so declaring evidence is a claim the run has to satisfy. + --- ## ⚙️ How it works @@ -338,8 +374,9 @@ flowchart TD ``` loop-engineering-anything/ ├── src/loopeng/ -│ ├── cli.py # loop-anything entrypoint (run / preflight / status / report / demo proof) +│ ├── cli.py # loop-anything entrypoint (run / contract / preflight / status / report / demo proof) │ ├── config.py # budgets, convergence knobs, dependency table +│ ├── contracts/ # loop.yaml run contract — parse, compile to Budget/Lane, verify evidence │ ├── preflight.py # per-mechanism dependency detection (+ refine-only gate) │ ├── adopt.py # catalog tool adopter — venv-isolated, env-pruned, full-SHA pin │ ├── proof.py # ProofPack builder + store-backed compounder @@ -354,7 +391,7 @@ loop-engineering-anything/ │ ├── loop/ # controller state machine, convergence, brief, compound, GitCheckpoint │ └── autonomous/ # research report + autonomous runner ├── skills/loop-anything/ # the /loop-anything agent skill -├── tests/ # 286 tests — loop dynamics validated against recorded verdicts +├── tests/ # 600 tests — loop dynamics validated against recorded verdicts └── docs/plans/ # the implementation plan ``` diff --git a/docs/examples/loop.yaml b/docs/examples/loop.yaml new file mode 100644 index 0000000..0d57973 --- /dev/null +++ b/docs/examples/loop.yaml @@ -0,0 +1,41 @@ +# A run contract — the reviewable order ticket for one loop. +# +# loop-anything contract check docs/examples/loop.yaml # validate, print the compiled plan +# loop-anything run --contract docs/examples/loop.yaml # run it +# loop-anything contract evidence docs/examples/loop.yaml --run 12 # verify what it promised +# +# Every key below compiles into a primitive the engine ALREADY enforces +# (config.Budget, config.Lane, the VerificationGate). Anything else is a parse +# error — a contract may not declare what nothing consumes. + +version: 1 +name: qms-agent-native +target: ./qms-kbp +goal: Make the factory QMS inspection workflow operable by AI agents. + +# Optional. Omit to let the router classify the target itself. +lane: codebase + +budget: + target_grade: A # A-F; the letter ladder the judge grades on + max_iterations: 8 # the hard guarantee + plateau_patience: 2 # stop after N iterations with no gain + plateau_pivots: 1 # rotate to the next-lowest dimension before stopping + token_budget: 250000 # enforced only for refiners that report cost + max_wall_seconds: 5400 # the universal cost backstop + min_score_gain: 0.0 # noise band; set from `loop-anything judge-variance` + +gate: + # May be omitted or true. `false` is rejected: the contract is caller-authored, + # and a caller can never pre-confirm its own run (anti-surrender). + require_human_confirm: true + +evidence: + # Checked against the run's real proof pack by `contract evidence`. A pack omits + # (never fakes) a field it has no source for, so a missing item is a real gap. + required: + - grade_trajectory # before_grade -> after_grade + - dimension_diff # per-dimension before/after/delta + - iterations + - convergence_status + - regression_tests # the tests /ce-compound recorded on accepted fixes diff --git a/docs/reportcards/collection.json b/docs/reportcards/collection.json index bd4b6c9..090ab51 100644 --- a/docs/reportcards/collection.json +++ b/docs/reportcards/collection.json @@ -82,5 +82,89 @@ "evidence": "scorecard output" } ] + }, + { + "id": "rc0004", + "delivery": "Verified the external review against the codebase (not the README, which is what it read). Shipped src/loopeng/contracts/ + 'run --contract' / 'contract check' / 'contract evidence' (items 1+7): a loop.yaml that compiles into config.Budget/Lane, rejects any key nothing enforces (parse error, not no-op), and makes the human gate monotonic (require_human_confirm:false is rejected). evidence.required is verified against the real ProofPack, exit 1 on a gap. Rejected item 2 (artifact_type as a closed Literal) as a regression of the existing open Domain registry, and rejected the section-20 directory rename. Recorded docs/solutions/external-eval-2026-08-12.md with an item-by-item verdict and a counter-scorecard. Suite 543 -> 600 passing, ruff clean.", + "objective": "Make loop-engineering-anything's public architecture as honest and reviewable as its engine already is", + "created_at": "2026-08-12T14:11:12-05:00", + "key_results": [ + { + "text": "External review adjudicated item-by-item against code", + "target": "12/12 items ruled", + "score": 1.0, + "evidence": "docs/solutions/external-eval-2026-08-12.md" + }, + { + "text": "Run inputs become a reviewable, fail-closed artifact", + "target": "run --contract ships with tests", + "score": 1.0, + "evidence": "tests/test_run_contract.py, 57 tests" + }, + { + "text": "No regression to the engine", + "target": "full suite green", + "score": 1.0, + "evidence": "600 passed, 2 skipped; ruff clean" + } + ], + "request": "Eval this post [external 8.6/10 architecture review of loop-engineering-anything] based on deep research, align and upgrade loop-engineering-anything code base.", + "headline": "Review answered; loop.yaml contract shipped", + "brief": "Someone graded the repo 8.6/10 and listed 12 upgrades. I checked each one against the actual code instead of the README. Six were real gaps, five were half-built already, and one would have made the repo worse. I built the biggest real one: you can now write the ask for a run in a file you commit, and the file is checked so it can never quietly promise something the engine does not do.", + "growth": [ + "C:met:Checked every claim against src/ before accepting it; caught that the review's central premise came from README-only sourcing", + "P:met:Derived the contract's three rules from the repo's existing invariants (KTD1, maker!=checker, anti-surrender) rather than copying the reviewer's schema" + ], + "why": "The review's headline claim (too CLI-shaped) is false about the engine and true about the README; acting on it literally would have replaced an open registry with a closed enum. Answering it with code evidence now prevents a 23-item backlog from being executed on a wrong premise.", + "needle": { + "name": "tests passing", + "before": "543", + "after": "600", + "better": "up", + "source": "pytest -q" + }, + "next_lever": "Item 9: a false-green rate benchmark \u2014 how often the loop says A when a stronger evaluator disagrees" + }, + { + "id": "rc0005", + "delivery": "Long-form article 'An AI Reviewed My Repo and Told Me to Make It Worse' at public/articles/ai-review-told-me-to-make-it-worse.html, per docs/ARTICLE_AUTHORING.md end to end: cover 1200x627 + 2-panel infographic (house style, rasterized, visually inspected, every count counted in the raster), byline, back-link, reader kit, rk-summary, related, engage.js, portfolio.yaml registration, promo kit (LinkedIn + 8-tweet thread), translation deferral recorded (machine-gated), CHANGELOG. Three external quotes verified verbatim against primary sources (Kubernetes blog, RFC 9413 raw text, arXiv 2404.13076); W3C DNT added as the mechanism's largest real instance. 5-S gate run twice with reviewer != maker: pass 1 FAIL at S2 3/5 (3 blockers), pass 2 PASS (S1 5, S2 4, S3 5, S5 4). Also corrected a 599->600 off-by-one the reviewer found in the engine repo's eval doc. npm test and npm run build both exit 0. Committed on branch article/ai-review-told-me-to-make-it-worse; NOT pushed.", + "objective": "Make loop-engineering-anything's public architecture as honest and reviewable as its engine already is", + "created_at": "2026-08-12T16:24:05-05:00", + "key_results": [ + { + "text": "The upgrade is published as long-form, playbook-complete", + "target": "every DoD item in ARTICLE_AUTHORING.md", + "score": 1.0, + "evidence": "npm test + npm run build exit 0, incl. test-translation-coverage 0 silent gaps" + }, + { + "text": "Nothing ships that an independent reviewer can falsify", + "target": "5-S gate passed with reviewer != maker", + "score": 1.0, + "evidence": "content/promos/ai-review-told-me-to-make-it-worse.scorecard.md, 2 passes" + }, + { + "text": "Claims about my own code are true", + "target": "every number reconciled against source", + "score": 1.0, + "evidence": "reviewer re-ran the CLI and the test suite; 3 blockers fixed" + } + ], + "request": "write a long form article about this upgrade and publish it to my portfolio writtings, all according to our playbooks", + "headline": "Article shipped; the gate failed it first", + "brief": "I wrote the long-form piece about the loop.yaml upgrade and put it in your portfolio. Your own publishing playbook made me have an independent reviewer grade it before shipping. It failed the first time \u2014 it caught me claiming something about my own code that was not true, a source listed but never used, and three numbers that did not add up. I fixed those, it passed the second round, and I fixed two more things it found even after passing.", + "growth": [ + "C:met:Ran the playbook's reviewer!=maker gate instead of self-certifying, and published the FAIL in the scorecard rather than only the PASS", + "P:met:When the reviewer found a contract key that compiles to nothing at runtime, disclosed it in the article as a violation of my own Rule 2 rather than defending it" + ], + "why": "The engine's whole thesis is that a declaration nobody enforces is a costume. Publishing an article about that thesis without running my own publishing gate would have been the same failure one level up \u2014 and the gate immediately proved it by catching a false claim about my own CLI flags.", + "needle": { + "name": "5-S mandatory dimensions at or above 4/5", + "before": "3 of 4 (S2 failed at 3)", + "after": "4 of 4", + "better": "up", + "source": "content/promos/ai-review-told-me-to-make-it-worse.scorecard.md" + }, + "next_lever": "Item 9 from the review: a false-green rate benchmark \u2014 how often the loop reports A when a stronger judge disagrees" } ] \ No newline at end of file diff --git a/docs/solutions/external-eval-2026-08-12.md b/docs/solutions/external-eval-2026-08-12.md new file mode 100644 index 0000000..c01386b --- /dev/null +++ b/docs/solutions/external-eval-2026-08-12.md @@ -0,0 +1,185 @@ +# Eval of an external architecture review (2026-08-12) + +An external reviewer scored this repo **8.6/10** and proposed 12 upgrades + a 23-item +backlog, headlined by: *"the architecture is too CLI-shaped; add `artifact_type` and +refactor around GoalContract → Artifact → Evaluator → Refiner → Gate → EvidencePack."* + +This document is the repo's answer, checked against the **code** rather than the README. + +--- + +## The meta-finding: it is a README review, not a code review + +Every citation in the review resolves to `[1] — the GitHub README`. That is not a +disqualifier — the README is the public contract, and a reviewer reading it and +concluding "this is CLI-shaped" is *evidence the README is CLI-shaped*. But it means +the review cannot see the seams that already exist, and three of its twelve proposals +would remove one. + +So the review splits cleanly: + +| Verdict | Items | Meaning | +|---|---|---| +| **Real gap** | 1, 5, 6, 7, 8, 9 | Genuinely absent. Worth building. | +| **Partial** | 3, 4, 10, 11, 12 | The mechanism exists; the surface or taxonomy is thin. | +| **Already exists** | 2 | Shipped as the `Domain` seam; the proposal would *regress* it. | +| **Reject** | §20 | A big-bang rename of 9k LOC / 600 tests with zero behavior change. | + +--- + +## Item-by-item + +### 1. `GoalContract` — **REAL GAP → shipped this turn** + +Correct. Run inputs existed only as CLI flags: unreviewable, un-diffable, impossible to +attach to the proof pack the run produces. + +Shipped as `src/loopeng/contracts/run_contract.py` — but deliberately **narrower** than +proposed. See [run-contract.md](run-contract.md) for the three rules. The short version: +the proposed schema contains fields nothing would consume (`safety_policy`, +`evaluation_adapter`, per-dimension weights). A declared-but-unenforced field is exactly +the false-green this engine exists to prevent, so the contract **rejects** them at parse +time instead of accepting them decoratively. + +### 2. "Split judge into evaluator types" — **ALREADY EXISTS; the proposal regresses it** + +The review proposes `Evaluator(Protocol)` + a table mapping artifact → evaluator. That +protocol has been in the tree since plan-004: + +- `adapters/base.py` — `Judge` / `Factory` / `Refiner` / `Compounder` / `Checkpoint` protocols +- `domains/base.py` — `Domain` binds *classify → factory → judge* for a target shape +- `domains/registry.py` — `DomainRegistry`, first-match-wins, "a new domain arrives as a + registration, never as an edit to `router.py` or the controller" +- Concrete referees already registered: `adapters/judge.py` (CLI-Judge), + `adapters/spec_judge.py`, `adapters/tooling_judge.py`, + `domains/physical_ai/sim_judge.py` + +The proposed `artifact_type: Literal["cli", "mcp_server", "agent_skill", …]` is a +**closed enum in a central contract file**. Adding an artifact type would then require +editing that file — re-centralizing exactly what the open registry decentralized. The +generalization axis here is *registration*, not *enumeration*. + +**Accepted correction:** the README does not say this, which is why a careful reader +concluded the opposite. That is a README defect, not an architecture defect. + +### 3. Typed recovery taxonomy — **PARTIAL, worth finishing** + +Already typed: `loop/convergence.py` emits machine-readable `reason_code` — +`PLATEAU` / `ITERATION_CAP` / `TOKEN_CAP` / `WALL_CAP` — plus terminal `BLOCKED_SAFETY`. +Transient-infra recovery is real too: `Refiner.last_infra_failure` + `Budget.max_tool_retries` +retries *only* the infra class, never a clean no-change or a post-judge safety failure. + +Genuinely missing from the taxonomy: `referee_unavailable` (must fail closed), +`adapter_contract`, `human_gate_timeout`. Small, additive, worth doing. + +### 4. `EvidencePack` — **PARTIAL; the honest half already works** + +`proof.ProofPack.from_run` builds `before_grade` / `after_grade` / `dim_diff` / +`iterations` / `convergence_status` / `elapsed_seconds` / `token_cost` / +`regression_tests` from a real store run, and **omits rather than fakes** any field it +has no source for. + +The proposed schema adds fields with no producer (`safety_events`, +`artifact_before/after` snapshots). One is closer than the review knew: +`MemoryStore.record_confirmation` already persists human decisions. + +Shipped this turn instead of a bigger schema: `evidence.required` in a contract is +**checked against the real pack** (`loop-anything contract evidence --run N`, +exit 1 on a gap). A promise now has to be met by a run. + +### 5. Risk-tiered human gates — **REAL GAP, but the proposed shape is unsafe** + +The tiering idea is right. The proposed YAML is not: it makes +`require_confirm_before_ship` a caller-settable boolean, which means a caller can author +`false`. `config.VerificationGate` deliberately has **no caller-settable bypass** — only a +CI-infrastructure env var, and never for `--scheduled` runs. + +The contract shipped here therefore makes the gate **monotonic**: `require_human_confirm` +may be omitted or `true`; `false` is a parse error. Tiering should land the same way — +tiers may only *add* reviewers, never remove the base confirmation. Deferred until there +is a real risk signal to classify on; a tier table with no classifier is a knob that only +ever lowers the default. + +### 6. Loop-readiness score — **REAL GAP, recommended next** + +Nothing grades whether a target is *loopable* before the loop starts. `preflight.py` +detects the four external **tools**; it says nothing about the target's testability, +rollback surface, or machine-readable success criteria. Highest-value remaining item for +adoption, and it composes with the contract shipped here (`readiness --contract`). + +### 7. `loop.yaml` — **REAL GAP → shipped this turn** (same unit as #1) + +### 8. MCP lane — **REAL GAP, and cheap** + +Correct, and cheaper than the review thinks: it is a `Domain` registration + an +`mcp_contract` judge, with no core change (see #2). This is where "anything" genuinely +extends — via the registry, not an enum. + +### 9. Benchmark mode + **false-green rate** — **REAL GAP; the best idea in the review** + +The single most valuable proposal. "How often does the loop say A when a stronger +evaluator disagrees" is the trust metric this repo lacks. The seed measurement exists — +`probe_grade_variance` / `loop-anything judge-variance` measures referee stability on an +unchanged tool — but nothing yet compares a converged verdict against a stronger judge. + +### 10. Ablation as a first-class command — **PARTIAL** + +`flywheel/ablation.py` exists and already enforces the honest discipline (an ablation is +`live_verified` only when both legs carry real run ids). What is missing is the CLI verb +and the `--without-rollback` / `--without-safety-gate` legs. Note the last one is +delicate: ablating the safety gate must never be runnable against a real target. + +### 11. Lane maturity badges — **MOSTLY EXISTS** + +`demos/result.py` carries `source: live_verified | illustrative`, `cli.py` has a single +write path to `live_verified` (only a real recorded run flips it), and +`showcase/generate.py` renders "illustrative — not a verified run". The proposed 5-badge +ladder (`adapter_ready`, `beta`, `research_only`) is a refinement of a discipline already +in place. + +### 12. Positioning — **PARTIAL** + +`docs/solutions/integrate-loop-engineering.md` already frames outer-loop vs inner-loop. +Adding LangGraph and Codex to that table is a straight improvement. + +### §20 proposed directory tree — **REJECT** + +A wholesale rename of a 9k-LOC, 600-test engine with no behavior change, on a repo whose +own fitness functions (`tests/test_architecture.py`) pin the current import arrows. The +correct move is the opposite: **make the README describe the architecture that exists** +(the `Domain` seam), and add lanes by registration. + +--- + +## Counter-scorecard (code-based, not README-based) + +| Dimension | Review | Here | Why the difference | +|---|--:|--:|---| +| Goal clarity | 8.5 | **8.5 → 9.0** | Agreed; closed this turn by the run contract. | +| Independent evaluation | 9.5 | **9.5** | Agreed. `loop/integrity.py` enforces maker ≠ checker *and* oracle ≠ checker at runner entry. | +| Safety gate | 9.0 | **8.5** | *Lower.* Terminal `BLOCKED_SAFETY` is right, but `gate.yaml`'s denylist is a declared policy, not an enforced one on every write path. | +| Recovery | 8.5 | **8.5** | Agreed — reason codes exist, the taxonomy is incomplete (#3). | +| Human gate | 8.0 | **9.0** | *Higher.* The review missed that there is deliberately no caller-settable bypass, and that `--scheduled --confirm` is rejected. | +| Generality | 7.0 | **8.5** | *Higher.* The `Domain` registry is the generality seam; what is missing is registered lanes, not architecture. | +| Observability | 8.0 | **7.5** | *Lower.* Proof packs are strong; there is no trace schema and no cross-run metric surface (#9). | +| Enterprise readiness | 7.8 | **7.0** | *Lower.* No tenancy, no RBAC, no secrets story, and the false-green rate is unmeasured. | + +**Net: the review's headline is wrong (the core is not CLI-shaped), its top-6 backlog is +about half real, and its best idea (#9, false-green rate) it ranks P1 rather than P0.** + +--- + +## What this turn shipped + +`run --contract loop.yaml` + `contract check` + `contract evidence` — items 1 and 7, +built so they cannot make items 2 and 5 worse. + +## Recommended order for the rest + +1. **#9 false-green benchmark** (P0 — it is the trust metric, and everything else is + easier to trust once it exists) +2. **#6 loop readiness** (adoption; composes with the contract) +3. **#8 MCP lane** (proves "anything" via registration, ~1 domain + 1 judge) +4. **#3 failure taxonomy** + **#10 ablate CLI** (small, additive) +5. **README rewrite** to describe the `Domain` seam — the fix for the meta-finding +6. **#5 risk tiers** — only after a real risk classifier exists diff --git a/docs/solutions/run-contract.md b/docs/solutions/run-contract.md new file mode 100644 index 0000000..9a54834 --- /dev/null +++ b/docs/solutions/run-contract.md @@ -0,0 +1,95 @@ +# The run contract (`loop.yaml`) + +**Problem.** Every input to a run — target, goal, lane, and ten budget knobs — existed +only as CLI flags. That makes a run unreviewable (no diff, no PR), unreproducible (the +invocation lives in someone's shell history), and impossible to attach to the proof pack +the run produces. A converged Grade A with no record of *what was asked for* is half an +artifact. + +**Solution.** A contract file that is parsed, validated, and **compiled into primitives +the engine already enforces**. + +```bash +loop-anything contract check docs/examples/loop.yaml # validate; print the compiled plan +loop-anything run --contract docs/examples/loop.yaml # run it +loop-anything contract evidence docs/examples/loop.yaml --run 12 # verify what it promised +``` + +See [`docs/examples/loop.yaml`](../examples/loop.yaml) for the annotated example (pinned +by a test, so it cannot rot). + +--- + +## Three rules that keep it from becoming decoration + +### 1. Compile, don't extend + +A contract compiles into `config.Budget` and `config.Lane`. It adds no controller state +(KTD1) and cannot introduce a knob the loop does not read. `_execute_run` receives a +`Budget` — the same object the flag path builds — so there is exactly one code path. + +### 2. Fail closed on anything unenforced + +Unknown keys are a **parse error**, not a silent no-op: + +``` +loop.yaml.budget: unknown key(s) ['max_iteratons']; accepted: [...]. +A contract may only declare what the engine enforces. +``` + +This is the rule that separates this from a config file. A typo'd `max_iteratons:`, or a +hopeful `safety: {forbidden: [delete_production_data]}` block, must not read as +"configured" when nothing consumes it — that is the false-green this engine exists to +prevent, dressed as governance. + +Deliberately **not accepted**, for that reason: + +| Rejected key | Why | +|---|---| +| `domain` | The `run` path routes by lane; `route()` has no forced-domain seam yet. Accepting it would silently do nothing. | +| `safety.forbidden` / `safety.require_human_approval` | Enforcement lives in `adapters/safety.py` (workspace jail, metachar rejection, `shell=False`) and `gate.yaml`. A second, unwired declaration site would be worse than none. | +| `evaluation.dimensions` / weights | The referee owns its rubric. A maker-authored file steering the grading is maker ≠ checker laundering. | +| `human_gate.require_two_person_review` | Not implemented (see the external eval, item 5). | + +Each becomes accept-able the moment something enforces it — the error message is the +to-do list. + +### 3. The gate is monotonic + +`gate.require_human_confirm` may be omitted or `true`. **`false` is a parse error.** + +A contract is caller-authored. `config.VerificationGate` deliberately exposes no +caller-settable bypass — only a CI-infrastructure env var, and never for `--scheduled` +runs — precisely so a scheduler cannot pre-confirm its own work. A contract that could +set `false` would hand that bypass straight back. A contract may only ever *tighten*. + +--- + +## Evidence is a claim, not a promise + +`evidence.required` names fields of the real proof pack (`proof.ProofPack.from_run`). +Only names a pack can actually carry are valid; anything else is a parse error. Then: + +```bash +$ loop-anything contract evidence loop.yaml --run 12 + present grade_trajectory + MISSING regression_tests +Error: run #12 is missing declared evidence: regression_tests +``` + +Exit code 1. A proof pack *omits* (never fakes) a field it has no source for, so a +missing item is a real gap between what the contract promised and what the run proved — +e.g. a converged run where `/ce-compound` recorded no regression test. + +## Conflicts fail closed + +Passing `--goal`, `--lane`, `--max-iterations`, or a positional `TARGET` alongside +`--contract` is an error rather than a silent precedence rule. Otherwise the committed +file would no longer describe the run it produced — which defeats the entire point of +having the file. + +## Provenance + +Shipped 2026-08-12 in response to an external architecture review +([external-eval-2026-08-12.md](external-eval-2026-08-12.md), items 1 and 7), built +narrower than proposed so it could not weaken items 2 and 5. diff --git a/skills/loop-anything/SKILL.md b/skills/loop-anything/SKILL.md index 2ef3421..ae0765e 100644 --- a/skills/loop-anything/SKILL.md +++ b/skills/loop-anything/SKILL.md @@ -95,6 +95,22 @@ lane is auto-classified; `--lane` forces it. `judge-variance` re-judges an unchanged tool K times to measure grader stability and recommend a `min_score_gain` threshold (rule 2 above). +### Run contract — the reviewable order ticket + +``` +loop-anything run --contract loop.yaml [--refiner ...] [--judge-adapter PATH] +loop-anything contract check [--json] # validate; print the compiled plan +loop-anything contract evidence --run # exit 1 if a declared item is missing +``` + +A contract supplies `target` / `goal` / `lane` / `budget` from a committed file, so a +run is diff-able and reviewable. Passing `--goal`, `--lane`, `--max-iterations`, or a +positional target *alongside* `--contract` is an error — the file must keep describing +the run it produced. Two rules to relay when a user hits them: an unknown key is a +**parse error** (a contract may only declare what the engine enforces), and +`gate.require_human_confirm: false` is **rejected** (a caller can never pre-confirm its +own run). Design: `docs/solutions/run-contract.md`; example: `docs/examples/loop.yaml`. + ### Fleet — dependency-ordered multi-target ``` diff --git a/src/loopeng/cli.py b/src/loopeng/cli.py index 2d23bdf..d906bbc 100644 --- a/src/loopeng/cli.py +++ b/src/loopeng/cli.py @@ -3,6 +3,7 @@ Subcommands: preflight Detect the four external dependencies. run Route a target, generate the tool, and drive a real refine loop. + contract Validate a run contract (loop.yaml) and verify its declared evidence. status Show recorded runs from the memory store. report Render the research report for a run. fleet Coordinate a fleet of self-improving loops. @@ -69,7 +70,7 @@ def preflight_cmd(as_json: bool, lane: str | None) -> None: def _execute_run( target: str, goal: str, lane: str | None, judge_adapter: str | None, judge_registry: str | None, refiner_kind: str, workspace: str, confirm: bool, - scheduled: bool, max_iterations: int | None, *, store, echo, + scheduled: bool, max_iterations: int | None, *, store, echo, budget=None, ): """Orchestrate route -> generate -> resolve -> deps -> refine loop (plan 2026-06-22 U3). @@ -115,7 +116,9 @@ def _execute_run( err=True, ) - config = Config() + # A contract-supplied Budget replaces the default wholesale (it was validated at + # parse time); --max-iterations still layers on top for the flag-driven path. + config = Config(budget=budget) if budget is not None else Config() if max_iterations is not None: config = dataclasses.replace( config, budget=dataclasses.replace(config.budget, max_iterations=max_iterations) @@ -132,8 +135,10 @@ def _execute_run( @main.command("run") -@click.argument("target") -@click.option("--goal", required=True, help="High-level goal for the loop.") +@click.argument("target", required=False) +@click.option("--goal", default=None, help="High-level goal for the loop.") +@click.option("--contract", "contract_path", default=None, + help="Run contract (loop.yaml) supplying target, goal, lane, and budget.") @click.option( "--lane", type=click.Choice([lane.value for lane in Lane]), @@ -152,11 +157,15 @@ def _execute_run( @click.option("--scheduled", is_flag=True, help="Mark an unattended run (gate stays confirm-required).") @click.option("--max-iterations", type=int, default=None, help="Override the loop's max iterations.") def run_cmd( - target: str, goal: str, lane: str | None, judge_adapter: str | None, - judge_registry: str | None, refiner_kind: str, workspace: str, confirm: bool, - scheduled: bool, max_iterations: int | None, + target: str | None, goal: str | None, contract_path: str | None, lane: str | None, + judge_adapter: str | None, judge_registry: str | None, refiner_kind: str, workspace: str, + confirm: bool, scheduled: bool, max_iterations: int | None, ) -> None: - """Route TARGET, generate the tool, and drive a real refine loop to Grade A.""" + """Route TARGET, generate the tool, and drive a real refine loop to Grade A. + + Supply TARGET/--goal directly, or ``--contract loop.yaml`` to run from a + reviewed, version-controlled contract. + """ # Anti-surrender: a scheduled (unattended) run cannot be pre-confirmed from the # CLI -- confirmation must come from a human after the run (R5). if scheduled and confirm: @@ -169,10 +178,32 @@ def run_cmd( from .loop.controller import LoopState from .memory.store import MemoryStore + budget = None + contract = None + if contract_path: + contract = _load_contract_or_fail(contract_path) + # Fail closed on a split source of truth: a flag that also appears in the + # contract would leave the file no longer describing the run it produced. + conflicts = [ + name for name, value in + (("TARGET", target), ("--goal", goal), ("--lane", lane), ("--max-iterations", max_iterations)) + if value is not None + ] + if conflicts: + raise click.ClickException( + f"--contract supplies {', '.join(conflicts)}; drop the flag(s) or edit {contract_path}." + ) + target, goal, budget = contract.target, contract.goal, contract.budget + lane = contract.lane.value if contract.lane else None + click.echo(f"Contract: {contract.name or contract.target} ({contract_path})") + elif not target or not goal: + raise click.ClickException("provide TARGET and --goal, or --contract .") + try: result, refiner_used = _execute_run( target, goal, lane, judge_adapter, judge_registry, refiner_kind, workspace, confirm, scheduled, max_iterations, store=MemoryStore.default(), echo=click.echo, + budget=budget, ) except (ValueError, JudgeAdapterError, RuntimeError) as e: # domain errors -> actionable message raise click.ClickException(str(e)) @@ -189,6 +220,79 @@ def run_cmd( + (f" -- gate: {result.gate_reason}" if result.gate_reason else "") ) click.echo(f"Inspect: loopeng report {result.run_id}") + if contract is not None and contract.evidence_required: + click.echo( + f"Evidence declared ({', '.join(contract.evidence_required)}); verify with: " + f"loop-anything contract evidence {contract_path} --run {result.run_id}" + ) + + +def _load_contract_or_fail(path: str): + """Load a run contract, mapping a ``ContractError`` to an actionable CLI error.""" + from .contracts import ContractError, load_contract + + try: + return load_contract(path) + except ContractError as exc: + raise click.ClickException(str(exc)) from exc + + +@main.group("contract") +def contract_grp() -> None: + """Validate and verify run contracts (loop.yaml).""" + + +@contract_grp.command("check") +@click.argument("path") +@click.option("--json", "as_json", is_flag=True, help="Emit the compiled plan as JSON.") +def contract_check_cmd(path: str, as_json: bool) -> None: + """Validate PATH and print the plan it compiles to (no run, no side effects).""" + from .contracts import describe + + plan = describe(_load_contract_or_fail(path)) + if as_json: + click.echo(json.dumps(plan, indent=2, sort_keys=True)) + return + click.echo(f"OK {plan['name']} ({path})") + click.echo(f" target: {plan['target']}") + click.echo(f" goal: {plan['goal']}") + click.echo(f" lane: {plan['lane'] or 'auto (routed)'}") + for key, value in plan["budget"].items(): + click.echo(f" budget.{key}: {value if value is not None else '-'}") + click.echo(f" require_human_confirm: {plan['require_human_confirm']}") + click.echo(f" evidence_required: {', '.join(plan['evidence_required']) or '-'}") + + +@contract_grp.command("evidence") +@click.argument("path") +@click.option("--run", "run_id", type=int, required=True, help="Run whose proof pack to verify.") +def contract_evidence_cmd(path: str, run_id: int) -> None: + """Verify RUN's proof pack carries every evidence item PATH declares. + + Exits non-zero when a declared item is missing -- the declaration is a claim + the run has to satisfy, not a promise in a file. + """ + from .contracts import missing_evidence + from .memory.store import MemoryStore + from .proof import ProofPack + + contract = _load_contract_or_fail(path) + if not contract.evidence_required: + click.echo(f"{path} declares no evidence; nothing to verify.") + return + try: + pack = ProofPack.from_run(MemoryStore.default(), run_id) + except ValueError as exc: + raise click.ClickException(str(exc)) from exc + + missing = missing_evidence(contract, pack) + for name in contract.evidence_required: + click.echo(f" {'MISSING' if name in missing else 'present'} {name}") + if missing: + raise click.ClickException( + f"run #{run_id} is missing declared evidence: {', '.join(missing)}" + ) + click.echo(f"OK run #{run_id} carries all {len(contract.evidence_required)} declared evidence item(s).") @main.command("judge-variance") diff --git a/src/loopeng/contracts/__init__.py b/src/loopeng/contracts/__init__.py new file mode 100644 index 0000000..af60ab0 --- /dev/null +++ b/src/loopeng/contracts/__init__.py @@ -0,0 +1,20 @@ +"""Declarative, reviewable inputs to the loop (the `loop.yaml` order ticket). + +A contract is *parsed and compiled* here into the primitives the engine already +enforces (``config.Budget``, ``config.Lane``, the ``VerificationGate``). It adds +no new controller state and no new runtime behavior: anything a contract cannot +be compiled into is rejected at parse time rather than silently ignored. +""" + +from __future__ import annotations + +from .run_contract import ( # noqa: F401 + CONTRACT_VERSION, + EVIDENCE_FIELDS, + ContractError, + RunContract, + describe, + load_contract, + missing_evidence, + parse_contract, +) diff --git a/src/loopeng/contracts/run_contract.py b/src/loopeng/contracts/run_contract.py new file mode 100644 index 0000000..582d7ed --- /dev/null +++ b/src/loopeng/contracts/run_contract.py @@ -0,0 +1,306 @@ +"""The run contract — a reviewable order ticket for one loop (plan 2026-08-12 U1). + +Before this module the inputs to a run (target, goal, lane, every budget knob) +existed only as CLI flags: unreviewable, un-diffable, and impossible to attach to +the proof pack a run produces. A contract makes them a file that lives in the +repo next to the thing it converges. + +Three rules make it honest rather than decorative: + +1. **Compile, don't extend.** A contract is compiled into ``config.Budget`` / + ``config.Lane`` — primitives the controller already enforces. It adds no + controller state (KTD1) and cannot introduce a knob the loop does not read. +2. **Fail closed on anything unenforced.** Unknown keys are an error, not a + silent no-op. A typo'd ``max_iteratons:`` or a hopeful ``safety: {forbidden: + [...]}`` block MUST NOT read as "configured" when nothing consumes it — a + declaration the engine ignores is exactly the false-green this repo exists to + prevent. Notably absent for that reason: ``domain`` (the ``run`` path routes + by lane and has no forced-domain seam yet) and per-dimension evaluation + weights (the referee owns its rubric — maker ≠ checker). +3. **The gate is monotonic.** ``require_human_confirm`` may be omitted or set + ``true``; ``false`` is rejected. A contract is caller-authored, and the + anti-surrender rule (``config.VerificationGate``) is that a caller can never + disable its own confirmation. A contract may only ever *tighten*. + +``evidence.required`` is checked against a real proof pack by +``missing_evidence`` — so declaring evidence is a claim the run must satisfy, +not a promise in a file. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from ..config import Budget, Lane +from ..grades import GRADE_RANK + +CONTRACT_VERSION = 1 + +_TOP_KEYS = frozenset({"version", "name", "target", "goal", "lane", "budget", "gate", "evidence"}) +_GATE_KEYS = frozenset({"require_human_confirm"}) +_EVIDENCE_KEYS = frozenset({"required"}) + +# Evidence name -> the ``proof.ProofPack`` keys that make it real. Only fields a +# real pack actually carries are nameable; an unlisted name is a parse error, so +# a contract can never require evidence the engine has no way to produce. +EVIDENCE_FIELDS: dict[str, tuple[str, ...]] = { + "grade_trajectory": ("before_grade", "after_grade"), + "dimension_diff": ("dim_diff",), + "iterations": ("iterations",), + "convergence_status": ("convergence_status",), + "elapsed": ("elapsed_seconds",), + "token_cost": ("token_cost",), + "regression_tests": ("regression_tests",), +} + +# Budget knobs, each mapped to how it is validated. ``kind`` is the python type; +# ``minimum`` is an inclusive floor; ``optional`` fields accept an explicit null. +_BUDGET_NUMERIC: dict[str, tuple[type, float, bool]] = { + # key: (kind, minimum, optional) + "target_score": (float, 0.0, True), + "max_iterations": (int, 1, False), + "plateau_patience": (int, 1, False), + "plateau_pivots": (int, 0, False), + "token_budget": (int, 1, True), + "max_wall_seconds": (float, 1.0, True), + "max_tool_retries": (int, 0, False), + "compression_interval": (int, 1, False), + "min_score_gain": (float, 0.0, False), +} +_BUDGET_KEYS = frozenset({"target_grade", *_BUDGET_NUMERIC}) + + +class ContractError(ValueError): + """A malformed run contract (unknown key, bad type, or a loosened gate).""" + + +@dataclass(frozen=True) +class RunContract: + """A parsed, compiled contract. Every field is something the engine enforces.""" + + target: str + goal: str + name: str = "" + lane: Lane | None = None + budget: Budget = Budget() + # Always True. Present as a field (rather than implied) so the compiled plan + # a reviewer reads states the gate explicitly; see rule 3 above. + require_human_confirm: bool = True + evidence_required: tuple[str, ...] = () + source: str = "" + + +# ----- primitive validators ------------------------------------------------ + + +def _fail(where: str, msg: str) -> None: + raise ContractError(f"{where}: {msg}") + + +def _mapping(value, where: str) -> dict: + if value is None: + return {} + if not isinstance(value, dict): + _fail(where, f"must be a mapping, got {type(value).__name__}") + return value + + +def _reject_unknown(data: dict, allowed: frozenset[str], where: str) -> None: + unknown = sorted(set(data) - allowed) + if unknown: + _fail( + where, + f"unknown key(s) {unknown}; accepted: {sorted(allowed)}. " + "A contract may only declare what the engine enforces.", + ) + + +def _string(data: dict, key: str, where: str, *, required: bool = False, default: str = "") -> str: + if key not in data or data[key] is None: + if required: + _fail(where, f"'{key}' is required") + return default + value = data[key] + if not isinstance(value, str) or not value.strip(): + _fail(where, f"'{key}' must be a non-empty string, got {value!r}") + return value.strip() + + +def _number(data: dict, key: str, where: str, default): + """Validate one numeric budget knob against ``_BUDGET_NUMERIC``.""" + if key not in data: + return default + kind, minimum, optional = _BUDGET_NUMERIC[key] + value = data[key] + if value is None: + if not optional: + _fail(where, f"'{key}' may not be null") + return None + # bool is an int subclass; a YAML `true` here is a typo, not a number. + if isinstance(value, bool) or not isinstance(value, (int, float)): + _fail(where, f"'{key}' must be a number, got {value!r}") + if kind is int and not isinstance(value, int): + _fail(where, f"'{key}' must be a whole number, got {value!r}") + if value < minimum: + _fail(where, f"'{key}' must be >= {minimum}, got {value!r}") + return kind(value) + + +# ----- section parsers ----------------------------------------------------- + + +def _parse_budget(raw, where: str) -> Budget: + data = _mapping(raw, where) + _reject_unknown(data, _BUDGET_KEYS, where) + defaults = Budget() + + grade = _string(data, "target_grade", where, default=defaults.target_grade).upper() + if grade not in GRADE_RANK: + _fail(where, f"'target_grade' must be one of {sorted(GRADE_RANK)}, got {grade!r}") + + knobs = {k: _number(data, k, where, getattr(defaults, k)) for k in _BUDGET_NUMERIC} + return Budget(target_grade=grade, **knobs) + + +def _parse_gate(raw, where: str) -> bool: + data = _mapping(raw, where) + _reject_unknown(data, _GATE_KEYS, where) + value = data.get("require_human_confirm", True) + if value is not True: + _fail( + where, + "'require_human_confirm' may only be true. A contract is caller-authored and " + "may only tighten the human gate, never disable it (anti-surrender, " + "config.VerificationGate).", + ) + return True + + +def _parse_evidence(raw, where: str) -> tuple[str, ...]: + data = _mapping(raw, where) + _reject_unknown(data, _EVIDENCE_KEYS, where) + required = data.get("required", []) + if not isinstance(required, list) or not all(isinstance(x, str) for x in required): + _fail(where, f"'required' must be a list of strings, got {required!r}") + unknown = sorted({x for x in required} - set(EVIDENCE_FIELDS)) + if unknown: + _fail( + where, + f"unknown evidence {unknown}; a proof pack can carry {sorted(EVIDENCE_FIELDS)}", + ) + # Order-preserving dedupe so the compiled plan reads as authored. + seen: list[str] = [] + for name in required: + if name not in seen: + seen.append(name) + return tuple(seen) + + +def _parse_lane(data: dict, where: str) -> Lane | None: + raw = _string(data, "lane", where) + if not raw: + return None + try: + return Lane(raw) + except ValueError: + _fail(where, f"'lane' must be one of {[ln.value for ln in Lane]}, got {raw!r}") + return None # pragma: no cover - _fail always raises + + +# ----- entry points -------------------------------------------------------- + + +def parse_contract(data, *, source: str = "contract") -> RunContract: + """Validate a contract mapping and compile it into a ``RunContract``. + + Raises ``ContractError`` — with the offending key named — on anything the + engine would not enforce. + """ + root = _mapping(data, source) + if not root: + _fail(source, "contract is empty") + _reject_unknown(root, _TOP_KEYS, source) + + version = root.get("version") + if version != CONTRACT_VERSION: + _fail(source, f"'version' must be {CONTRACT_VERSION}, got {version!r}") + + target = _string(root, "target", source, required=True) + goal = _string(root, "goal", source, required=True) + return RunContract( + target=target, + goal=goal, + name=_string(root, "name", source, default=""), + lane=_parse_lane(root, source), + budget=_parse_budget(root.get("budget"), f"{source}.budget"), + require_human_confirm=_parse_gate(root.get("gate"), f"{source}.gate"), + evidence_required=_parse_evidence(root.get("evidence"), f"{source}.evidence"), + source=source, + ) + + +def load_contract(path: str) -> RunContract: + """Read a ``.yaml``/``.yml``/``.json`` contract from ``path`` and parse it.""" + import json + import pathlib + + p = pathlib.Path(path) + suffix = p.suffix.lower() + if suffix not in {".yaml", ".yml", ".json"}: + raise ContractError(f"{path}: contract must be a .yaml, .yml, or .json file") + try: + text = p.read_text(encoding="utf-8") + except OSError as exc: + raise ContractError(f"{path}: cannot read contract ({exc})") from exc + + try: + if suffix == ".json": + data = json.loads(text) + else: + import yaml + + data = yaml.safe_load(text) + except Exception as exc: # yaml.YAMLError is not a ValueError -- catch broadly, report precisely + raise ContractError(f"{path}: could not parse ({exc})") from exc + + return parse_contract(data, source=path) + + +def missing_evidence(contract: RunContract, pack: dict) -> list[str]: + """Evidence names the contract requires that ``pack`` does not carry. + + A proof pack omits (never fakes) a field it has no source for, so absence + here is a real, reportable gap between what was promised and what was proved. + """ + missing = [] + for name in contract.evidence_required: + keys = EVIDENCE_FIELDS[name] + if any(pack.get(k) in (None, "", [], {}) for k in keys): + missing.append(name) + return missing + + +def describe(contract: RunContract) -> dict: + """The compiled plan, as a reviewer (or `--json`) sees it.""" + b = contract.budget + return { + "name": contract.name or contract.target, + "target": contract.target, + "goal": contract.goal, + "lane": contract.lane.value if contract.lane else None, + "budget": { + "target_grade": b.target_grade, + "target_score": b.target_score, + "max_iterations": b.max_iterations, + "plateau_patience": b.plateau_patience, + "plateau_pivots": b.plateau_pivots, + "token_budget": b.token_budget, + "max_wall_seconds": b.max_wall_seconds, + "max_tool_retries": b.max_tool_retries, + "compression_interval": b.compression_interval, + "min_score_gain": b.min_score_gain, + }, + "require_human_confirm": contract.require_human_confirm, + "evidence_required": list(contract.evidence_required), + "source": contract.source, + } diff --git a/tests/test_run_contract.py b/tests/test_run_contract.py new file mode 100644 index 0000000..7dcdb7d --- /dev/null +++ b/tests/test_run_contract.py @@ -0,0 +1,427 @@ +"""Run-contract tests (plan 2026-08-12 U1). + +The contract's whole value is that it is *fail-closed*: it may only declare what +the engine enforces, and it may only tighten the human gate. These pin both, plus +the compile path into ``config.Budget`` and the evidence check against a real +proof pack. +""" + +from __future__ import annotations + +import json + +import pytest +from click.testing import CliRunner + +from loopeng.cli import main +from loopeng.config import Budget, Lane +from loopeng.contracts import ( + ContractError, + RunContract, + describe, + load_contract, + missing_evidence, + parse_contract, +) + +MINIMAL = {"version": 1, "target": "./repo", "goal": "make it agent-native"} + + +def _write(tmp_path, text: str, name: str = "loop.yaml"): + p = tmp_path / name + p.write_text(text, encoding="utf-8") + return str(p) + + +# ----- parsing: the happy path compiles into engine primitives ------------- + + +def test_minimal_contract_compiles_to_engine_defaults(): + c = parse_contract(MINIMAL) + assert (c.target, c.goal) == ("./repo", "make it agent-native") + assert c.lane is None # unset -> the router still classifies + assert c.budget == Budget() # no knob is invented by the contract layer + assert c.evidence_required == () + + +def test_budget_block_compiles_every_knob_the_loop_reads(): + c = parse_contract({ + **MINIMAL, + "budget": { + "target_grade": "b", "max_iterations": 8, "plateau_patience": 2, + "plateau_pivots": 0, "token_budget": 250000, "max_wall_seconds": 5400, + "max_tool_retries": 1, "compression_interval": 3, "min_score_gain": 0.5, + "target_score": 0.9, + }, + }) + assert c.budget == Budget( + target_grade="B", target_score=0.9, max_iterations=8, plateau_patience=2, + plateau_pivots=0, token_budget=250000, max_wall_seconds=5400.0, + max_tool_retries=1, compression_interval=3, min_score_gain=0.5, + ) + + +def test_lane_compiles_to_the_lane_enum(): + assert parse_contract({**MINIMAL, "lane": "service"}).lane is Lane.SERVICE + + +def test_optional_budget_knobs_accept_explicit_null(): + c = parse_contract({**MINIMAL, "budget": {"token_budget": None, "max_wall_seconds": None}}) + assert c.budget.token_budget is None and c.budget.max_wall_seconds is None + + +# ----- fail-closed: an unenforced declaration is an ERROR, never a no-op ---- + + +@pytest.mark.parametrize( + "data", + [ + {**MINIMAL, "safety": {"forbidden": ["delete_production_data"]}}, # nothing consumes it + {**MINIMAL, "domain": "software-codebase"}, # the run path cannot force a domain + {**MINIMAL, "budget": {"max_iteratons": 8}}, # typo + {**MINIMAL, "evaluation": {"dimensions": ["safety"]}}, # the referee owns its rubric + {**MINIMAL, "gate": {"require_two_person_review": True}}, # not implemented + ], +) +def test_unknown_keys_are_rejected_not_silently_ignored(data): + with pytest.raises(ContractError) as exc: + parse_contract(data) + assert "unknown key" in str(exc.value) + + +def test_error_names_the_offending_section_and_source(): + with pytest.raises(ContractError, match=r"loop\.yaml\.budget:.*max_iteratons"): + parse_contract({**MINIMAL, "budget": {"max_iteratons": 8}}, source="loop.yaml") + + +@pytest.mark.parametrize( + "data,fragment", + [ + ({"target": "./r", "goal": "g"}, "'version' must be 1"), + ({"version": 2, "target": "./r", "goal": "g"}, "'version' must be 1"), + ({"version": 1, "goal": "g"}, "'target' is required"), + ({"version": 1, "target": "./r"}, "'goal' is required"), + ({"version": 1, "target": " ", "goal": "g"}, "non-empty string"), + ({**MINIMAL, "lane": "cli"}, "'lane' must be one of"), + ({**MINIMAL, "budget": {"target_grade": "Z"}}, "'target_grade' must be one of"), + ({**MINIMAL, "budget": {"max_iterations": 0}}, "must be >= 1"), + ({**MINIMAL, "budget": {"max_iterations": "eight"}}, "must be a number"), + ({**MINIMAL, "budget": {"max_iterations": True}}, "must be a number"), + ({**MINIMAL, "budget": {"max_iterations": 2.5}}, "must be a whole number"), + ({**MINIMAL, "budget": {"max_iterations": None}}, "may not be null"), + ({**MINIMAL, "evidence": {"required": ["vibes"]}}, "unknown evidence"), + ({**MINIMAL, "evidence": {"required": "grade_trajectory"}}, "must be a list"), + ({}, "contract is empty"), + ([1, 2], "must be a mapping"), + ], +) +def test_malformed_contracts_are_rejected(data, fragment): + with pytest.raises(ContractError) as exc: + parse_contract(data) + assert fragment in str(exc.value) + + +# ----- the gate is monotonic: a contract may only tighten ------------------ + + +def test_gate_defaults_to_confirm_required(): + assert parse_contract(MINIMAL).require_human_confirm is True + + +def test_gate_may_be_restated_true(): + assert parse_contract({**MINIMAL, "gate": {"require_human_confirm": True}}).require_human_confirm + + +@pytest.mark.parametrize("value", [False, "false", 0, None]) +def test_contract_can_never_disable_the_human_gate(value): + """Anti-surrender: the caller authors the contract, so it must not be able to + pre-confirm its own run (config.VerificationGate).""" + with pytest.raises(ContractError, match="may only be true"): + parse_contract({**MINIMAL, "gate": {"require_human_confirm": value}}) + + +# ----- evidence is a claim checked against a real proof pack --------------- + + +def test_evidence_names_are_deduped_in_authored_order(): + c = parse_contract({**MINIMAL, "evidence": {"required": ["iterations", "grade_trajectory", "iterations"]}}) + assert c.evidence_required == ("iterations", "grade_trajectory") + + +def test_missing_evidence_reports_what_the_pack_lacks(): + c = parse_contract({**MINIMAL, "evidence": {"required": ["grade_trajectory", "regression_tests", "token_cost"]}}) + pack = {"before_grade": "F", "after_grade": "A", "iterations": 3, "regression_tests": ["t.py"]} + assert missing_evidence(c, pack) == ["token_cost"] # a pack omits, never fakes, a field + + +def test_no_declared_evidence_is_vacuously_satisfied(): + assert missing_evidence(parse_contract(MINIMAL), {}) == [] + + +# ----- loading from disk --------------------------------------------------- + + +def test_load_yaml_contract(tmp_path): + path = _write(tmp_path, """ +version: 1 +name: qms-agent-native +target: ./qms-kbp +goal: Make the QMS operable by agents. +lane: codebase +budget: + target_grade: A + max_iterations: 8 +evidence: + required: + - grade_trajectory + - regression_tests +""") + c = load_contract(path) + assert c.name == "qms-agent-native" and c.lane is Lane.CODEBASE + assert c.budget.max_iterations == 8 + assert c.evidence_required == ("grade_trajectory", "regression_tests") + assert c.source == path # errors point at the file, not "contract" + + +def test_load_json_contract(tmp_path): + path = _write(tmp_path, json.dumps(MINIMAL), name="loop.json") + assert load_contract(path).target == "./repo" + + +def test_load_rejects_unsupported_extension(tmp_path): + with pytest.raises(ContractError, match="must be a .yaml"): + load_contract(_write(tmp_path, "version: 1", name="loop.txt")) + + +def test_load_reports_unreadable_file(tmp_path): + with pytest.raises(ContractError, match="cannot read contract"): + load_contract(str(tmp_path / "nope.yaml")) + + +def test_load_reports_malformed_yaml(tmp_path): + with pytest.raises(ContractError, match="could not parse"): + load_contract(_write(tmp_path, "version: 1\n bad: [indent\n")) + + +def test_describe_is_json_serializable(): + plan = describe(parse_contract({**MINIMAL, "lane": "codebase"})) + assert json.loads(json.dumps(plan))["lane"] == "codebase" + assert plan["require_human_confirm"] is True + + +def test_contract_is_a_frozen_value_object(): + with pytest.raises(Exception): + RunContract(target="a", goal="b").target = "c" # type: ignore[misc] + + +# ----- CLI surface --------------------------------------------------------- + + +def test_contract_check_prints_the_compiled_plan(tmp_path): + path = _write(tmp_path, "version: 1\ntarget: ./r\ngoal: g\nbudget:\n max_iterations: 4\n") + res = CliRunner().invoke(main, ["contract", "check", path]) + assert res.exit_code == 0, res.output + assert "budget.max_iterations: 4" in res.output + assert "require_human_confirm: True" in res.output + + +def test_contract_check_json_is_machine_readable(tmp_path): + path = _write(tmp_path, "version: 1\ntarget: ./r\ngoal: g\n") + res = CliRunner().invoke(main, ["contract", "check", path, "--json"]) + assert res.exit_code == 0, res.output + assert json.loads(res.output)["target"] == "./r" + + +def test_contract_check_fails_loudly_on_a_bad_contract(tmp_path): + path = _write(tmp_path, "version: 1\ntarget: ./r\ngoal: g\nsafety:\n forbidden: [rm]\n") + res = CliRunner().invoke(main, ["contract", "check", path]) + assert res.exit_code != 0 + assert "unknown key" in res.output + + +# ----- `run --contract` wiring (plan 2026-08-12 U1) ------------------------ +# +# Reuses the `wired` fixture's stubbing idiom from test_run_cli.py so the wiring +# is proven without touching a real factory, judge, or refiner. + + +@pytest.fixture +def wired(tmp_path, monkeypatch): + from loopeng.adapters.base import GenerateResult + from loopeng.autonomous.runner import RunResult + from loopeng.loop.controller import LoopOutcome, LoopState + from loopeng.memory.store import MemoryStore + + tool = tmp_path / "ws" + tool.mkdir() + adapter = tmp_path / "adapters" / "x.py" + adapter.parent.mkdir() + adapter.write_text("x\n") + repo = tmp_path / "repo" + repo.mkdir() + captured: dict = {} + + class FakeFactory: + def generate(self, target, goal, workdir): + captured["target"] = target + captured["gen_goal"] = goal + return GenerateResult(tool_path=str(tool), lane="codebase", ok=True, manifest={}) + + def fake_refine(tool_path, goal, **kw): + captured["goal"] = goal + captured.update(kw) + return RunResult( + run_id=7, + outcome=LoopOutcome(LoopState.CONVERGED, "A", "ok", 2, score=0.0, dims={}), + shippable=True, + ) + + monkeypatch.setattr("loopeng.cli.missing_for_lane", lambda lane: []) + monkeypatch.setattr( + "loopeng.autonomous.runner._default_factories", + lambda: {"cli-anything": FakeFactory(), "printing-press": FakeFactory()}, + ) + monkeypatch.setattr("loopeng.autonomous.runner.run_refine_loop", fake_refine) + monkeypatch.setattr(MemoryStore, "default", classmethod(lambda cls: MemoryStore(tmp_path / "db.sqlite"))) + return {"repo": repo, "adapter": adapter, "captured": captured} + + +def test_run_from_contract_drives_the_loop_with_the_compiled_budget(wired, tmp_path): + path = _write(tmp_path, f""" +version: 1 +name: demo +target: {wired["repo"]} +goal: make it agent-native +budget: + target_grade: B + max_iterations: 3 +""") + res = CliRunner().invoke( + main, ["run", "--contract", path, "--judge-adapter", str(wired["adapter"])] + ) + assert res.exit_code == 0, res.output + cap = wired["captured"] + assert cap["goal"] == "make it agent-native" + assert cap["config"].budget.max_iterations == 3 + assert cap["config"].budget.target_grade == "B" + assert "Contract: demo" in res.output + + +def test_run_without_target_or_contract_is_actionable(wired): + res = CliRunner().invoke(main, ["run"]) + assert res.exit_code != 0 + assert "--contract" in res.output + + +@pytest.mark.parametrize( + "extra", [["--goal", "other"], ["--lane", "service"], ["--max-iterations", "9"]] +) +def test_contract_and_conflicting_flag_fail_closed(wired, tmp_path, extra): + """A flag that also lives in the contract would leave the file no longer + describing the run it produced -- reject rather than silently pick a winner.""" + path = _write(tmp_path, f"version: 1\ntarget: {wired['repo']}\ngoal: g\n") + res = CliRunner().invoke( + main, ["run", "--contract", path, "--judge-adapter", str(wired["adapter"]), *extra] + ) + assert res.exit_code != 0 + assert "drop the flag" in res.output + assert "goal" not in wired["captured"] # the loop never started + + +def test_positional_target_still_conflicts_with_a_contract(wired, tmp_path): + path = _write(tmp_path, f"version: 1\ntarget: {wired['repo']}\ngoal: g\n") + res = CliRunner().invoke(main, ["run", str(wired["repo"]), "--contract", path]) + assert res.exit_code != 0 + assert "TARGET" in res.output + + +def test_run_reminds_the_operator_to_verify_declared_evidence(wired, tmp_path): + path = _write(tmp_path, f""" +version: 1 +target: {wired["repo"]} +goal: g +evidence: + required: [grade_trajectory] +""") + res = CliRunner().invoke( + main, ["run", "--contract", path, "--judge-adapter", str(wired["adapter"])] + ) + assert res.exit_code == 0, res.output + assert "contract evidence" in res.output and "--run 7" in res.output + + +# ----- `contract evidence` verifies against a real recorded run ------------ + + +@pytest.fixture +def store(tmp_path, monkeypatch): + from loopeng.memory.store import MemoryStore + + s = MemoryStore(tmp_path / "ev.db") + monkeypatch.setattr(MemoryStore, "default", classmethod(lambda cls: s)) + yield s + s.close() + + +def _record_run(store, *, with_learning: bool): + run_id = store.create_run("./repo", "codebase", "g", "2026-08-12T10:00:00") + store.record_iteration(run_id, 1, "F", {"correctness": 10}, True, score=0.1) + store.record_iteration(run_id, 2, "A", {"correctness": 40}, True, score=0.9) + if with_learning: + store.record_learning(run_id, None, "fixed the json contract", "tests/test_x.py") + store.finish_run(run_id, "converged", "A") + return run_id + + +def test_evidence_passes_when_the_pack_carries_every_declared_item(store, tmp_path): + run_id = _record_run(store, with_learning=True) + path = _write(tmp_path, """ +version: 1 +target: ./repo +goal: g +evidence: + required: [grade_trajectory, dimension_diff, regression_tests] +""") + res = CliRunner().invoke(main, ["contract", "evidence", path, "--run", str(run_id)]) + assert res.exit_code == 0, res.output + assert "carries all 3" in res.output + + +def test_evidence_fails_when_a_declared_item_is_absent(store, tmp_path): + run_id = _record_run(store, with_learning=False) # no /ce-compound learning -> no regression tests + path = _write(tmp_path, """ +version: 1 +target: ./repo +goal: g +evidence: + required: [grade_trajectory, regression_tests] +""") + res = CliRunner().invoke(main, ["contract", "evidence", path, "--run", str(run_id)]) + assert res.exit_code != 0 + assert "MISSING regression_tests" in res.output + assert "present grade_trajectory" in res.output + + +def test_evidence_on_an_unknown_run_is_actionable(store, tmp_path): + path = _write(tmp_path, "version: 1\ntarget: ./r\ngoal: g\nevidence:\n required: [iterations]\n") + res = CliRunner().invoke(main, ["contract", "evidence", path, "--run", "999"]) + assert res.exit_code != 0 + assert "999" in res.output + + +def test_evidence_with_nothing_declared_says_so(store, tmp_path): + path = _write(tmp_path, "version: 1\ntarget: ./r\ngoal: g\n") + res = CliRunner().invoke(main, ["contract", "evidence", path, "--run", "1"]) + assert res.exit_code == 0, res.output + assert "declares no evidence" in res.output + + +def test_shipped_example_contract_stays_valid(): + """The example in docs/ is executable documentation -- it must parse.""" + import pathlib + + path = pathlib.Path(__file__).resolve().parent.parent / "docs" / "examples" / "loop.yaml" + c = load_contract(str(path)) + assert c.budget.max_iterations == 8 + assert "grade_trajectory" in c.evidence_required + assert c.require_human_confirm is True