diff --git a/docs/architecture/rfcs/hierarchical-agent-stride-control-v0.md b/docs/architecture/rfcs/hierarchical-agent-stride-control-v0.md index b1d3968f04..01f6499484 100644 --- a/docs/architecture/rfcs/hierarchical-agent-stride-control-v0.md +++ b/docs/architecture/rfcs/hierarchical-agent-stride-control-v0.md @@ -697,8 +697,11 @@ adoption, restoration or improved results. These are refinements of the existing Agent questions, not new decision fields, score thresholds or settlement gates; the strategy remains default-off. Identify the remaining authoritative goal gap, its limiting component and plausible headroom before choosing the next mechanism. Compare -continuation with one feasible different approach that could affect that -component, including complete cost. A changed auxiliary mechanism can be useful, +continuation with plausible alternatives that change the limiting assumption or +mechanism, including complete cost. A stalled route calls for a discriminating +experiment across those explanations, not only more local parameter changes. +Distinguish an unexecuted intervention, failed transfer and insufficient tested +route potential; each calls for different evidence. A changed auxiliary mechanism can be useful, but its novelty does not establish that it addresses the remaining gap. Local or synthetic improvement needs applicable transfer evidence; repeated identical outcomes are not new information. Prefer the smallest coverage that resolves the @@ -825,18 +828,36 @@ projection/review semantics. Both historical v1 decision shapes retain read compatibility, but cannot discharge a fresh enabled review. Frozen workers retain their exact source. Pin implementation revisions for comparisons; do not reinterpret old decisions. The policy seam can evolve when evidence warrants it, without maintaining speculative variants. -For an enabled periodic review, every settlement path now requires a current, +For an enabled periodic review or an existing observed-stall review +(`typed_progress_repeat`, `no_progress_streak`, `repeated_action_loop`, +`external_progress_review_drift`), every settlement path requires a current, source-bound stride decision. Typed progress, successor and terminal results remain valid evidence/outcomes, while a bare Todo successor transition cannot -stand in for the review. This closes the decision-recording gap; it does not -establish that the proposed action was adopted or improved the Goal. +stand in for the review. An evidenced path decision can retain unfinished useful +work without a fabricated new progress identifier or Todo. Explicit acceptance +and capability outcome restrictions still apply; default-off and other sources +retain their prior exit policies. No new detector, threshold or action is added. + +The frozen [EdgeBench diagnostic window](https://huggingface.co/datasets/songwells/loopxbench/blob/c5dd30bfaa2f5af42ccd7d0ca9ed59b865a81d42/public/edgebench-20261018-v1/diagnostic_windows.csv) +reports 902 completed native turns and no submissions over one post-hoc selected +230-minute window. This motivates checking whether review changes subsequent +work, but does not identify the cause or establish a general stagnation rate; +zero `apply_patch` calls cannot exclude writes through shell commands. + +Real File/SQLite CLI coverage now takes an existing typed-progress stall through +decision rejection/recovery, an executed comparison of local choice, reverse +priority and joint assignment under one fixed objective, result readback and the +next executable transfer-validation Todo. The current decision can retain the +original Todo; comparison results, not a new label, choose the next mechanism. +This proves the scripted control path, not model adoption or EdgeBench score gain. +Existing negative-result and later-work followthrough coverage remains necessary. The effective review cadence remains owned by the [Goal Vision/replan contract](../../reference/protocols/goal-vision-replan-contract-v0.md#default-review-cadence) and Goal/device configuration. Six effective work Turns is the default; enabling the stride strategy for one Goal inherits its live review cadence, and only an explicit Goal cadence override pins a separate count or unit. The cadence is not Todo size, an experiment boundary or a mandatory route switch; this refinement -changes neither counting nor discharge. Matched continuation must qualify actual +does not change counting. Matched continuation must qualify actual adoption, target benefit and full cost before promotion or synergy claims. Exploratory iteration may run only strategy-enabled test arms and reuse historical diff --git a/docs/architecture/rfcs/hierarchical-agent-stride-control-v0.zh-CN.md b/docs/architecture/rfcs/hierarchical-agent-stride-control-v0.zh-CN.md index c04e62d895..99feb6d376 100644 --- a/docs/architecture/rfcs/hierarchical-agent-stride-control-v0.zh-CN.md +++ b/docs/architecture/rfcs/hierarchical-agent-stride-control-v0.zh-CN.md @@ -589,7 +589,9 @@ switch(尝试不同机制)或 verify(解决决定扩大或切换的具体 或结果改善。这些是既有Agent问题的细化,不增加决策字段、分数阈值或结算门槛; 策略仍默认关闭。先识别仍未满足的 权威目标、当前瓶颈分量和可能的收益空间,再选择下一机制。比较原路线接续与 -一种能影响该瓶颈的可行替代路线,并计入完整成本。辅助机制变化可能有用,但 +改变该瓶颈假设或机制的可行替代路线,并计入完整成本。路线停滞时,应以能区分 +这些解释的完整实验推进,不能只继续局部调参。区分干预未执行、局部改善未迁移, +以及被当前测试限制的路线潜力;它们需要不同证据。辅助机制变化可能有用,但 新颖本身不证明能解决剩余缺口。局部或合成数据的提升须有适用的迁移证据;重复 相同结果不算新信息。优先用能决定路线选择的最少验证范围,避免再做宽泛扫测, 同时保留相关失败证据。 @@ -674,13 +676,28 @@ Benchmark 的显式 `loopx-planned` provider 在未修改的规划 packet 外附 普通投影/复盘;两种历史 v1 决策保留原义读取兼容,不能完成新的启用策略复盘; 新提案使用 v2 数据格式,仍只有同一个可选策略,冻结 worker 保持确切源码。比较固定 实现 revision,不重释旧决定;有证据后迭代同一组件,避免维护假设性的策略变体。 -启用策略的周期复盘到期后,任何结算路径都须留下绑定本次来源的 stride 决定。 +启用策略的周期复盘到期,或既有停滞复核触发时(`typed_progress_repeat`、 +`no_progress_streak`、`repeated_action_loop`、`external_progress_review_drift`), +任何结算路径都须留下绑定本次来源的 stride 决定。 类型化进展、后续任务和终止结果仍可作为证据与结果,但单独的 Todo 后续转换不能 -代替复盘。这补齐决定记录闭环,不证明提案已被采用或提升了 Goal。 +代替复盘。有证据的路径决定可以保留尚未完成的有用工作,不必编造新进展标识或 +Todo。验收和能力 owner 显式限定的结果仍须遵守;默认关闭和其他来源保持原有 +出口规则。不增加检测器、阈值或动作。 + +冻结的 [EdgeBench 诊断窗口](https://huggingface.co/datasets/songwells/loopxbench/blob/c5dd30bfaa2f5af42ccd7d0ca9ed59b865a81d42/public/edgebench-20261018-v1/diagnostic_windows.csv) +记录一个事后选取的 230 分钟窗口中,原生 turn 完成 902 次、提交为零。这支持 +检查复盘是否改变后续工作,但不能识别原因或推算普遍停滞率;`apply_patch` 为零 +也不能排除 shell 命令修改文件。 + +真实 File/SQLite CLI 验证现覆盖既有 typed-progress 停滞、缺决定时拒绝及修正、 +同一固定目标下实际执行局部选择/逆序优先/联合分配三种方法、读取结果,以及 +创建下一项可执行的迁移验证 Todo。本次决定可以保留原 Todo;实验结果驱动下一 +机制选择,改名本身不算换路。这证明脚本化控制链路,不证明模型采用或 EdgeBench +分数提升;既有负结果与后续实际工作跟进验证仍有必要。 有效复核周期仍由[Goal Vision/replan 契约](../../reference/protocols/goal-vision-replan-contract-v0.md#default-review-cadence) 及 Goal/设备配置负责。六个有效工作 Turn 是默认值;对单个 Goal 启用步幅策略时 沿用实时复盘周期,只有显式设置 Goal 复盘周期,才固定该 Goal 的计数依据或数量。 -周期不是 Todo 大小、实验边界或强制换路信号;本细化不改变计数或解除条件。 +周期不是 Todo 大小、实验边界或强制换路信号;本细化不改变计数。 策略采用、目标收益、完整成本和交互增益须通过匹配接续等实验验证后再作结论。 敏捷探索可只跑启用策略的测试组,以历史基线作描述性参照;不要求每次迭代都 diff --git a/docs/reference/protocols/goal-vision-replan-contract-v0.md b/docs/reference/protocols/goal-vision-replan-contract-v0.md index 9fab3977e8..1c86f45b5f 100644 --- a/docs/reference/protocols/goal-vision-replan-contract-v0.md +++ b/docs/reference/protocols/goal-vision-replan-contract-v0.md @@ -553,6 +553,25 @@ resolve the execution gap or name the changed inputs/new uncertainty first. A `verify` action names the grow-or-switch decision its result will inform; repeated checking without new information is not a default continuation policy. +When evidence weakens the current route, the existing `route_distance` guidance +asks for a small set of genuinely different explanations or approaches before +selecting an experiment: representation, decomposition, search method or +information source as applicable. Contrast predictions and full costs, preserve +the qualified incumbent, and choose a comparison that can change the next +decision. Parameter variants alone are not distinct mechanisms. Missing feedback +is a wait/evidence gap, not evidence that a route failed or alternatives were +exhausted. This is opt-in Agent guidance in the existing axes/reasons and +`next_probe`, not a new proposal field, forced pivot, branch scheduler, permission +or automatic experiment launch. A larger portfolio or a recorded switch does +not establish adoption or benefit; subsequent evidence must still qualify both. + +当证据削弱当前路线时,现有 `route_distance` 指引要求先提出少量真正不同的解释或 +方法,再选择实验;可考虑表征、问题分解、搜索机制或信息来源。比较各自预测和 +完整成本,保留已验证的当前最佳产物,选择能改变下一步决策的对照。相同机制的 +参数变体不算不同路线;缺少反馈属于等待或证据缺口,不证明路线失败或其他方法 +已经穷尽。这仅补充启用策略后的既有轴、理由与 `next_probe` 指引,不增加字段、 +强制转向、分支调度或执行权限。记录了切换或更多候选,不等于执行或效果已经成立。 + When a prior accepted decision is delivered, `previous_review` references its exact `replan-decision-*` id and separately records: @@ -597,14 +616,22 @@ adopted it. Existing v2 decisions without pages and historical v1 reads retain their behavior. Other detail routes are not supplemental citation sources. The typed owner validates shape, references and declared provenance before the -fresh Vision path exit. When an enabled Goal reaches a periodic review, every +fresh Vision path exit. When an enabled Goal reaches a periodic review or an +existing observed-stall review (`typed_progress_repeat`, `no_progress_streak`, +`repeated_action_loop`, `external_progress_review_drift`), every accepted exit must also record a current `agent_vision.stride_decision`: a new progress observation, runnable successor, blocker or coverage-backed terminal result cannot silently replace that decision. The typed outcome remains legal and retains its own evidence gate. The decision references both the due obligation and its delivered review context. A terminal review may attach it to an evidence-linked `stop` path. A Todo successor transition alone cannot settle -the due review. Other replan sources retain their existing exit policies. +the review. An evidence-linked `continue` or `no_change` decision can retain useful +unfinished work without manufacturing a new progress identifier or successor. +Explicit acceptance/capability outcome restrictions still apply. Other replan +sources retain their existing exit policies; `off` retains the prior stall exits. +This changes the completeness of an enabled review, not stall detection, cadence, +authority or the choice of action. + All attachment text counts toward the existing 1,800-character Vision budget. The live packet exposes field limits. Legacy recorded declarations and old explicit off-mode attachments retain transport @@ -632,11 +659,16 @@ revision 与具体 finding,说明采用、暂缓、无适用证据或读取不 按原 Turn 身份重新审视;不开额外执行权或配额。机器校验提案结构与引用,不能证明 模型判断正确或已提升收益。两种历史 v1 决策保留原义读取兼容,但不能完成新的 启用策略复盘;数据格式更新不增加可选策略版本。 -策略启用且周期复盘到期时,所有可接受出口都须记录绑定本次义务的 +策略启用且周期复盘到期,或既有停滞复核触发时(`typed_progress_repeat`、 +`no_progress_streak`、`repeated_action_loop`、`external_progress_review_drift`), +所有可接受出口都须记录绑定本次义务的 `agent_vision.stride_decision`。新进展、可执行后续任务、阻塞或有覆盖证据的终止 仍可作为结果,但不能代替复盘决定;决定须引用本次义务及交付的复盘上下文, 单独创建后续 Todo 也不能结算到期义务。 -终止时可把决定附在有证据的 `stop` 路径上。其他 replan 来源沿用原有出口规则。 +终止时可把决定附在有证据的 `stop` 路径上。有证据的 `continue` 或 `no_change` +可以保留尚未完成的有用工作,不必编造新进展标识或后续任务;验收和能力 owner +显式限定的结果仍须遵守。其他 replan 来源和 `off` 模式的停滞出口保持原规则。 +这补齐启用后的复核内容,不改变停滞检测、周期、权限或替 Agent 选择动作。 The same opt-in strategy covers ordinary engineering, organizing and maintenance. `grow` expands a coherent accepted deliverable, including related integration, @@ -656,6 +688,94 @@ not improved model decisions or net efficiency. 应在原 Turn 和原复盘义务下补读并重新评估。读取不写入回执或扣额度,分页引用随 决策保留到历史和下次复盘;来源核验不证明模型理解、采用或研究收益。 +### Source-bound comparison selection + +With Explore planning and `evidence_conditioned_v1` enabled, the existing +turn-context result page can include `comparison_candidates`. Each card links +an actionable, own-or-unclaimed canonical Todo to up to three current results +through that Todo's explicit Explore node references. An outcome contains +`node_id`, `finding_id`, and `source_sha256`, the digest of the full scoped +result. These links survive into `stride_control.explore_basis`; an Agent can +compare the actual result contents before planning further work. Planner rank +does not grant or remove execution permission: eligible linked Todos from the +planner's unselected pool can also supply cards. Negative findings do not +automatically penalize, forbid or replace a route. + +A v2 decision may optionally copy one exact outcome and its Todo into +`explore_review.selected_candidate`: + +```json +{ + "todo_id": "todo_current_probe", + "node_id": "current-question", + "finding_id": "current-result", + "source_sha256": "<64 lowercase hexadecimal characters from the source card>" +} +``` + +This declaration requires `disposition: "used"`, the selected finding in +`considered_refs`, and `retain:todo_current_probe` in the same Vision's existing +`todo_delta`. Qualification of a fresh Vision stride decision checks the +current source link and exact content digest; an unrelated fresh-path or +successor outcome cannot make an invalid binding qualify as that decision. +Changed source requires reassessment. For an enabled periodic or observed-stall +review listed above, a standalone canonical successor ACK cannot replace the +current evidence-linked stride decision. Other obligation sources and the +disabled strategy keep their existing outcome gates; a successor ACK alone +does not make an attached comparison selection an accepted decision or executed +work. +The optional fields share the existing Vision budget. A decision without this +field, disabled planning, and the strategy's `off` mode retain their existing +behavior and authority. + +Cards use the existing result page coordinates and limits. If a card is not +visible, read a current global result page and attach its coordinates in +`explore_review.result_pages`. Increasing the existing `--result-limit` to its +maximum of 20 can expose more Todos linked to the same finding. The page's +`omitted_comparison_candidates` reports remaining omissions; more than 20 +matching Todos on one page are not all selectable through this optional +binding. Up to three replayed pages and three considered findings remain the +existing upper bounds. No new pagination or scheduling protocol is created. + +The selection records evidence-backed **planning intent** for the retained +Todo. It does not reserve that Todo or force the next scheduler choice. Another +legally admitted Todo may run; its result remains attributable to its own work. +The existing `decision_followthrough.planned_work` and scoped history distinguish +the declared Todo from subsequent evidence, keep execution and Goal gain +unknown without proof, and reject an execution claim supported only by +unrelated work. An identifier or changed source digest cannot certify a new +mechanism. The Agent must explain the mechanism, predicted benefit, evidence +and uncertainty, then perform the chosen work and satisfy the original +acceptance through the existing owners. + +The local comparison fixture exercises one original implementation Todo with +three measured alternatives: retaining its program, searching parameters in a +fixed representation, and generating programs from a finite arithmetic grammar. +Each family performs 125 score calls against the same six training inputs. +The choice uses those measured results; an additive positive control can retain +the original program. The selected source is then deployed and executed by an +independent pinned owner verifier on withheld inputs. This checks actual +program adoption within the Todo, not mandatory cross-Todo scheduling or live +provider creativity. Grammar generation CPU, token usage and money are not +equalized by the score-call limit. The controlled Host does not read withheld +inputs before choosing; the fixture is not an adversarial data-access sandbox. +Keep measured choice, reasons and final verifier results in the existing +`path_delta` and Explore findings so the shipped evidence view can show them +beside linked Todo state. + +**中文:** 当前结果页可把真实可执行 Todo 与其明确关联的实验结果一起交给复盘, +包括未进入高分候选的合法路线;排名不替代准入。可选 `selected_candidate` 精确 +绑定 Todo、节点、finding 及完整来源摘要,同时要求同一 Vision 保留该 Todo。 +错源、已变化的结果或没有保留对应工作均不能借别的后继结果混过核验。上述已启用 +的周期或停滞复核均不能用单独的后继 ACK 代替当前决策;其他义务和关闭策略时 +保留原出口边界,后继 ACK 本身也不证明比较选择已被接受或执行。分页仍沿用 +既有坐标与最多 20 项的上限,同一结果关联过多 Todo 时明确显示省略数,不能声称 +无限候选均可达。这是有来源的规划意图,不是调度命令;合法 B 任务可以执行, +但不能凭 B 的结果声称所选 A 已执行。实际采用、机制差异和原验收仍需真实证据。 +受控实验比较停留原式、局部调参与结构语法搜索,评分调用预算相同;它证明原 +Todo 内的实际选程序、部署与独立验收,不证明付费模型创造力、完整成本或普遍 +研究收益。选择、理由和实际结果复用现有路径说明、Explore finding 与任务回读。 + ### Path Delta A vision packet may include a top-level `path_delta` object; `goal_path_delta_v0` diff --git a/examples/personal-workspace-browser/evidence/research-continuity/README.md b/examples/personal-workspace-browser/evidence/research-continuity/README.md index 1cc71779ad..6ef0e50cf6 100644 --- a/examples/personal-workspace-browser/evidence/research-continuity/README.md +++ b/examples/personal-workspace-browser/evidence/research-continuity/README.md @@ -1,4 +1,6 @@ -# Research worker readback: packaged first viewports +# Research continuity evidence + +## Research worker readback: packaged first viewports These screenshots use a disposable synthetic Goal and Agent. They contain no live workspace, model session, private trace or user data. @@ -49,3 +51,54 @@ An absent optional Agent endpoint may return its typed session-preparation error; this is handled separately from business writes. The fixture does not install a model provider, invoke a paid model or claim to measure complete provider costs. + +## Measured method comparison in the existing results view + +These retained captures use disposable synthetic data. They were produced for +the original research-continuity proposal, before the independent main-targeted +slice. They record that source and are not a new validation of this branch. +Worker attempt/readback evidence belongs to PR #6885; this slice carries only +the comparison evidence and its existing Explore/Todo owners. + + +The original run also read a completed synthetic method comparison through the +same packaged Chat server and its real Explore HTTP API. It uses the File +case produced by `tests/control_plane/test_research_comparison_cli.py`: +three search families measure their actual generated expressions, the same +original Todo adopts one, and the unchanged owner verifier checks deployed +source on withheld inputs. The test separately covers SQLite and an additive +positive control that retains the original expression. + +| View | Capture | +| --- | --- | +| Desktop results, selected adoption and original completed Todo | [Packaged viewport](comparison-adopted-desktop.png) | +| Mobile viewport: measured losses and adoption explanation | [Result viewport](comparison-adopted-mobile-result-viewport.png) | +| Mobile viewport: source and original completed Todo | [Scope viewport](comparison-adopted-mobile-scope-viewport.png) | + +These historical captures show the original assembled candidate after its merged-source CLI comparison, using +the existing results layout. Both mobile images capture the actual +390 × 844 viewport at overlapping scroll positions: its 346-pixel content +region cannot show the full 477-pixel evidence reader at once. No layout, +DOM or image cropping was applied. The readback +checks the recorded 54 / 16 / 0 training losses, the original Todo's `done` +state, source references and adoption explanation. Navigation and reading +append no work or settlement history. The original training observations keep +their then-unknown heldout result; a later finding records the independent +verification and bounded adoption. The interface still distinguishes a saved +claim from independently certified adoption. + +Generate the source with the source-checkout Python environment: + +```bash +python -m pytest --no-cov tests/control_plane/test_research_comparison_cli.py +``` + +The finite program-search Host is deterministic. Equal score-call budgets do +not establish equal grammar-generation CPU, model tokens, money, or improved +EdgeBench performance. No model provider is used by the readback. + +**中文:** 新增截图读取实际合成实验完成后的原 Explore/Todo 数据,展示三种 +方法的测量、所选程序的采用说明和原任务完成状态。两张手机图分别记录实际 +视口中相互重叠的滚动位置,连续展示结果和关联任务,没有修改页面或裁切。 +查看与导航没有追加执行或结算;历史实验中的未知与后续已验证 +结果分别保留。该证据不等于真实模型或完整成本的效果对照。 diff --git a/examples/personal-workspace-browser/evidence/research-continuity/comparison-adopted-desktop.png b/examples/personal-workspace-browser/evidence/research-continuity/comparison-adopted-desktop.png new file mode 100644 index 0000000000..405a537063 Binary files /dev/null and b/examples/personal-workspace-browser/evidence/research-continuity/comparison-adopted-desktop.png differ diff --git a/examples/personal-workspace-browser/evidence/research-continuity/comparison-adopted-mobile-result-viewport.png b/examples/personal-workspace-browser/evidence/research-continuity/comparison-adopted-mobile-result-viewport.png new file mode 100644 index 0000000000..b1d013fab5 Binary files /dev/null and b/examples/personal-workspace-browser/evidence/research-continuity/comparison-adopted-mobile-result-viewport.png differ diff --git a/examples/personal-workspace-browser/evidence/research-continuity/comparison-adopted-mobile-scope-viewport.png b/examples/personal-workspace-browser/evidence/research-continuity/comparison-adopted-mobile-scope-viewport.png new file mode 100644 index 0000000000..58be0a5c84 Binary files /dev/null and b/examples/personal-workspace-browser/evidence/research-continuity/comparison-adopted-mobile-scope-viewport.png differ diff --git a/loopx/control_plane/capabilities/explore_turn_context.ts b/loopx/control_plane/capabilities/explore_turn_context.ts index 498b6134c4..6acf3c2fd3 100644 --- a/loopx/control_plane/capabilities/explore_turn_context.ts +++ b/loopx/control_plane/capabilities/explore_turn_context.ts @@ -3,6 +3,7 @@ import {createHash} from "node:crypto"; import type {JsonObject} from "../effect_program.ts"; import {EffectRuntimeRequestError} from "../effect_runtime_errors.ts"; import {requireInteger, requireJsonObject, requireNonEmptyString, requireStringArray} from "../runtime_decode.ts"; +import {PLANNING_TODO_ID} from "../work_items/planning_relations.ts"; import {EXPLORE_WRITEBACK_SUMMARY_LIMIT, exploreResultWritebackAffordance} from "./explore_result_writeback.ts"; import {normalizeExploreResultPages} from "./explore_result_page.ts"; @@ -116,6 +117,30 @@ export function projectExploreTurnContext(params: JsonObject): JsonObject { const resultNode = params.result_node == null ? null : requireNonEmptyString(params.result_node, "result_node"); const resultRows = resultNode == null ? writebackResults : writebackResults.filter(row => row.node_id === resultNode); const nodeById = new Map(nodes.map(node => [node.node_id, node])); + // Bind current Todo links to the results on this page. Include the full + // planner's nonselected branches: a heuristic/batch budget cannot prevent + // the Agent from comparing a lower-ranked method. This grants no claim, + // lease, quota or execution permission; their existing owners still apply. + const comparisonCandidates = (page: JsonObject[], limit: number): JsonObject => { + if (!harness) return {}; + const linkedCandidates = [...branches, ...rejected].flatMap(branch => { + if (typeof branch.todo_id !== "string" || !PLANNING_TODO_ID.test(branch.todo_id) + || branch.actionable_open !== true || branch.claimed_by && branch.claimed_by !== agent + || branch.typed_evidence_audit == null) return []; + const audit = requireJsonObject(branch.typed_evidence_audit, "Explore evidence audit"); + const linked = new Set(requireStringArray(audit.requested_node_refs ?? [], "Explore node refs") + .filter(ref => nodeById.has(ref))); + const outcomes = page.filter(row => linked.has(String(row.node_id))).slice(0, 3); + return outcomes.length ? [{todo_id: branch.todo_id, outcomes}] : []; + }); + if (!linkedCandidates.length) return {}; + return { + comparison_candidates: linkedCandidates.slice(0, limit).map(card => ({todo_id: card.todo_id, + outcomes: card.outcomes.map(row => ({node_id: row.node_id, finding_id: row.finding_id, + source_sha256: createHash("sha256").update(JSON.stringify(row)).digest("hex")}))})), + omitted_comparison_candidates: Math.max(0, linkedCandidates.length - limit), + }; + }; const relationsByNode = new Map(); for (const edge of rows(projection.edges)) { for (const id of new Set([edge.from_node, edge.to_node])) { @@ -157,9 +182,11 @@ export function projectExploreTurnContext(params: JsonObject): JsonObject { // context/read route; the replan owner refuses any unresolved citation. return []; } - return [{...page, finding_refs: resultRows.slice(Number(page.offset), Number(page.offset) + Number(page.limit)) - .map(row => row.finding_id)}]; + const found = resultRows.slice(Number(page.offset), Number(page.offset) + Number(page.limit)); + return [{...page, finding_refs: found.map(row => row.finding_id), + ...comparisonCandidates(found, Number(page.limit))}]; }); + const visibleComparisons = comparisonCandidates(visibleResults, resultLimit); const remainingResults = Math.max(0, resultRows.length - resultOffset - visibleResults.length); const resultRead = (...args: string[]) => command("turn-context", "--agent-id", agent, ...args); const writeback = exploreResultWritebackAffordance(); @@ -211,6 +238,7 @@ export function projectExploreTurnContext(params: JsonObject): JsonObject { orchestration_gate: gate, candidate_count: plan.candidate_count ?? 0, selected_branches: branches.slice(0, 3).map(branch => branchContext(branch, findingSources)), + ...visibleComparisons, omitted_selected_branches: Math.max(0, branches.length - 3), rejected_candidates: rejected.slice(0, 3).map(branch => branchContext(branch)), omitted_rejected_candidates: Math.max(0, diff --git a/loopx/control_plane/work_items/replan_context.ts b/loopx/control_plane/work_items/replan_context.ts index 3627cd7cb0..ab6c1d6b4f 100644 --- a/loopx/control_plane/work_items/replan_context.ts +++ b/loopx/control_plane/work_items/replan_context.ts @@ -577,7 +577,12 @@ export function projectReplanContext(value: unknown): JsonObject { ? (obligation.triggers as JsonObject[]).find(trigger => trigger.progress_baseline)?.progress_baseline : null ) ?? null; - const frontier = {baseline, required_any_of: obligation ? requiredSemanticOutcomes(obligation) : []}; + const strideStrategy = normalizeStrideStrategy(request.stride_strategy); + // This context has not been attached yet. Project outcomes with its current + // effective strategy through the same owner used by requirements/writeback. + const frontier = {baseline, required_any_of: obligation ? requiredSemanticOutcomes({ + ...obligation, replan_context: {stride_strategy: strideStrategy}, + }) : []}; const prefix = request.read_prefix === undefined ? "loopx" : requireNonEmptyString(request.read_prefix, "read_prefix"); const groups = new Map(); for (const row of evidence) { @@ -626,7 +631,6 @@ export function projectReplanContext(value: unknown): JsonObject { ? [String((row.progress_observation as JsonObject).fingerprint)] : [])); const ledger = [...coverage].filter(([key]) => selectedCoverage.has(key)).map(([, value]) => value); const core = projectCoreGoal(request.goal_facts); - const strideStrategy = normalizeStrideStrategy(request.stride_strategy); const exploreBasis = strideStrategy !== "off" ? projectStrideExploreBasis(request.explore_context, goal, agent) : null; const followthrough = agent && strideStrategy !== "off" ? decisionFollowthrough(rows, request.goal_facts, request.from_full_index === true, request.todo_items) : null; diff --git a/loopx/control_plane/work_items/replan_semantics.ts b/loopx/control_plane/work_items/replan_semantics.ts index b3a7b44cb6..d55971a6e2 100644 --- a/loopx/control_plane/work_items/replan_semantics.ts +++ b/loopx/control_plane/work_items/replan_semantics.ts @@ -73,12 +73,15 @@ function isExternalReview(obligation: JsonObject): boolean { return triggerKinds(obligation).some(kind => EXTERNAL_REVIEW_TRIGGERS.has(kind)); } -// A due cadence review is a decision checkpoint. Typed progress may still be -// its outcome, but cannot stand in for the review when the strategy is enabled. -function requiresPeriodicStrideDecision(obligation: JsonObject, fallbackStrategy: unknown = "off"): boolean { +// Existing cadence and observed-stall triggers become decision checkpoints +// only for the opted-in strategy. This does not detect stalls or force a pivot. +const STRIDE_REVIEW_TRIGGERS = new Set(["periodic_review_due", "typed_progress_repeat", + "no_progress_streak", "repeated_action_loop", "external_progress_review_drift"]); +function requiresStrideDecision(obligation: JsonObject, fallbackStrategy: unknown = "off"): boolean { const context = object(obligation.replan_context); const strategy = Object.hasOwn(context, "stride_strategy") ? context.stride_strategy : fallbackStrategy; - return triggerKinds(obligation).includes("periodic_review_due") && normalizeStrideStrategy(strategy) !== "off"; + return triggerKinds(obligation).some(kind => STRIDE_REVIEW_TRIGGERS.has(kind)) && + normalizeStrideStrategy(strategy) !== "off"; } /** Route only a revalidated canonical transition to its original Turn guard. @@ -99,7 +102,7 @@ function projectTurnTransition(request: JsonObject): JsonObject { const candidateSource = object(value); const sourceObligation = object(candidateSource.obligation); if (sourceObligation.obligation_id !== selected) continue; - if (requiresPeriodicStrideDecision(sourceObligation, request.stride_strategy)) continue; + if (requiresStrideDecision(sourceObligation, request.stride_strategy)) continue; const ack = object(candidateSource.ack); const candidate = object(ack.semantic_delta); if (ack.schema_version === "autonomous_replan_ack_v0" && @@ -287,9 +290,10 @@ export function requiredSemanticOutcomes(obligation: JsonObject): SemanticOutcom // Periodic and long-chain reviews may retain existing runnable work. The // vision decision must close the checkpoint without manufacturing another // successor or progress identifier. Keep previously legal progress exits. - // An external progress review likewise lets the Agent keep its plan on - // evidence (a fresh vision path) or pivot with a typed progress delta. - return kinds.some(kind => kind === "long_todo_chain" || + // An enabled stall review can also retain the unfinished useful experiment + // after a current evidenced decision, without inventing progress or a Todo. + // Explicit capability/acceptance outcome restrictions above still apply. + return requiresStrideDecision(obligation) || kinds.some(kind => kind === "long_todo_chain" || kind === "periodic_review_due" || EXTERNAL_REVIEW_TRIGGERS.has(kind)) ? ["fresh_vision_path_outcome", ...PROGRESS_OUTCOMES] : [...PROGRESS_OUTCOMES]; } @@ -403,21 +407,21 @@ export function projectReplanSemantics(value: unknown): JsonObject { if (request.operation === "receipt_bound_obligation") return preserveReceiptBoundObligation(request); const obligation = requireJsonObject(request.obligation, "obligation"); if (request.operation === "transition_eligibility") { - return {allowed: !requiresPeriodicStrideDecision(obligation, request.stride_strategy)}; + return {allowed: !requiresStrideDecision(obligation, request.stride_strategy)}; } const required = requiredSemanticOutcomes(obligation); const externalReview = isExternalReview(obligation); if (request.operation === "requirements") { const strideControl = strideControlContract(object(obligation.replan_context), obligation.obligation_id); const writeback = writebackProjection(required, externalReview); - if (requiresPeriodicStrideDecision(obligation)) { + if (requiresStrideDecision(obligation)) { const contract = object(writeback.writeback_contract); writeback.cli_semantic_args = VISION_CLI_ARGS; writeback.writeback_contract = {...contract, preferred_input: "evidence_linked_vision_path", required_fields: [...new Set([...strings(contract.required_fields), "agent_vision.stride_decision"])], path_outcomes: [...FRESH_PATH_DISPOSITIONS, "stop"], - rule: "This due periodic review must record a current evidence-linked stride_decision before any typed progress, successor, blocker or terminal outcome can settle it. " + + rule: "This cadence or observed-stall review must record a current evidence-linked stride_decision before any typed progress, successor, blocker or terminal outcome can settle it. " + String(contract.rule ?? "")}; } return {required_any_of: required, planning_guidance: [...REPLAN_PLANNING_GUIDANCE, @@ -437,21 +441,22 @@ export function projectReplanSemantics(value: unknown): JsonObject { const context = object(obligation.replan_context); const [strideDecision] = normalizeStrideDecision(vision.stride_decision); const structuredReview = normalizeStrideStrategy(context.stride_strategy) !== "off"; - const periodicStrideReview = requiresPeriodicStrideDecision(obligation); + const selectedComparison = object(strideDecision?.explore_review).selected_candidate != null; + const requiredStrideReview = requiresStrideDecision(obligation); const pathOutcome = String(path.outcome ?? "").trim(); const evidenceLinkedPath = Boolean(String(patch.acceptance_summary ?? "").trim() && (FRESH_PATH_DISPOSITIONS.has(pathOutcome) || pathOutcome === "stop") && strings(path.evidence_refs).length); const freshPath = evidenceLinkedPath && FRESH_PATH_DISPOSITIONS.has(pathOutcome); const independentStride = strideDecision?.schema_version === STRIDE_DECISION_SCHEMA; - const missingStride = structuredReview && (freshPath || periodicStrideReview) && + const missingStride = structuredReview && (freshPath || requiredStrideReview) && (!strideDecision || !independentStride); - const missingStridePath = periodicStrideReview && !evidenceLinkedPath; - const staleStride = !!strideDecision && structuredReview && + const missingStridePath = requiredStrideReview && !evidenceLinkedPath; + const staleStride = !!strideDecision && (structuredReview || selectedComparison) && strideDecision.reference_obligation_id !== obligation.obligation_id; - const reviewMismatch = strideDecision && structuredReview - ? periodicStrideReview && strideDecision.reference_context_id !== context.context_id + const reviewMismatch = strideDecision && (structuredReview || selectedComparison) + ? (requiredStrideReview || selectedComparison) && strideDecision.reference_context_id !== context.context_id ? "Stride decision must reference the delivered due-review context." - : strideReviewMismatch(strideDecision, context, request.stride_review_evidence) + : strideReviewMismatch(strideDecision, context, request.stride_review_evidence, vision.todo_delta) : null; let outcomes = strings(observation.delta_kinds); if (outcomes.some(value => (CAPABILITY_OUTCOMES as readonly string[]).includes(value))) { @@ -482,7 +487,8 @@ export function projectReplanSemantics(value: unknown): JsonObject { outcomes.push("fresh_vision_path_outcome"); } if (staleStride || missingStride || reviewMismatch) outcomes = outcomes.filter(outcome => outcome !== "fresh_vision_path_outcome"); - const satisfying = inconsistentTerminal || (periodicStrideReview && (missingStride || missingStridePath || staleStride || reviewMismatch)) + const satisfying = inconsistentTerminal || (requiredStrideReview && (missingStride || missingStridePath || staleStride || reviewMismatch)) + || (selectedComparison && (staleStride || reviewMismatch)) ? [] : outcomes.filter(outcome => required.includes(outcome as SemanticOutcome)); const replayRefused = (replayed || knownBlocker) && !satisfying.length && !inconsistentTerminal; const identityRefused = identityWithoutEvidence && !satisfying.length && !inconsistentTerminal; diff --git a/loopx/control_plane/work_items/stride_policy.ts b/loopx/control_plane/work_items/stride_policy.ts index dcb588e0ad..622e8364b2 100644 --- a/loopx/control_plane/work_items/stride_policy.ts +++ b/loopx/control_plane/work_items/stride_policy.ts @@ -7,6 +7,7 @@ import {EffectRuntimeRequestError} from "../effect_runtime_errors.ts"; import type {JsonObject} from "../effect_program.ts"; import {requireJsonObject, requireNonEmptyString, requireStringLiteral} from "../runtime_decode.ts"; import {normalizeExploreResultPages} from "../capabilities/explore_result_page.ts"; +import {PLANNING_TODO_ID} from "./planning_relations.ts"; export function normalizeStrideStrategy(value: unknown): string { if (value === undefined) return schema.default; @@ -26,6 +27,9 @@ const decisionValueReview = { route_distance: "Name the remaining authoritative goal gap, its limiting component and the plausible headroom. " + "Keep the retained best artifact as the comparison anchor and check whether subsequent work actually implements the proposed mechanism change. " + "Compare a different approach that can affect that component. Distinguish an unexecuted intervention, failed transfer and insufficient route potential: they imply different next experiments. " + + "When evidence weakens the current route, generate a small set of genuinely different explanations or approaches before selecting one: consider representation, decomposition, search method or information source where relevant. " + + "Contrast their predictions and full costs, then choose a discriminating experiment; parameter variants of the same mechanism are not independent alternatives. " + + "Missing feedback is a wait or evidence gap, not proof of route failure or exhausted alternatives. " + "A changed auxiliary mechanism alone does not show that it addresses the gap.", validation_scope: "Use evidence applicable to the limiting outcome and changed mechanism. " + "Bind comparisons to the artifact version, input conditions and original budget, and distinguish when feedback became available from when work was submitted. " + @@ -87,6 +91,35 @@ function refs(value: unknown, label: string): string[] { } return value as string[]; } +function selectedCandidate(value: unknown): JsonObject { + const input = requireJsonObject(value, "explore_review.selected_candidate"); + exactFields(input, ["todo_id", "node_id", "finding_id", "source_sha256"], "selected_candidate"); + const todo = requireNonEmptyString(input.todo_id, "selected_candidate.todo_id"); + if (!PLANNING_TODO_ID.test(todo)) throw new EffectRuntimeRequestError("selected_candidate requires a public Todo id"); + const node = refs([input.node_id], "selected_candidate.node_id")[0]; + const finding = refs([input.finding_id], "selected_candidate.finding_id")[0]; + const digest = requireNonEmptyString(input.source_sha256, "selected_candidate.source_sha256"); + if (!BARE_SHA256_PATTERN.test(digest)) throw new EffectRuntimeRequestError("selected_candidate requires the exact source digest"); + return {todo_id: todo, node_id: node, finding_id: finding, source_sha256: digest}; +} +function comparisonCandidates(value: unknown): JsonObject[] { + if (value === undefined) return []; + if (!Array.isArray(value) || value.length > 20) throw new EffectRuntimeRequestError("comparison_candidates must be a bounded source page"); + return value.map(value => { + const card = requireJsonObject(value, "comparison candidate"); + exactFields(card, ["todo_id", "outcomes"], "comparison candidate"); + if (!Array.isArray(card.outcomes) || !card.outcomes.length || card.outcomes.length > 3) { + throw new EffectRuntimeRequestError("comparison candidate requires at most three scoped outcomes"); + } + const outcomes = card.outcomes.map(value => { + const outcome = requireJsonObject(value, "comparison outcome"); + exactFields(outcome, ["node_id", "finding_id", "source_sha256"], "comparison outcome"); + const checked = selectedCandidate({todo_id: card.todo_id, ...outcome}); + return {node_id: checked.node_id, finding_id: checked.finding_id, source_sha256: checked.source_sha256}; + }); + return {todo_id: card.todo_id, outcomes}; + }); +} /** Agent interpretations are structured claims, not certified evidence or authority. */ export function normalizeStrideDecision(value: unknown, bounded: BoundedText = text): @@ -164,7 +197,7 @@ export function normalizeStrideDecision(value: unknown, bounded: BoundedText = t } if (input.explore_review != null) { const review = requireJsonObject(input.explore_review, "stride_decision.explore_review"); - exactFields(review, ["revision", "disposition", "considered_refs", "reason", ...(!historical ? ["result_pages"] : [])], "explore_review"); + exactFields(review, ["revision", "disposition", "considered_refs", "reason", ...(!historical ? ["result_pages", "selected_candidate"] : [])], "explore_review"); if (review.revision !== null && !BARE_SHA256_PATTERN.test(requireNonEmptyString(review.revision, "explore_review.revision"))) throw new EffectRuntimeRequestError("explore_review requires a delivered revision or null when unavailable"); requireStringLiteral(review.disposition, decisionSchema.explore_dispositions, "explore_review.disposition"); const considered = refs(review.considered_refs, "explore_review.considered_refs"); @@ -180,6 +213,14 @@ export function normalizeStrideDecision(value: unknown, bounded: BoundedText = t (output.explore_review as JsonObject).result_pages = pages; usage["stride_decision.explore_review.result_pages"] = JSON.stringify(pages).length; } + if (Object.hasOwn(review, "selected_candidate")) { + const candidate = selectedCandidate(review.selected_candidate); + if (review.disposition !== "used" || review.revision === null || !considered.includes(String(candidate.finding_id))) { + throw new EffectRuntimeRequestError("selected_candidate requires used, cited Explore evidence"); + } + (output.explore_review as JsonObject).selected_candidate = candidate; + usage["stride_decision.explore_review.selected_candidate"] = JSON.stringify(candidate).length; + } } return [output, usage]; } @@ -231,6 +272,7 @@ export function projectStrideExploreBasis(value: unknown, goal: string, agent: s const findings = Array.isArray(graph?.writeback_results) ? graph.writeback_results as JsonObject[] : []; const recent = Array.isArray(graph?.recent_findings) ? graph.recent_findings as JsonObject[] : []; const branches = Array.isArray(harness?.selected_branches) ? harness.selected_branches as JsonObject[] : []; + const candidates = source.harness_enabled === true ? comparisonCandidates(harness?.comparison_candidates) : []; const audits = branches.flatMap(branch => { const audit = branch.typed_evidence_audit as JsonObject | undefined; return Array.isArray(audit?.findings) ? audit.findings as JsonObject[] : []; @@ -247,8 +289,11 @@ export function projectStrideExploreBasis(value: unknown, goal: string, agent: s // Planner scores/counts and unrelated pool churn do not alter a cited // finding's source. Bind the actual bounded evidence, not the whole view. revision: createHash("sha256").update(JSON.stringify([goal, agent, source.graph_enabled, - source.harness_enabled, [...byId.values()].slice(0, 6)])).digest("hex"), + source.harness_enabled, [...byId.values()].slice(0, 6), ...(candidates.length ? [candidates] : [])])).digest("hex"), finding_refs: [...byId.values()].slice(0, 6), + ...(candidates.length ? {comparison_candidates: candidates} : {}), + ...(harness?.omitted_comparison_candidates !== undefined ? { + omitted_comparison_candidates: harness.omitted_comparison_candidates} : {}), branches: branches.slice(0, 3).map(branch => ({todo_id: branch.todo_id, selection_status: branch.selection_status, actionable_open: branch.actionable_open})), omitted_finding_refs: Math.max(0, byId.size - 6), @@ -259,7 +304,7 @@ export function projectStrideExploreBasis(value: unknown, goal: string, agent: s } /** Context binding checks shape and declared provenance, never truth of Agent claims. */ -export function strideReviewMismatch(decision: JsonObject, context: JsonObject, sourceReview?: unknown): string | null { +export function strideReviewMismatch(decision: JsonObject, context: JsonObject, sourceReview?: unknown, todoDelta?: unknown): string | null { const follow = context.decision_followthrough as JsonObject | undefined; const previous = follow?.previous_decision as JsonObject | undefined; const review = decision.previous_review as JsonObject | undefined; @@ -319,6 +364,8 @@ export function strideReviewMismatch(decision: JsonObject, context: JsonObject, } if (explore.disposition === "unavailable") return "Available Explore evidence cannot be declared unreadable."; const allowed = new Set((basis.finding_refs as JsonObject[]).map(row => row.finding_id)); + const candidates = comparisonCandidates(basis.comparison_candidates); + for (const card of candidates) for (const outcome of card.outcomes as JsonObject[]) allowed.add(outcome.finding_id); const deliveredPages = Array.isArray(context.explore_result_pages) ? context.explore_result_pages as JsonObject[] : []; for (const page of (explore.result_pages ?? []) as JsonObject[]) { const resolved = deliveredPages.find(row => row.revision === page.revision && @@ -327,8 +374,21 @@ export function strideReviewMismatch(decision: JsonObject, context: JsonObject, return "Explore result pages require current source replay; reread global pages under the original Turn."; } for (const ref of resolved.finding_refs as string[]) allowed.add(ref); + candidates.push(...comparisonCandidates(resolved.comparison_candidates)); } if (considered.some(ref => !allowed.has(ref))) return "Explore review cites an undelivered or foreign finding."; + if (explore.selected_candidate != null) { + const selected = selectedCandidate(explore.selected_candidate); + if (!candidates.some(card => card.todo_id === selected.todo_id && (card.outcomes as JsonObject[]).some(outcome => + ["node_id", "finding_id", "source_sha256"].every(key => outcome[key] === selected[key])))) { + return "Selected candidate requires the current Todo's explicit link to the exact scoped result; reread changed sources."; + } + // The existing Vision Todo plan and followthrough owner retain execution + // lineage. This optional claim cannot invent work or grant its admission. + if (!Array.isArray(todoDelta) || !todoDelta.includes(`retain:${selected.todo_id}`)) { + return "Selected candidate must be retained as the same existing Todo in agent_vision.todo_delta."; + } + } return null; } @@ -360,10 +420,20 @@ export function strideControlContract(context: JsonObject, obligationId: unknown ...(follow ? {decision_followthrough: follow} : {}), previous_review_options: {adoption: decisionSchema.adoption_options, information_gain: decisionSchema.information_options, goal_gain: decisionSchema.gain_options}, explore_basis: context.explore_basis ?? null, explore_dispositions: decisionSchema.explore_dispositions, + ...(basis?.harness_enabled === true ? { + selected_candidate_option: { + attachment: "explore_review.selected_candidate", + fields: ["todo_id", "node_id", "finding_id", "source_sha256"], + rule: "Optionally bind the next probe to an exact comparison_candidates Todo/outcome, cite that finding with disposition=used, and retain the same existing Todo in agent_vision.todo_delta. " + + "For an omitted alternative, read the existing global result pages and attach their coordinates in result_pages; current Todo links are replayed with those results. " + + "Cards share the page limit: expand --result-limit up to 20 for multiple Todos linked to one result. omitted_comparison_candidates discloses the remaining bound; omission does not mean ineligible. " + + "Choose continuation, a different mechanism or verification from the evidence; this source binding does not certify novelty, execution or utility and grants no admission.", + }, + } : {}), minimal_example: example, attachment: "agent_vision.stride_decision", requires_evidence_linked_path: true, required_on_periodic_review: true, - rule: "Every due periodic review with this strategy enabled must record a stride decision before settlement, including when a typed progress, successor, blocker or terminal outcome is also reported. " + + rule: "Every due periodic review and observed-stall review (typed_progress_repeat, no_progress_streak, repeated_action_loop or external_progress_review_drift) with this strategy enabled must record a stride decision before settlement, including when a typed progress, successor, blocker or terminal outcome is also reported. " + "The decision requires an evidence-linked Vision path (including a terminal stop when supported by evidence), an explicit grow/switch/verify action and a concrete plan/reason for each axis. " + "Grow expands a supported route; switch names the reference method and changed mechanism; verify resolves a named grow-or-switch question. Size every action for a coherent accepted deliverable or complete interpretable experiment, including ordinary engineering and maintenance. Grow changes useful completed scope, not authority or settlement boundaries. " + "The minimal example demonstrates syntax, not the recommended action for the current evidence. " + diff --git a/tests/control_plane/research_comparison_fixture.py b/tests/control_plane/research_comparison_fixture.py new file mode 100644 index 0000000000..01c7763b13 --- /dev/null +++ b/tests/control_plane/research_comparison_fixture.py @@ -0,0 +1,276 @@ +"""Controlled local Host: finite program search through real LoopX CLI owners. + +This is a deterministic experiment actor, not a model or an implementation of +LoopX's choice policy. It only reads training data; the owner verifier is a +separate program and receives withheld cases after a candidate is deployed. +""" +from __future__ import annotations + +from hashlib import sha256 +from itertools import islice, product +import json +from pathlib import Path +import random +import subprocess +import sys +import traceback + + +ROLES = ("retained", "parameter", "structural") + + +def encoded(value): + return json.dumps(value, sort_keys=True, separators=(",", ":")) + + +def simplify(op, left, right): + if isinstance(left, int) and isinstance(right, int): + return {"+": lambda: left + right, "-": lambda: left - right, "*": lambda: left * right}[op]() + if op == "+" and left == 0: + return right + if op in ("+", "-") and right == 0: + return left + if op == "*" and (left == 0 or right == 0): + return 0 + if op == "*" and left == 1: + return right + if op == "*" and right == 1: + return left + if op in ("+", "*") and encoded(left) > encoded(right): + left, right = right, left + return [op, left, right] + + +def programs(role, budget, seed): + incumbent = simplify("+", "x", "y") + if role == "retained": + yield from (incumbent for _ in range(budget)) + return + if role == "parameter": + # A fixed representation: a*x + b*y + c. No input interaction is + # available in this family, regardless of the parameter order. + coefficients = list(product(range(-2, 3), repeat=3)) + random.Random(seed).shuffle(coefficients) + for a, b, c in coefficients: + yield simplify("+", simplify("+", simplify("*", a, "x"), simplify("*", b, "y")), c) + return + assert role == "structural" + atoms = ["x", "y", -1, 0, 1] + random.Random(seed).shuffle(atoms) + by_size, seen = {1: atoms}, set() + for size in (1, 3, 5): + if size > 1: + by_size[size] = [simplify(op, left, right) + for left_size in range(1, size - 1, 2) + for left in by_size[left_size] for right in by_size[size - 1 - left_size] + for op in ("+", "-", "*")] + for program in by_size[size]: + key = encoded(program) + if key not in seen: + seen.add(key) + yield program + + +def evaluate(program, x, y): + if isinstance(program, int): + return program + if isinstance(program, str): + return {"x": x, "y": y}[program] + op, left, right = program + a, b = evaluate(left, x, y), evaluate(right, x, y) + return {"+": lambda: a + b, "-": lambda: a - b, "*": lambda: a * b}[op]() + + +def size(program): + return 1 if not isinstance(program, list) else 1 + size(program[1]) + size(program[2]) + + +def render(program): + return str(program) if not isinstance(program, list) else f"({render(program[1])}{program[0]}{render(program[2])})" + + +def cli(request, *arguments, expected=0): + result = subprocess.run([sys.executable, "-m", "loopx.cli", "--registry", request["registry"], + "--runtime-root", request["runtime"], "--format", "json", *arguments], + cwd=request["project"], capture_output=True, text=True, check=False) + assert result.returncode == expected, (arguments, result.stdout, result.stderr) + return json.loads(result.stdout) + + +def actor(request): + return ("--goal-id", request["goal_id"], "--agent-id", request["agent_id"]) + + +def guard(request, turn, *, todo=True): + return cli(request, "quota", "should-run", *actor(request), "--codex-app", "--scan-path", request["project"], + "--turn-instance-id", turn, *(("--todo-id", request["todo_id"]) if todo else ())) + + +def vision(request, evidence, *, outcome="continue"): + return {"schema_version": "goal_vision_replan_contract_v0", "state": "vision_patch_proposed", + "vision_patch": {"acceptance_summary": request["objective"]}, + "path_delta": {"schema_version": "goal_path_delta_v0", "outcome": outcome, + "prior_assumption": "The retained program may not express the observed relation.", + "observed_reality": "The bounded experiment produced an attributable result; withheld acceptance remains independent.", + "retained": ["Original owner acceptance and the best measured program."], "evidence_refs": [evidence]}} + + +def refresh(request, turn, packet, *, obligation=None): + path = Path(request["project"]) / (turn + "-vision.json") + path.write_text(json.dumps(packet)) + # The guard reconciles this exact existing Todo into the same Turn. Its + # receipt retains the due review; a planning recommendation alone is not a + # new replan-settlement identity or a reason to manufacture another Todo. + binding = ("--todo-id", request["todo_id"]) + result = cli(request, "refresh-state", *actor(request), *binding, "--turn-instance-id", turn, + "--classification", "comparison_measured_work", "--delivery-batch-scale", "implementation", + "--delivery-outcome", "outcome_progress", "--delivery-boundary", "semantic_closeout" if obligation else "in_flight_continuation", + "--agent-vision-json", str(path), "--no-global-sync", "--suppress-external-sinks") + cli(request, "quota", "spend-slot", *actor(request), *binding, "--turn-instance-id", turn, + "--slots", "1", "--source", "heartbeat", "--execute", "--scan-path", request["project"]) + return result + + +def probe(request, role): + turn = "comparison-probe-" + role + admitted = guard(request, turn) + assert admitted["should_run"] and admitted["selected_todo"]["todo_id"] == request["todo_id"], admitted + assert not admitted.get("autonomous_replan_obligation"), admitted + training = json.loads(Path(request["training"]).read_text()) + trace = [] + for program in islice(programs(role, request["budget"], request["seed"]), request["budget"]): + predicted = [evaluate(program, row["x"], row["y"]) for row in training] + trace.append({"program": program, "predicted": predicted, + "squared_error": sum((prediction - row["observed"]) ** 2 for prediction, row in zip(predicted, training))}) + assert len(trace) == request["budget"] + best = min(trace, key=lambda row: (row["squared_error"], size(row["program"]), encoded(row["program"]))) + result = {"role": role, "training_sha256": sha256(Path(request["training"]).read_bytes()).hexdigest(), + "score_calls": len(trace), "sample_evaluations": len(trace) * len(training), "trace": trace, "best": best} + artifact = Path(request["project"]) / ("experiment-" + role + ".json") + artifact.write_text(encoded(result)) + evidence = "experiment:" + sha256(artifact.read_bytes()).hexdigest() + packet = vision(request, evidence) + packet["path_delta"]["observed_reality"] = ( + f"The {role} family measured {render(best['program'])} with squared error " + f"{best['squared_error']} in {len(trace)} score calls; withheld performance is still unknown.") + packet["explore_result"] = {"schema_version": "explore_result_attachment_v0", "node_id": "comparison-" + role, + "question": "Can this bounded search family express the observed relation?", + "applicability": "Only the recorded training cases and the common scoring budget; heldout outcome is unknown.", + "input_revision": result["training_sha256"], "status": "confirmed" if best["squared_error"] == 0 else "refuted", + "observation": f"Best program {render(best['program'])} has squared error {best['squared_error']} in {len(trace)} score calls.", + "interpretation": "Compare the measured programs before adopting one; an unsuccessful representation may need a structural alternative.", + "evidence_refs": [evidence]} + written = refresh(request, turn, packet) + assert written["explore_result_delivery"]["ok"], written + return {"result": result, "writeback": written} + + +def propose(request): + pending = guard(request, "comparison-review", todo=False) + assert pending["heartbeat_receipt"]["settlement_binding_owed"] is True, pending + admitted = guard(request, "comparison-review") + assert admitted["heartbeat_receipt"]["settlement_identity"]["todo_id"] == request["todo_id"], admitted + obligation = admitted["autonomous_replan_obligation"] + contract = admitted["replan_action_packet"]["stride_control"] + current = cli(request, "explore", "turn-context", *actor(request), "--result-limit", "20") + findings = {row["finding_id"]: row for row in current["graph"]["writeback_results"]} + artifacts = {} + for role in ROLES: + path = Path(request["project"]) / ("experiment-" + role + ".json") + artifacts["experiment:" + sha256(path.read_bytes()).hexdigest()] = json.loads(path.read_text()) + eligible = [] + for card in contract["explore_basis"]["comparison_candidates"]: + for outcome in card["outcomes"]: + measured = next((artifacts[ref] for ref in findings[outcome["finding_id"]]["evidence_refs"] if ref in artifacts), None) + if measured: + eligible.append(({"todo_id": card["todo_id"], **outcome}, measured)) + assert len(eligible) == 3, eligible + choice, result = min(eligible, key=lambda pair: (pair[1]["best"]["squared_error"], size(pair[1]["best"]["program"]), ROLES.index(pair[1]["role"]))) + comparison = "; ".join(f"{item['role']} {render(item['best']['program'])}: SSE {item['best']['squared_error']}" + for _, item in sorted(eligible, key=lambda pair: ROLES.index(pair[1]["role"]))) + decision = json.loads(json.dumps(contract["minimal_example"])) + decision.update(basis="useful_new_route" if result["role"] != "retained" else "supported_progress", + action="switch" if result["role"] != "retained" else "verify") + decision["axes"] = { + "work_slice": {"plan": "Deploy and validate the measured program.", "reason": "Training alone leaves acceptance open."}, + "route_distance": {"plan": "Use " + render(result["best"]["program"]) + " from the selected comparison.", "reason": "Measured loss determines retention or change."}, + "validation_scope": {"plan": "Execute the pinned verifier on the deployed module.", "reason": "Withheld inputs check transfer."}} + decision["next_probe"] = {"scope": "One deployment and independent validation.", "prediction": "The program transfers to withheld inputs.", + "adoption_criterion": "Original owner verifier succeeds on deployed code.", + "cost_bound": "One implementation Turn.", "minimum_evidence": "Actual source, experiment provenance, acceptance receipt."} + decision["explore_review"] = {"revision": contract["explore_basis"]["revision"], "disposition": "used", + "considered_refs": [pair[0]["finding_id"] for pair in eligible], "selected_candidate": choice, + "reason": "Compare measured loss and representation size; retain the incumbent on equal evidence."} + packet = vision(request, choice["finding_id"], outcome="replan") + packet["stride_decision"], packet["todo_delta"] = decision, ["retain:" + choice["todo_id"]] + packet["path_delta"]["observed_reality"] = comparison + ". Each family used 125 score calls and 750 sample evaluations; heldout outcome is unknown." + packet["path_delta"]["changed"] = [f"Deploy {render(result['best']['program'])} from {result['role']}, then execute the unchanged owner verifier."] + return {"obligation_id": obligation["obligation_id"], "vision": packet, "selected_result": result} + + +def adopt(request): + proposal = propose(request) + written = refresh(request, "comparison-review", proposal["vision"], obligation=proposal["obligation_id"]) + assert written["autonomous_replan_ack"]["semantic_delta"]["accepted"], written + # Reread the committed source, then execute its selected artifact. A submitted + # proposal or a context card alone is not counted as implementation. + recorded = json.loads(Path(written["json_path"]).read_text())["agent_vision"] + selected = recorded["stride_decision"]["explore_review"]["selected_candidate"] + assert selected == proposal["vision"]["stride_decision"]["explore_review"]["selected_candidate"] + from loopx.goal_mode_mcp import GoalModeMCPConfig, GoalModeMCPControlPlane + control = GoalModeMCPControlPlane(GoalModeMCPConfig(server_name="comparison-fixture", runtime_profile="generic_cli", legacy_host_surface="claude_code"), + lambda: {"goal_id": request["goal_id"], "agent_id": request["agent_id"], "registry": request["registry"]}) + control.command_prefix = lambda: [sys.executable, "-m", "loopx.cli", "--runtime-root", request["runtime"]] + admitted = json.loads(control.should_run()) + assert admitted["should_run"] and admitted["selected_todo"]["todo_id"] == selected["todo_id"], admitted + artifact = Path(request["project"]) / ("experiment-" + proposal["selected_result"]["role"] + ".json") + result = json.loads(artifact.read_text()) + module = Path(request["project"]) / "predictor.py" + module.write_text("import json,sys\n\ndef predict(x,y):\n return " + render(result["best"]["program"]) + + "\n\nif __name__ == '__main__':\n print(json.dumps([predict(*pair) for pair in json.load(sys.stdin)]))\n") + training = json.loads(Path(request["training"]).read_text()) + execution = subprocess.run([sys.executable, str(module)], input=json.dumps([[row["x"], row["y"]] for row in training]), + capture_output=True, text=True, check=True) + deployed = {"selected_candidate": selected, "experiment_sha256": sha256(artifact.read_bytes()).hexdigest(), + "program": result["best"]["program"], "role": result["role"], "module_sha256": sha256(module.read_bytes()).hexdigest(), + "training_predictions": json.loads(execution.stdout)} + target = Path(request["project"]) / "deployed.json" + target.write_text(encoded(deployed)) + evidence = "deployed:" + sha256(target.read_bytes()).hexdigest() + terminal = vision(request, evidence, outcome="no_change") + terminal["state"] = "no_followup" + terminal["path_delta"]["observed_reality"] = f"Deployed {render(result['best']['program'])} from {result['role']}; actual training execution agrees. Completion will execute the unchanged independent verifier." + completed = json.loads(control.complete_task(selected["todo_id"], request["agent_id"], evidence, no_follow_up=True, agent_vision=terminal)) + assert completed["ok"], completed + verified = cli(request, "goal-acceptance", "verify", "--goal-id", request["goal_id"], "--execute") + assert verified["goal_acceptance_contract"]["status"] == "accepted", verified + # Publish the measured adoption through the existing evidence owner after + # actual verification. The shipped Explore view already links this node to + # the original Todo and shows the finding beside its completion state. + adoption = cli(request, "explore", "finding", *actor(request), "--node", selected["node_id"], + "--title", f"Adopted {render(result['best']['program'])} after independent acceptance.", "--status", "confirmed", + "--summary", proposal["vision"]["path_delta"]["observed_reality"].replace("heldout outcome is unknown", "the deployed source now passed the original heldout verifier") + + " Selection used measured loss and program size, retaining the incumbent on ties. This finite result does not establish live model creativity or broader utility.", + "--evidence-ref", evidence, "--evidence-ref", selected["finding_id"], "--tag", "comparison-adoption") + return {"proposal": proposal, "review": written, "deployment": deployed, "completion": completed, + "verification": verified, "adoption_finding": adoption} + + +def main(): + request = json.loads(Path(sys.argv[1]).read_text()) + try: + if request["stage"] == "probe": + result = probe(request, request["role"]) + elif request["stage"] == "propose": + result = propose(request) + else: + assert request["stage"] == "adopt" + result = adopt(request) + except Exception: + Path(request["response"]).write_text(json.dumps({"error": traceback.format_exc()})) + raise + Path(request["response"]).write_text(json.dumps(result)) + + +if __name__ == "__main__": + main() diff --git a/tests/control_plane/test_replan_progress_native_cli.py b/tests/control_plane/test_replan_progress_native_cli.py index 348741eb0d..ed79423bcd 100644 --- a/tests/control_plane/test_replan_progress_native_cli.py +++ b/tests/control_plane/test_replan_progress_native_cli.py @@ -205,3 +205,214 @@ def test_typed_progress_cli_preserves_replan_authority( assert authority["todos"][0]["status"] == "open" finally: restart_effect_runtime(wait_for_exit=True) + + +@pytest.mark.parametrize("provider", ["file", "sqlite"]) +@pytest.mark.parametrize("comparison_binding", [False, True], ids=["original", "source_bound"]) +def test_stalled_stride_review_drives_an_executed_mechanism_comparison( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, provider: str, + comparison_binding: bool, +) -> None: + """A real stall review reaches work and a later decision under one Goal. + + The bounded assignment problem is synthetic, with an unchanged minimum-cost + objective. It tests the control path, not benchmark or model effectiveness. + """ + isolate_sqlite_runtime(tmp_path, monkeypatch) + fixture = make_progress_cli_fixture(tmp_path, provider, "advanced") + registry = json.loads(fixture["registry"].read_text()) + registry["goals"][0]["quota"]["allowed_slots"] = 8 + fixture["registry"].write_text(json.dumps(registry)) + + def call(*args: str, expected_code: int = 0) -> dict: + return call_progress_cli(fixture, *args, expected_code=expected_code) + + def guard(turn: str, *, todo: str | None = TODO) -> dict: + return call("quota", "should-run", "--codex-app", "--goal-id", GOAL, + "--agent-id", AGENT, "--turn-instance-id", turn, + *(["--todo-id", todo] if todo else []), "--scan-path", str(fixture["project"])) + + def spend(turn: str, *, todo: str | None = TODO, obligation: str | None = None) -> dict: + return call("quota", "spend-slot", "--goal-id", GOAL, "--agent-id", AGENT, + "--turn-instance-id", turn, *(["--todo-id", todo] if todo else []), + *(["--replan-obligation-id", obligation] if obligation else []), + "--slots", "1", "--source", "heartbeat", "--execute") + + try: + call("configure-goal", "--goal-id", GOAL, "--execution-replan-after-turns", "1", + "--execution-stride-strategy", "evidence_conditioned_v1", "--execute") + if comparison_binding: + call("configure-goal", "--goal-id", GOAL, "--explore-mode", "planning", "--execute") + call("explore", "node", "--goal-id", GOAL, "--agent-id", AGENT, + "--node-id", "recorded-local-route", "--kind", "question", + "--title", "Do the existing observations establish useful progress?") + call("explore", "finding", "--goal-id", GOAL, "--agent-id", AGENT, + "--node", "recorded-local-route", "--finding-id", "finding-recorded-local", + "--title", "The recorded local observations are unchanged.", "--status", "tentative", + "--summary", "Only the fixture's recorded observations; alternate mechanisms remain untested.", + "--tag", "writeback-result", "--evidence-ref", "evidence-existing") + call("todo", "update", "--goal-id", GOAL, "--agent-id", AGENT, "--todo-id", TODO, + "--explore-result-node-ref", "recorded-local-route") + guard(TURN, todo=None) + admitted = guard(TURN) + obligation = admitted["autonomous_replan_obligation"] + assert [item["kind"] for item in obligation["triggers"]] == ["typed_progress_repeat"] + assert "fresh_vision_path_outcome" in obligation["replan_context"]["uncovered_frontier"]["required_any_of"] + assert admitted["replan_action_packet"]["uncovered_frontier"] == obligation["replan_context"]["uncovered_frontier"] + original_index = fixture["index"].read_bytes() + refused = call(*fixture["refresh"], expected_code=1) + assert refused["replan_transition"]["reason_code"] == "stride_decision_required" + assert fixture["index"].read_bytes() == original_index + + control = admitted["replan_action_packet"]["stride_control"] + decision = {**control["minimal_example"], "basis": "unexplained_gap", "action": "verify", + "axes": { + "work_slice": {"plan": "Run and compare three complete assignment methods.", + "reason": "Repeated local changes have not resolved the cost gap."}, + "route_distance": {"plan": "Compare local choice, reverse priority and joint assignment.", + "reason": "Row interactions may defeat independent greedy choices."}, + "validation_scope": {"plan": "Use the same costs and one-use-per-column constraint.", + "reason": "A changed objective cannot establish a better route."}}, + "next_probe": {"scope": "One complete bounded assignment comparison.", + "prediction": "Joint choices can beat independent local choices.", + "minimum_evidence": "Executable methods, assignments, constraint checks and costs.", + "adoption_criterion": "A valid lower-cost assignment under the original objective.", + "cost_bound": "Three methods on a three-row fixture; at most six joint assignments."}} + choice = None + if comparison_binding: + basis = control["explore_basis"] + card = next(card for card in basis["comparison_candidates"] if card["todo_id"] == TODO) + choice = {"todo_id": TODO, **card["outcomes"][0]} + assert choice["finding_id"] == "finding-recorded-local" + decision["explore_review"] = {"revision": basis["revision"], "disposition": "used", + "considered_refs": [choice["finding_id"]], "selected_candidate": choice, + "reason": "Use the recorded outcome to compare mechanisms within the original unfinished task."} + packet = {"schema_version": "goal_vision_replan_contract_v0", "state": "vision_patch_proposed", + "vision_patch": {"acceptance_summary": "The original minimum-cost validation remains open."}, + "path_delta": {"outcome": "continue", "prior_assumption": "Local changes can resolve the gap.", + "observed_reality": "Repeated observations do not establish that explanation.", + "retained": ["Original objective and unfinished validation"], + "evidence_refs": ["evidence-existing"]}, + "stride_decision": decision, "todo_delta": [f"retain:{TODO}"]} + vision = tmp_path / "comparison-review.json" + vision.write_text(json.dumps(packet)) + refresh_args = ("refresh-state", "--goal-id", GOAL, "--agent-id", AGENT, + "--todo-id", TODO, "--turn-instance-id", TURN, "--agent-vision-json", str(vision), + "--classification", "bounded_replan_progress", "--delivery-batch-scale", "single_surface", + "--delivery-outcome", "outcome_progress", "--no-global-sync", "--suppress-external-sinks") + if comparison_binding: + # Reuse the admitted current context, but falsify only the source + # digest. The typed progress exit must not hide this mismatch. + current_source = _authority(fixture) + current_state = fixture["state"].read_bytes() + wrong = json.loads(json.dumps(packet)) + digest = choice["source_sha256"] + wrong["stride_decision"]["explore_review"]["selected_candidate"]["source_sha256"] = ( + ("0" if digest[0] != "0" else "1") + digest[1:]) + vision.write_text(json.dumps(wrong)) + refused = call(*refresh_args, "--progress-result-class", "advanced", + "--progress-probe-kind", "probe-independent", "--progress-evidence-id", "evidence-current", + expected_code=1) + assert "exact scoped result" in refused["error"], refused + assert fixture["index"].read_bytes() == original_index + assert fixture["state"].read_bytes() == current_state and _authority(fixture) == current_source + vision.write_text(json.dumps(packet)) + accepted = call(*refresh_args) + assert accepted["autonomous_replan_ack"]["semantic_delta"]["satisfying_outcomes"] == ["fresh_vision_path_outcome"] + if comparison_binding: + persisted = json.loads(Path(accepted["json_path"]).read_text()) + assert persisted["agent_vision"]["stride_decision"]["explore_review"]["selected_candidate"] == choice + restart_effect_runtime(wait_for_exit=True) + assert call(*refresh_args)["appended"] is False + spend(TURN) + + work_turn = "turn-executed-route-comparison" + selected = guard(work_turn) + assert selected["selected_todo"]["todo_id"] == TODO + artifact = fixture["project"] / "assignment-comparison.json" + subprocess.run([sys.executable, "-c", """ +import itertools, json, sys +from pathlib import Path +costs = [[1, 2, 9], [1, 9, 9], [9, 1, 2]] +def greedy(order): + assignment, available = [None] * len(costs), set(range(len(costs))) + for row in order: + column = min(available, key=lambda column: (costs[row][column], column)) + assignment[row] = column + available.remove(column) + return assignment +def cost(assignment): + assert sorted(assignment) == list(range(len(costs))) + return sum(costs[row][column] for row, column in enumerate(assignment)) +assignments = {'local': greedy(range(3)), 'reverse_priority': greedy(range(2, -1, -1)), + 'joint': min(itertools.permutations(range(3)), key=cost)} +results = {name: {'assignment': list(value), 'cost': cost(value)} for name, value in assignments.items()} +winner = min(results, key=lambda name: results[name]['cost']) +Path(sys.argv[1]).write_text(json.dumps({'results': results, 'winner': winner, + 'objective': 'minimum_total_cost', 'constraint': 'each_column_once'})) +""", str(artifact)], check=True, cwd=fixture["project"], timeout=10) + measured = json.loads(artifact.read_text()) + assert measured["results"]["local"]["cost"] == 12 + assert measured["results"]["reverse_priority"]["cost"] == 11 + assert measured["results"]["joint"]["cost"] == 5 + assert measured["winner"] == "joint" + summary = "Local cost 12; reverse-priority cost 11; joint cost 5. Retain the joint result; validate transfer." + work = call("refresh-state", "--goal-id", GOAL, "--agent-id", AGENT, + "--todo-id", TODO, "--turn-instance-id", work_turn, + "--classification", "validated_progress", "--delivery-batch-scale", "implementation", + "--delivery-outcome", "outcome_progress", "--delivery-boundary", "in_flight_continuation", + "--recommended-action", summary, "--no-global-sync", "--suppress-external-sinks") + spend(work_turn) + + next_turn = "turn-review-executed-comparison" + next_guard = guard(next_turn, todo=None) + next_obligation = next_guard["autonomous_replan_obligation"] + follow = next_obligation["replan_context"]["decision_followthrough"] + planned = next(item for item in follow["planned_work"] if item["todo_id"] == TODO) + result_ref = planned["ordinary_evidence_refs"][0] + readback = call("history", "--goal-id", GOAL, "--agent-id", AGENT, + "--evidence-ref", result_ref)["evidence"] + assert readback["recommended_action"] == summary + assert readback["generated_at"] == json.loads(Path(work["json_path"]).read_text())["generated_at"] + assert follow["execution_and_goal_gain"] == "unknown", "Synthetic provenance is not authoritative transfer" + if comparison_binding: + assert follow["previous_decision"]["stride_decision"]["explore_review"]["selected_candidate"] == choice + + # The measured choice changes the next executable work, not just prose. + successor = call("todo", "add", "--goal-id", GOAL, "--role", "agent", "--claimed-by", AGENT, + "--text", f"Validate transfer of {measured['winner']} assignment on independent costs", + "--task-class", "advancement_task", "--action-kind", "validate", + "--target-key", "joint-assignment-transfer", "--replan-obligation-id", next_obligation["obligation_id"]) + next_guard = guard(next_turn, todo=successor["todo_id"]) + next_control = next_guard["replan_action_packet"]["stride_control"] + next_decision = {**decision, "action": "switch", "basis": "useful_new_route", + "reference_context_id": next_control["reference_context_id"], + "reference_obligation_id": next_obligation["obligation_id"], + "previous_review": {"reference_decision_ref": follow["previous_decision"]["evidence_ref"], + "adoption": "executed", "information_gain": "new_evidence", "goal_gain": "unknown", + "evidence_refs": [result_ref], "lesson": "Joint choices improved this fixture; transfer remains unproven."}} + next_decision["axes"] = {**decision["axes"], "route_distance": { + "plan": "Carry joint assignment into an independent transfer check.", + "reason": "The paired experiment exposed the local-choice limitation."}} + if comparison_binding: + # B is new work justified by the actual comparison. The optional + # retained-A choice is not a routing command or evidence of B. + next_decision["explore_review"] = {"revision": next_control["explore_basis"]["revision"], + "disposition": "deferred", "considered_refs": [], + "reason": "Transfer remains untested; the next work uses the executed comparison's history evidence."} + packet = {**packet, "stride_decision": next_decision, "todo_delta": [f"create:{successor['todo_id']}"]} + vision.write_text(json.dumps(packet)) + adopted = call("refresh-state", "--goal-id", GOAL, "--agent-id", AGENT, + "--replan-obligation-id", next_obligation["obligation_id"], "--turn-instance-id", next_turn, + "--agent-vision-json", str(vision), "--classification", "bounded_replan_progress", + "--delivery-batch-scale", "single_surface", "--delivery-outcome", "outcome_progress", + "--no-global-sync", "--suppress-external-sinks") + assert adopted["autonomous_replan_ack"]["semantic_delta"]["accepted"] is True + spend(next_turn, todo=None, obligation=next_obligation["obligation_id"]) + if comparison_binding: + spent = [json.loads(line) for line in fixture["index"].read_text().splitlines() + if json.loads(line).get("quota_spend_commit")] + assert len(spent) == 3 + assert {row["turn_instance_id"] for row in spent} == {TURN, work_turn, next_turn} + finally: + restart_effect_runtime(wait_for_exit=True) diff --git a/tests/control_plane/test_research_comparison_cli.py b/tests/control_plane/test_research_comparison_cli.py new file mode 100644 index 0000000000..fa01876f26 --- /dev/null +++ b/tests/control_plane/test_research_comparison_cli.py @@ -0,0 +1,193 @@ +"""Evidence changes actual selected programs under real CLI/worker/acceptance. + +The local Host performs finite symbolic search, not a live provider call. The +retained route, parameter neighbourhood and structural grammar have the same +scoring budget. The oracle independently executes generated source on withheld +inputs and never supplies those inputs to the search or choice actor. +""" +from __future__ import annotations + +from hashlib import sha256 +import json +from pathlib import Path +import shlex +import sys + +import pytest + +from canonical_authority_fixture import isolate_sqlite_runtime +import test_quota_settlement_cli as fixture +from test_external_scheduler_worker import _args, worker + + +HOST = Path(__file__).with_name("research_comparison_fixture.py") +ROLES = ("retained", "parameter", "structural") +OBJECTIVE = "Deploy a program that explains the observed relation and passes independent withheld cases under the original bounded study." + +# A separate evaluator: it compiles the generated module and reconstructs the +# recorded expressions with Python's AST, without importing the search actor. +ORACLE = r'''import ast,hashlib,importlib.util,json +from pathlib import Path + +def expression(program): + if type(program) is int: + return ast.Constant(program) + if program in ("x","y"): + return ast.Name(id=program,ctx=ast.Load()) + assert isinstance(program,list) and len(program)==3 + operation={"+":ast.Add,"-":ast.Sub,"*":ast.Mult}[program[0]] + return ast.BinOp(left=expression(program[1]),op=operation(),right=expression(program[2])) + +def compiled(program): + return compile(ast.fix_missing_locations(ast.Expression(body=expression(program))),"recorded-expression","eval") + +training=json.loads(Path("training.json").read_text()) +study=json.loads(Path("study.json").read_text()) +deployed=json.loads(Path("deployed.json").read_text()) +spec=importlib.util.spec_from_file_location("actual_deployment",Path("predictor.py")) +module=importlib.util.module_from_spec(spec) +spec.loader.exec_module(module) +assert deployed["module_sha256"]==hashlib.sha256(Path("predictor.py").read_bytes()).hexdigest() +assert all(module.predict(row["x"],row["y"])==row["observed"] for row in study["withheld"]) +assert deployed["training_predictions"]==[module.predict(row["x"],row["y"]) for row in training] +candidate=compiled(deployed["program"]) +assert all(eval(candidate,{"__builtins__":{}},{"x":row["x"],"y":row["y"]})==module.predict(row["x"],row["y"]) + for row in study["withheld"]) +measured=[] +for role in ("retained","parameter","structural"): + path=Path("experiment-"+role+".json") + result=json.loads(path.read_text()) + assert result["training_sha256"]==hashlib.sha256(Path("training.json").read_bytes()).hexdigest() + assert result["score_calls"]==len(result["trace"])==study["score_budget"] + assert result["sample_evaluations"]==study["score_budget"]*len(training) + for row in result["trace"]: + executable=compiled(row["program"]) + predicted=[eval(executable,{"__builtins__":{}},{"x":case["x"],"y":case["y"]}) for case in training] + assert row["predicted"]==predicted + assert row["squared_error"]==sum((value-case["observed"])**2 for value,case in zip(predicted,training)) + assert result["best"] in result["trace"] + assert result["best"]["squared_error"]==min(row["squared_error"] for row in result["trace"]) + measured.append(result["best"]["squared_error"]) + if role==deployed["role"]: + assert deployed["experiment_sha256"]==hashlib.sha256(path.read_bytes()).hexdigest() + assert deployed["program"]==result["best"]["program"] + selected_loss=result["best"]["squared_error"] +assert selected_loss==min(measured) +with Path("oracle-executions.jsonl").open("a") as stream: + stream.write(json.dumps({"withheld_cases":len(study["withheld"]),"module_sha256":deployed["module_sha256"]})+"\n") +''' + + +@pytest.mark.parametrize("provider", ["file", "sqlite"]) +@pytest.mark.parametrize("relation", ["interaction", "additive"]) +def test_measured_comparison_generates_and_adopts_a_program_with_original_acceptance(tmp_path, monkeypatch, provider, relation): + isolate_sqlite_runtime(tmp_path, monkeypatch) + monkeypatch.setenv("LOOPX_USAGE_PING", "0") + monkeypatch.setenv("PYTHONPATH", str(fixture.REPO_ROOT)) + project, runtime, registry = fixture._write_fixture(tmp_path) + monkeypatch.chdir(project) + config = json.loads(registry.read_text()) + config["goals"][0]["quota"]["allowed_slots"] = 16 + registry.write_text(json.dumps(config)) + fixture._initialize_fixture_authority(project, runtime, provider=provider) + + def cli(*arguments, code=0): + observed, packet = fixture._run_cli(registry, runtime, *arguments, cwd=project) + assert observed == code, json.dumps(packet) + return packet + + cli("configure-goal", "--goal-id", fixture.GOAL_ID, "--explore-mode", "planning", + "--execution-replan-after-turns", "3", "--execution-stride-strategy", "evidence_conditioned_v1", "--execute") + # The owner defines three search families and a complete implementation + # task, not a winning expression or a prewritten next-route Todo. Links are + # part of that initial work declaration; later result capture adds evidence. + cli("todo", "update", "--goal-id", fixture.GOAL_ID, "--todo-id", fixture.TODO_ID, + "--agent-id", fixture.AGENT_ID, "--text", OBJECTIVE, + *[value for role in ROLES for value in ("--explore-result-node-ref", "comparison-" + role)]) + # Both relations use the same search/data order for a provider. The second + # provider independently reverses training order and changes grammar order; + # neither relation nor withheld outcomes enter the actor's route choice. + seed = 17 if provider == "file" else 18 + pairs = [(-2, -1), (-1, 2), (0, 3), (1, -2), (2, 1), (3, 0)] + if seed % 2: + pairs.reverse() + operation = (lambda x, y: x * y) if relation == "interaction" else (lambda x, y: x + y) + training = [{"x": x, "y": y, "observed": operation(x, y)} for x, y in pairs] + withheld = [{"x": x, "y": y, "observed": operation(x, y)} for x, y in [(-7, 4), (5, -6), (11, 13), (0, -9)]] + (project / "training.json").write_text(json.dumps(training)) + (project / "study.json").write_text(json.dumps({"score_budget": 125, "withheld": withheld})) + (project / "verify.py").write_text(ORACLE) + document = {"scope": {"kind": "all_advancement"}, "objective": OBJECTIVE, + "non_goals": ["No claim about live model creativity or performance outside this finite arithmetic study."], + "criteria": [{"id": "transfer", "description": "Actual deployed source passes withheld inputs and preserves the bounded comparison evidence.", + "validation_argv": [sys.executable, "verify.py"], "validation_timeout_seconds": 10, + "validation_files": [{"path": name, "sha256": sha256((project / name).read_bytes()).hexdigest()} + for name in ("verify.py", "study.json", "training.json")]}], + "bindings": [{"todo_id": fixture.TODO_ID, "criterion_ids": ["transfer"]}]} + document_path = project / "requirements.json" + document_path.write_text(json.dumps(document)) + initial = cli("goal-acceptance", "inspect", "--goal-id", fixture.GOAL_ID) + configured = cli("goal-acceptance", "configure", "--goal-id", fixture.GOAL_ID, "--document", str(document_path), + "--expected-provider-revision", initial["provider_revision"], "--operation-id", "owner-frozen-study", "--execute")["goal_acceptance_contract"] + args = _args(fake_cli=Path(sys.executable).parent / "loopx", state_file=tmp_path / "worker.json", + quota_timeout_seconds=90, wake_timeout_seconds=180) + args.registry, args.runtime_root = str(registry), str(runtime) + args.goal_id, args.agent_id = fixture.GOAL_ID, fixture.AGENT_ID + request = {"project": str(project), "registry": str(registry), "runtime": str(runtime), + "goal_id": fixture.GOAL_ID, "agent_id": fixture.AGENT_ID, "todo_id": fixture.TODO_ID, + "objective": OBJECTIVE, "training": str(project / "training.json"), "budget": 125, "seed": seed} + + def run_host(stage, role=""): + key = stage + ("-" + role if role else "") + response, path = project / (key + "-response.json"), project / (key + "-request.json") + path.write_text(json.dumps({**request, "stage": stage, "role": role, "response": str(response)})) + args.wake_cmd = shlex.join([sys.executable, str(HOST), str(path)]) + code = worker.run_worker(args) + assert code == 0, response.read_text() if response.exists() else key + assert response.exists(), "The authorized worker did not execute the controlled Host." + return json.loads(response.read_text()) + + results = {role: run_host("probe", role)["result"] for role in ROLES} + assert {row["score_calls"] for row in results.values()} == {125} + assert {row["sample_evaluations"] for row in results.values()} == {750} + assert len({json.dumps(row["program"]) for row in results["structural"]["trace"]}) == 125 + assert not (project / "deployed.json").exists() and not (project / "oracle-executions.jsonl").exists() + if relation == "interaction": + assert results["retained"]["best"]["squared_error"] > 0 + assert results["parameter"]["best"]["squared_error"] > 0 + else: + assert results["retained"]["best"]["squared_error"] == 0 + adopted = run_host("adopt") + deployed = adopted["deployment"] + expected_role = "structural" if relation == "interaction" else "retained" + assert deployed["role"] == expected_role + # Neither the Todo nor the Host request contains the expected winning + # program; the actual evaluated source selected from current cards does. + assert deployed["program"] == results[expected_role]["best"]["program"] + assert deployed["selected_candidate"] == adopted["proposal"]["vision"]["stride_decision"]["explore_review"]["selected_candidate"] + assert adopted["review"]["autonomous_replan_ack"]["semantic_delta"]["accepted"] + final = cli("goal-acceptance", "inspect", "--goal-id", fixture.GOAL_ID)["goal_acceptance_contract"] + assert final["status"] == "accepted" and final["revision"] == configured["revision"] + assert final["digest"] == configured["digest"] + # The existing adapter's completion/terminal readback runs two validation + # checks; the explicit original Goal verification runs the third. Count + # all of them instead of hiding validation cost behind one completion. + assert len((project / "oracle-executions.jsonl").read_text().splitlines()) == 3 + current = cli("todo", "list", "--goal-id", fixture.GOAL_ID, "--todo-id", fixture.TODO_ID)["todo"] + assert current["status"] == "done" and current["no_followup"] is True + before = (runtime / "goals" / fixture.GOAL_ID / "runs/index.jsonl").read_bytes() + rows = [json.loads(line) for line in before.splitlines()] + spent = [row for row in rows if row.get("quota_spend_commit")] + work = [row for row in rows if not row.get("quota_spend_commit")] + expected_turns = {"comparison-review", *["comparison-probe-" + role for role in ROLES], + adopted["completion"]["settlement_identity"]["turn_instance_id"]} + assert len(rows) == 10 and len(work) == len(spent) == len(expected_turns) == 5 + assert {row["turn_instance_id"] for row in work} == expected_turns + assert {row["turn_instance_id"] for row in spent} == expected_turns + assert {row["todo_id"] for row in rows} == {fixture.TODO_ID} + for _ in range(2): + assert worker.run_worker(args) == 0 + after = (runtime / "goals" / fixture.GOAL_ID / "runs/index.jsonl").read_bytes() + assert after == before, "A completed finite comparison must not dispatch or settle another research Turn." + final_guard = cli("quota", "should-run", "--goal-id", fixture.GOAL_ID, "--agent-id", fixture.AGENT_ID) + assert final_guard["should_run"] is False diff --git a/tests/control_plane/test_stride_comparison_followthrough_cli.py b/tests/control_plane/test_stride_comparison_followthrough_cli.py new file mode 100644 index 0000000000..9ea2bde6f7 --- /dev/null +++ b/tests/control_plane/test_stride_comparison_followthrough_cli.py @@ -0,0 +1,149 @@ +"""Exact comparison provenance is planning intent, not execution authority. + +Exercise a real canonical B successor beside a selected/retained A decision. +Subsequent B work must remain B work and cannot substantiate execution of A. +""" +from __future__ import annotations + +from copy import deepcopy +import json +from pathlib import Path +import sys + +import pytest + +from canonical_authority_fixture import isolate_sqlite_runtime +import research_comparison_fixture as actor +import test_quota_settlement_cli as fixture + + +@pytest.mark.parametrize("provider", ["file", "sqlite"]) +def test_current_source_replay_and_different_successor_do_not_certify_selected_work(tmp_path, monkeypatch, provider): + isolate_sqlite_runtime(tmp_path, monkeypatch) + monkeypatch.setenv("LOOPX_USAGE_PING", "0") + monkeypatch.setenv("PYTHONPATH", str(fixture.REPO_ROOT)) + project, runtime, registry = fixture._write_fixture(tmp_path) + monkeypatch.chdir(project) + config = json.loads(registry.read_text()) + config["goals"][0]["quota"]["allowed_slots"] = 12 + registry.write_text(json.dumps(config)) + fixture._initialize_fixture_authority(project, runtime, provider=provider) + + def cli(*arguments, expected=0): + code, packet = fixture._run_cli(registry, runtime, *arguments, cwd=project) + assert code == expected, json.dumps(packet) + return packet + + identity = ("--goal-id", fixture.GOAL_ID, "--agent-id", fixture.AGENT_ID) + cli("configure-goal", "--goal-id", fixture.GOAL_ID, "--explore-mode", "planning", + "--execution-replan-after-turns", "1", "--execution-stride-strategy", "evidence_conditioned_v1", "--execute") + cli("todo", "update", *identity, "--todo-id", fixture.TODO_ID, + "--explore-result-node-ref", "comparison-retained") + training = project / "training.json" + training.write_text(json.dumps([{"x": x, "y": y, "observed": x * y} for x, y in [(-2, -1), (1, -2)]])) + request = {"project": str(project), "registry": str(registry), "runtime": str(runtime), + "goal_id": fixture.GOAL_ID, "agent_id": fixture.AGENT_ID, "todo_id": fixture.TODO_ID, + "objective": "Keep the comparison's current source distinct from later unrelated work.", + "training": str(training), "budget": 125, "seed": 17} + actor.probe(request, "retained") + + def guard(turn, todo=None): + return cli("quota", "should-run", *identity, "--codex-app", "--scan-path", str(project), + "--turn-instance-id", turn, *(("--todo-id", todo) if todo else ())) + + turn = "comparison-a-plan-b-successor" + initial = guard(turn) + obligation = initial["autonomous_replan_obligation"] + assert {row["kind"] for row in obligation["triggers"]} == {"periodic_review_due"} + old_choice = {"todo_id": fixture.TODO_ID, + **initial["replan_action_packet"]["stride_control"]["explore_basis"]["comparison_candidates"][0]["outcomes"][0]} + # This is an actual current-source update through the public Explore owner, + # with the same finding id. A previously read hash must not survive it. + cli("explore", "finding", *identity, "--finding-id", old_choice["finding_id"], + "--node", old_choice["node_id"], "--title", "The retained program fails the recorded interaction cases.", + "--summary", "Updated applicability: only these measured inputs; a later independent task is not this experiment.", + "--status", "refuted", "--tag", "writeback-result", "--evidence-ref", "measured:retained-inputs") + successor = cli("todo", "add", "--goal-id", fixture.GOAL_ID, "--role", "agent", + "--claimed-by", fixture.AGENT_ID, "--text", "Independently measure another input relation.", + "--task-class", "advancement_task", "--action-kind", "validate", "--target-key", "independent-b", + "--replan-obligation-id", obligation["obligation_id"]) + other_id = successor["todo_id"] + assert other_id != fixture.TODO_ID + bound = guard(turn, fixture.TODO_ID) + assert bound["heartbeat_receipt"]["settlement_identity"]["todo_id"] == fixture.TODO_ID + assert bound["autonomous_replan_obligation"]["obligation_id"] == obligation["obligation_id"] + control = bound["replan_action_packet"]["stride_control"] + card = next(card for card in control["explore_basis"]["comparison_candidates"] if card["todo_id"] == fixture.TODO_ID) + current_choice = {"todo_id": fixture.TODO_ID, **card["outcomes"][0]} + assert current_choice["source_sha256"] != old_choice["source_sha256"] + decision = deepcopy(control["minimal_example"]) + decision["explore_review"] = {"revision": control["explore_basis"]["revision"], "disposition": "used", + "considered_refs": [current_choice["finding_id"]], "selected_candidate": old_choice, + "reason": "Keep exact source provenance for the retained A plan."} + packet = actor.vision(request, current_choice["finding_id"], outcome="replan") + packet.update(stride_decision=decision, todo_delta=["retain:" + fixture.TODO_ID]) + packet["path_delta"]["changed"] = ["A remains the declared next probe; the separate B task does not count as its execution."] + path = project / "review-a.json" + index = runtime / "goals" / fixture.GOAL_ID / "runs/index.jsonl" + + def review(value, turn_id=turn, *, expected=0): + path.write_text(json.dumps(value)) + return cli("refresh-state", *identity, "--todo-id", fixture.TODO_ID, "--turn-instance-id", turn_id, + "--classification", "source_bound_comparison", "--delivery-batch-scale", "implementation", + "--delivery-outcome", "outcome_progress", "--agent-vision-json", str(path), + "--no-global-sync", "--suppress-external-sinks", expected=expected) + + before = index.read_bytes() + refused = review(packet, expected=1) + assert not refused.get("appended") + assert "exact scoped result" in refused["error"], refused + assert index.read_bytes() == before + packet["stride_decision"]["explore_review"]["selected_candidate"] = current_choice + accepted = review(packet) + delta = accepted["autonomous_replan_ack"]["semantic_delta"] + assert delta["accepted"] and "fresh_vision_path_outcome" in delta["satisfying_outcomes"] + persisted = json.loads(Path(accepted["json_path"]).read_text()) + assert persisted["agent_vision"]["todo_delta"] == ["retain:" + fixture.TODO_ID] + assert persisted["agent_vision"]["stride_decision"]["explore_review"]["selected_candidate"] == current_choice + cli("quota", "spend-slot", *identity, "--todo-id", fixture.TODO_ID, "--turn-instance-id", turn, + "--slots", "1", "--source", "heartbeat", "--execute") + + # B receives its own normal admission, performs actual work, then settles. + # The optional A selection never reserves or silently redirects execution. + actual_turn = "actual-independent-b" + actual = guard(actual_turn, other_id) + assert actual["should_run"] and actual["selected_todo"]["todo_id"] == other_id + import subprocess + artifact = project / "actual-b.json" + subprocess.run([sys.executable, "-c", "import json,sys;from pathlib import Path;" + "Path(sys.argv[1]).write_text(json.dumps({'measured_sum':sum(x*x for x in [-2,0,3])}))", str(artifact)], check=True) + assert json.loads(artifact.read_text())["measured_sum"] == 13 + cli("refresh-state", *identity, "--todo-id", other_id, "--turn-instance-id", actual_turn, + "--classification", "independent_b_experiment", "--delivery-batch-scale", "implementation", + "--delivery-outcome", "outcome_progress", "--delivery-boundary", "in_flight_continuation", + "--progress-result-class", "advanced", "--progress-surface-id", "independent-b", + "--progress-probe-kind", "sum-of-squares", "--progress-evidence-id", "measured:actual-b", + "--no-global-sync", "--suppress-external-sinks") + cli("quota", "spend-slot", *identity, "--todo-id", other_id, "--turn-instance-id", actual_turn, + "--slots", "1", "--source", "heartbeat", "--execute") + next_turn = "review-a-execution-claim" + guard(next_turn) + next_guard = guard(next_turn, fixture.TODO_ID) + next_control = next_guard["replan_action_packet"]["stride_control"] + follow = next_control["decision_followthrough"] + assert follow["previous_decision"]["stride_decision"]["explore_review"]["selected_candidate"] == current_choice + assert follow["execution_and_goal_gain"] == "unknown" + [planned] = follow["planned_work"] + assert planned["todo_id"] == fixture.TODO_ID + assert planned["observation_refs"] == [] and not planned.get("ordinary_evidence_refs") + observed_b = next(row for row in follow["observations"] if row["todo_id"] == other_id) + next_packet = actor.vision(request, observed_b["evidence_ref"], outcome="replan") + next_decision = deepcopy(next_control["minimal_example"]) + next_decision["previous_review"] = {"reference_decision_ref": follow["previous_decision"]["evidence_ref"], + "adoption": "executed", "information_gain": "new_evidence", "goal_gain": "unknown", + "evidence_refs": [observed_b["evidence_ref"]], "lesson": "Attempt to pass off B's measurement as A's execution."} + next_packet.update(stride_decision=next_decision, todo_delta=["retain:" + fixture.TODO_ID]) + before = index.read_bytes() + refused = review(next_packet, next_turn, expected=1) + assert "declared work" in refused["error"] and not refused.get("appended"), refused + assert index.read_bytes() == before diff --git a/tests/control_plane_ts/replan_context.test.ts b/tests/control_plane_ts/replan_context.test.ts index e7413a2259..d85a28e524 100644 --- a/tests/control_plane_ts/replan_context.test.ts +++ b/tests/control_plane_ts/replan_context.test.ts @@ -491,6 +491,22 @@ test("another lane or retired Goal instance never supplies current decision evid .decision_followthrough as JsonObject).previous_decision as JsonObject).goal_ref, currentRef); }); +test("stall frontier projects its current strategy through the shared outcome owner", () => { + for (const kind of ["typed_progress_repeat", "no_progress_streak", "repeated_action_loop"]) { + const obligation = {...request.obligation, triggers: [{kind}], + replan_context: {stride_strategy: "off"}}; + const enabled = projectReplanContext({...request, obligation, stride_strategy: "evidence_conditioned_v1"}); + assert.ok(((enabled.uncovered_frontier as JsonObject).required_any_of as string[]) + .includes("fresh_vision_path_outcome")); + const off = projectReplanContext({...request, obligation, stride_strategy: "off"}); + assert.equal(((off.uncovered_frontier as JsonObject).required_any_of as string[]) + .includes("fresh_vision_path_outcome"), false); + const restricted = projectReplanContext({...request, stride_strategy: "evidence_conditioned_v1", + obligation: {...obligation, satisfying_semantic_outcomes: ["new_runnable_successor"]}}); + assert.deepEqual((restricted.uncovered_frontier as JsonObject).required_any_of, ["new_runnable_successor"]); + } +}); + test("strategy off preserves context identity and selection despite decision transport facts", () => { const rows = [reviewed(), subsequent(1)]; const ordinary = projectReplanContext({...request, rows}); diff --git a/tests/control_plane_ts/replan_semantics.test.ts b/tests/control_plane_ts/replan_semantics.test.ts index 9f129eaf83..7dd1e30361 100644 --- a/tests/control_plane_ts/replan_semantics.test.ts +++ b/tests/control_plane_ts/replan_semantics.test.ts @@ -344,7 +344,8 @@ test("versioned stride advice is opt-in and never a semantic discharge", () => { const withStrategy = projectReplanSemantics({operation: "requirements", obligation: enabled}); assert.deepEqual((withStrategy.planning_guidance as string[]).slice(0, 5), base.planning_guidance); assert.equal((withStrategy.planning_guidance as string[]).length, 6); - assert.deepEqual(withStrategy.required_any_of, base.required_any_of); + assert.deepEqual(withStrategy.required_any_of, + ["fresh_vision_path_outcome", ...(base.required_any_of as string[])]); assert.match((withStrategy.planning_guidance as string[])[5], /work size, route distance and validation breadth independently/); assert.match((withStrategy.planning_guidance as string[])[5], /Do not restart unfinished coherent work/); assert.equal(projectReplanSemantics({operation: "qualify", obligation: enabled, diff --git a/tests/control_plane_ts/stride_comparison_selection.test.ts b/tests/control_plane_ts/stride_comparison_selection.test.ts new file mode 100644 index 0000000000..4bb8af244c --- /dev/null +++ b/tests/control_plane_ts/stride_comparison_selection.test.ts @@ -0,0 +1,223 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import type {JsonObject} from "../../loopx/control_plane/effect_program.ts"; +import {projectExploreTurnContext} from "../../loopx/control_plane/capabilities/explore_turn_context.ts"; +import {projectReplanContext} from "../../loopx/control_plane/work_items/replan_context.ts"; +import {normalizeStrideDecision, strideControlContract, strideReviewMismatch} from "../../loopx/control_plane/work_items/stride_policy.ts"; +import {projectReplanSemantics} from "../../loopx/control_plane/work_items/replan_semantics.ts"; + +const goal = "route-comparison", agent = "researcher", obligationId = "replan-" + "a".repeat(16); +const findings = [ + {finding_id: "finding-local", node_id: "local-method", finding: "The local method misses the interaction.", + status: "refuted", summary: "The measured result applies only to input-v1 and the tested local search.", + evidence_refs: ["measurement:local-v1"], tags: ["writeback-result"]}, + {finding_id: "finding-joint", node_id: "joint-method", finding: "The joint method resolves the tested interaction.", + status: "confirmed", summary: "The measured result applies only to input-v1; independent validation remains.", + evidence_refs: ["measurement:joint-v1"], tags: ["writeback-result"]}, +]; +const branches = findings.map((finding, i) => ({todo_id: `todo_method_${i}`, text: `Validate mechanism ${i}.`, + status: "open", actionable_open: true, selection_status: "selected", score: 100 - i, + typed_evidence_audit: {mode: "diagnostic_only", score_delta: 0, + requested_node_refs: [finding.node_id], unknown_node_refs: [], + nodes: [{node_id: finding.node_id, status: "exploring"}], findings: [finding], relevant_edges: []}})); +const input: JsonObject = {goal_id: goal, agent_id: agent, route: ["loopx"], graph_enabled: true, + harness_gate: {enabled: true}, plan: {selected_branches: branches}, + projection: {nodes: findings.map(row => ({node_id: row.node_id, status: "exploring"})), findings}, + include_finding_digests: true}; + +function context(source = projectExploreTurnContext(input)): JsonObject { + return projectReplanContext({operation: "project", goal_id: goal, agent_id: agent, rows: [], goal_facts: {}, + obligation: {obligation_id: obligationId}, stride_strategy: "evidence_conditioned_v1", explore_context: source}); +} +function selected(ctx: JsonObject): JsonObject { + const cards = (ctx.explore_basis as JsonObject).comparison_candidates as JsonObject[]; + const card = cards.find(row => row.todo_id === "todo_method_1")!; + return {todo_id: card.todo_id, ...(card.outcomes as JsonObject[])[0]}; +} +function decision(ctx: JsonObject): JsonObject { + const choice = selected(ctx); + return {...strideControlContract(ctx, obligationId)!.minimal_example as JsonObject, + explore_review: {revision: (ctx.explore_basis as JsonObject).revision, disposition: "used", + considered_refs: [choice.finding_id], reason: "Compare the measured mechanism under unchanged acceptance.", + selected_candidate: choice}}; +} +function qualify(ctx: JsonObject, proposal: JsonObject, todoDelta = ["retain:todo_method_1"]): JsonObject { + return projectReplanSemantics({operation: "qualify", obligation: {obligation_id: obligationId, + triggers: [{kind: "periodic_review_due"}], replan_context: ctx}, agent_vision: { + stride_decision: proposal, todo_delta: todoDelta, vision_patch: {acceptance_summary: "Original acceptance remains."}, + path_delta: {outcome: "replan", evidence_refs: ["measurement:joint-v1"]}}, + observation_delta: {delta_kinds: ["new_runnable_successor"]}}); +} + +test("admitted stride context retains executable candidate links to exact scoped outcomes", () => { + const source = projectExploreTurnContext(input), ctx = context(source); + const cards = (ctx.explore_basis as JsonObject).comparison_candidates as JsonObject[]; + assert.equal(cards?.length, 2, "The executable comparison relation must survive the replan projection."); + assert.deepEqual(cards.map(row => row.todo_id), ["todo_method_0", "todo_method_1"]); + assert.equal((cards[0].outcomes as JsonObject[])[0].node_id, "local-method"); + assert.equal((cards[1].outcomes as JsonObject[])[0].node_id, "joint-method"); + assert.match(String(selected(ctx).source_sha256), /^[0-9a-f]{64}$/); + assert.equal((source.harness as JsonObject).selected_branches instanceof Array, true); + assert.deepEqual(((source.harness as JsonObject).selected_branches as JsonObject[]).map(row => row.score), [100, 99]); + assert.equal(qualify(ctx, decision(ctx)).accepted, true); +}); + +test("candidate selection binds current Todo linkage, source content and retained planned work", () => { + const ctx = context(), proposal = decision(ctx), choice = selected(ctx); + for (const patch of [{todo_id: "todo_other"}, {node_id: "local-method"}, + {finding_id: "finding-local"}, {source_sha256: "b".repeat(64)}]) { + const changed = {...proposal, explore_review: {...proposal.explore_review as JsonObject, + considered_refs: [patch.finding_id ?? choice.finding_id], + selected_candidate: {...choice, ...patch}}}; + assert.equal(qualify(ctx, changed).accepted, false, JSON.stringify(patch)); + } + for (const todoDelta of [[], ["retain:todo_other"], ["The chosen method will run."]]) { + assert.equal(qualify(ctx, proposal, todoDelta).accepted, false); + } + const changedSource = projectExploreTurnContext({...input, projection: {...input.projection as JsonObject, + findings: findings.map(row => row.finding_id === choice.finding_id ? {...row, + summary: "New input-v2 contradicts the prior applicability."} : row)}}); + assert.equal(qualify(context(changedSource), proposal).accepted, false); + const unlinked = projectExploreTurnContext({...input, plan: {selected_branches: branches.map(row => ({...row, + typed_evidence_audit: {...row.typed_evidence_audit, requested_node_refs: ["unknown"], unknown_node_refs: ["unknown"]}}))}}); + assert.equal(qualify(context(unlinked), proposal).accepted, false); + for (const patch of [{actionable_open: false}, {claimed_by: "other-agent"}]) { + const closed = projectExploreTurnContext({...input, plan: {selected_branches: branches.map(row => ({...row, ...patch}))}}); + assert.equal((closed.harness as JsonObject).comparison_candidates, undefined); + assert.equal(qualify(context(closed), proposal).accepted, false); + } +}); + +for (const kind of ["periodic_review_due", "typed_progress_repeat", "no_progress_streak", + "repeated_action_loop", "external_progress_review_drift"]) { + test(`${kind} combines current comparison evidence with its required decision`, () => { + const ctx = context(), proposal = decision(ctx), choice = selected(ctx); + const obligation = {obligation_id: obligationId, triggers: [{kind}], replan_context: ctx}; + const vision = {stride_decision: proposal, todo_delta: ["retain:todo_method_1"], + vision_patch: {acceptance_summary: "Original acceptance remains."}, + path_delta: {outcome: "continue", evidence_refs: ["measurement:joint-v1"]}}; + const request = {operation: "qualify", obligation, agent_vision: vision}; + const accepted = projectReplanSemantics(request); + assert.deepEqual(accepted.satisfying_outcomes, ["fresh_vision_path_outcome"], + "Current evidence may justify finishing useful work without inventing a successor."); + const successor = {delta_kinds: ["new_runnable_successor"]}; + const wrongSource = {...proposal, explore_review: {...proposal.explore_review as JsonObject, + selected_candidate: {...choice, source_sha256: "b".repeat(64)}}}; + for (const change of [ + {stride_decision: wrongSource}, + {stride_decision: {...proposal, reference_context_id: "replan-context-" + "b".repeat(24)}}, + {stride_decision: {...proposal, reference_obligation_id: "replan-" + "b".repeat(16)}}, + {todo_delta: ["retain:todo_other"]}, + {stride_decision: undefined}, + ]) { + assert.equal(projectReplanSemantics({...request, observation_delta: successor, + agent_vision: {...vision, ...change}}).accepted, false, + "A typed successor cannot bypass either the current decision or its explicit source binding."); + } + assert.equal(projectReplanSemantics({operation: "transition_eligibility", obligation}).allowed, false); + assert.equal(projectReplanSemantics({...request, obligation: {...obligation, + satisfying_semantic_outcomes: ["new_runnable_successor"]}}).accepted, false, + "An explicit source restriction still cannot be satisfied by a fresh path alone."); + const off = {...obligation, replan_context: {stride_strategy: "off"}}; + assert.equal(projectReplanSemantics({operation: "qualify", obligation: off, + observation_delta: successor}).accepted, true); + assert.equal(projectReplanSemantics({operation: "transition_eligibility", obligation: off}).allowed, true); + }); +} + +test("explicit selection remains optional and cannot be turned into authority or a forced pivot", () => { + const ctx = context(), proposal = decision(ctx); + const ordinary = {...proposal, explore_review: {...proposal.explore_review as JsonObject}}; + delete (ordinary.explore_review as JsonObject).selected_candidate; + assert.equal(strideReviewMismatch(normalizeStrideDecision(ordinary)[0]!, ctx), null); + assert.equal(qualify(ctx, ordinary, []).accepted, true); + const incumbent = (ctx.explore_basis as JsonObject).comparison_candidates as JsonObject[]; + const keep = {todo_id: incumbent[0].todo_id, ...(incumbent[0].outcomes as JsonObject[])[0]}; + assert.equal(qualify(ctx, {...proposal, action: "verify", explore_review: { + ...proposal.explore_review as JsonObject, considered_refs: [keep.finding_id], selected_candidate: keep}}, + ["retain:todo_method_0"]).accepted, true, "Scoped negative evidence does not ban a new discriminating probe."); + const off = context(projectExploreTurnContext({...input, graph_enabled: false, harness_gate: {enabled: false}, plan: {}})); + assert.equal(off.explore_basis, undefined); + assert.equal(qualify(off, proposal).accepted, false); + const disabledStrategy = projectReplanContext({operation: "project", goal_id: goal, agent_id: agent, rows: [], + goal_facts: {}, obligation: {obligation_id: obligationId}, stride_strategy: "off", + explore_context: projectExploreTurnContext(input)}); + assert.equal(disabledStrategy.explore_basis, undefined); + assert.equal(qualify(disabledStrategy, proposal).accepted, false); + const nonperiodic = {operation: "qualify", obligation: {obligation_id: obligationId, + triggers: [{kind: "vision_acceptance_gap"}], replan_context: ctx}, + agent_vision: {stride_decision: proposal, todo_delta: ["retain:todo_other"]}, + observation_delta: {delta_kinds: ["new_runnable_successor"]}}; + assert.equal(projectReplanSemantics(nonperiodic).accepted, false, + "An explicit invalid binding cannot settle through an unrelated typed successor outcome."); +}); + +test("selection declarations reject malformed identities, unsupported authority and uncited results", () => { + const ctx = context(), proposal = decision(ctx), choice = selected(ctx); + for (const patch of [{todo_id: ""}, {source_sha256: "A".repeat(64)}, {source_sha256: "sha256:" + "a".repeat(64)}, + {node_id: "/private/path"}, {execute: true}]) { + assert.throws(() => normalizeStrideDecision({...proposal, explore_review: { + ...proposal.explore_review as JsonObject, selected_candidate: {...choice, ...patch}}})); + } + for (const patch of [{disposition: "deferred"}, {considered_refs: []}, {considered_refs: ["finding-local"]}]) { + assert.throws(() => normalizeStrideDecision({...proposal, explore_review: {...proposal.explore_review as JsonObject, ...patch}})); + } +}); + +test("source-replayed result pages reach a lower-ranked method without changing planner permission", () => { + const measured = Array.from({length: 4}, (_, index) => ({...findings[index === 3 ? 1 : 0], + finding_id: `finding-ranked-${index}`, node_id: `mechanism-${index}`})); + const pool = measured.map((finding, index) => ({...branches[0], todo_id: `todo_ranked_${index}`, + score: 100 - 20 * index, selection_status: index < 3 ? "selected" : "verification_budget_exhausted", + typed_evidence_audit: {...branches[0].typed_evidence_audit, requested_node_refs: [finding.node_id], + nodes: [{node_id: finding.node_id, status: "exploring"}], findings: [finding]}})); + const full: JsonObject = {...input, plan: {selected_branches: pool.slice(0, 3), rejected_candidates: pool.slice(3)}, + projection: {nodes: measured.map(row => ({node_id: row.node_id, status: "exploring"})), findings: measured}}; + const first = projectExploreTurnContext(full), original = context(first); + const page = {revision: ((first.graph as JsonObject).result_page as JsonObject).revision, offset: 3, limit: 3}; + const read = projectExploreTurnContext({...full, result_offset: 3, result_revision: page.revision}); + const card = ((read.harness as JsonObject).comparison_candidates as JsonObject[])[0]; + const choice = {todo_id: card.todo_id, ...(card.outcomes as JsonObject[])[0]}; + assert.equal(choice.todo_id, "todo_ranked_3"); + const proposal = {...strideControlContract(original, obligationId)!.minimal_example as JsonObject, + explore_review: {revision: (original.explore_basis as JsonObject).revision, disposition: "used", + considered_refs: [choice.finding_id], reason: "Compare the measured alternative under the original budget.", + selected_candidate: choice, result_pages: [page]}}; + assert.equal(qualify(original, proposal, ["retain:todo_ranked_3"]).accepted, false); + const replayed = context(projectExploreTurnContext({...full, review_pages: [page]})); + assert.equal(replayed.context_id, original.context_id); + assert.equal(qualify(replayed, proposal, ["retain:todo_ranked_3"]).accepted, true); + assert.deepEqual(((first.harness as JsonObject).selected_branches as JsonObject[]).map(row => row.todo_id), + ["todo_ranked_0", "todo_ranked_1", "todo_ranked_2"]); + assert.equal(((read.harness as JsonObject).rejected_candidates as JsonObject[])[0].selection_status, + "verification_budget_exhausted", "Evidence access does not rewrite planner execution permission."); + const unlinked = {...full, plan: {...full.plan as JsonObject, rejected_candidates: [ + {...pool[3], typed_evidence_audit: {...pool[3].typed_evidence_audit, requested_node_refs: []}}]}}; + assert.equal(qualify(context(projectExploreTurnContext({...unlinked, review_pages: [page]})), + proposal, ["retain:todo_ranked_3"]).accepted, false, "A page cannot revive a removed current Todo link."); +}); + +test("multiple Todos sharing one finding disclose omission and use the existing expanded page", () => { + const pool = Array.from({length: 21}, (_, index) => ({...branches[0], todo_id: `todo_shared_${index}`, + score: 100 - index, selection_status: index < 3 ? "selected" : "verification_budget_exhausted"})); + const full: JsonObject = {...input, plan: {selected_branches: pool.slice(0, 3), rejected_candidates: pool.slice(3)}, + projection: {nodes: [{node_id: findings[0].node_id, status: "exploring"}], findings: [findings[0]]}}; + const first = projectExploreTurnContext(full), original = context(first); + assert.equal((first.harness as JsonObject).omitted_comparison_candidates, 18); + const read = projectExploreTurnContext({...full, result_limit: 20}); + const cards = (read.harness as JsonObject).comparison_candidates as JsonObject[]; + assert.equal(cards.length, 20); + assert.equal((read.harness as JsonObject).omitted_comparison_candidates, 1); + const card = cards[3], choice = {todo_id: card.todo_id, ...(card.outcomes as JsonObject[])[0]}; + const page = {revision: ((first.graph as JsonObject).result_page as JsonObject).revision, offset: 0, limit: 20}; + const proposal = {...strideControlContract(original, obligationId)!.minimal_example as JsonObject, + explore_review: {revision: (original.explore_basis as JsonObject).revision, disposition: "used", + considered_refs: [choice.finding_id], reason: "Check a different current work item against the same scoped result.", + selected_candidate: choice, result_pages: [page]}}; + const replayed = context(projectExploreTurnContext({...full, review_pages: [page]})); + assert.equal(qualify(replayed, proposal, ["retain:todo_shared_3"]).accepted, true); + const omitted = {...proposal, explore_review: {...proposal.explore_review as JsonObject, + selected_candidate: {...choice, todo_id: "todo_shared_20"}}}; + assert.equal(qualify(replayed, omitted, ["retain:todo_shared_20"]).accepted, false, + "A full planner row alone is not a delivered comparison source binding."); +}); diff --git a/tests/control_plane_ts/stride_policy.test.ts b/tests/control_plane_ts/stride_policy.test.ts index e14f378b88..ac1d48fc54 100644 --- a/tests/control_plane_ts/stride_policy.test.ts +++ b/tests/control_plane_ts/stride_policy.test.ts @@ -5,6 +5,8 @@ import {buildVisionCheckpoint, VISION_REFRESH_REQUEST_SCHEMA} from "../../loopx/ import {projectReplanSemantics} from "../../loopx/control_plane/work_items/replan_semantics.ts"; const contextId = "replan-context-" + "a".repeat(24); +const decisionReviewTriggers = ["periodic_review_due", "typed_progress_repeat", + "no_progress_streak", "repeated_action_loop", "external_progress_review_drift"]; const proposal = {schema_version: "stride_decision_v2", strategy_version: "evidence_conditioned_v1", reference_context_id: contextId, reference_obligation_id: "replan-" + "a".repeat(16), basis: "unexplained_gap", action: "switch", axes: { work_slice: {plan: "Build the complete alternate engine and compare it with the incumbent.", reason: "A partial component cannot test the end-to-end explanation."}, @@ -108,6 +110,11 @@ test("goal-value questions are enabled review input, not persisted decisions or assert.match(work, /recovering an avoidable initial quality deficit/); assert.match(route, /retained best artifact/); assert.match(route, /subsequent work actually implements/); + assert.match(route, /generate a small set of genuinely different explanations or approaches/); + assert.match(route, /representation, decomposition, search method or information source/); + assert.match(route, /Contrast their predictions and full costs/); + assert.match(route, /parameter variants of the same mechanism are not independent alternatives/); + assert.match(route, /Missing feedback is a wait or evidence gap/); assert.match(validation, /artifact version, input conditions/); assert.match(validation, /feedback became available/); assert.match(validation, /not proof of restoration or continued execution/); @@ -161,9 +168,9 @@ test("an accepted schema and vector do not authorize work or certify gain; stale assert.equal(projectReplanSemantics({operation: "qualify", obligation, stride_decision: proposal}).accepted, false); }); -test("every enabled due review records stride even when progress, a successor or a blocker is the exit", () => { +for (const kind of decisionReviewTriggers) test(`${kind} records stride even when progress, a successor or a blocker is the exit`, () => { const obligation = {obligation_id: proposal.reference_obligation_id, - triggers: [{kind: "periodic_review_due"}], + triggers: [{kind}], replan_context: {stride_strategy: "evidence_conditioned_v1", context_id: contextId}}; const authored = buildVisionCheckpoint(request).agent_vision; const withoutDecision = {...authored, stride_decision: undefined}; @@ -197,7 +204,7 @@ test("every enabled due review records stride even when progress, a successor or assert.equal(bareDecision.reason_code, "stride_path_evidence_required"); }); -test("terminal due review can record stride without manufacturing a continuing path", () => { +for (const kind of decisionReviewTriggers) test(`${kind} can record a terminal decision without manufacturing a continuing path`, () => { const terminalPacket = {...packet, state: "no_followup", path_delta: {...packet.path_delta, outcome: "stop"}}; assert.throws(() => buildVisionCheckpoint({...request, agent_vision_packet: terminalPacket}), /stride_decision requires/); @@ -206,7 +213,7 @@ test("terminal due review can record stride without manufacturing a continuing p const terminal = buildVisionCheckpoint({...request, allow_terminal_stride_path: true, agent_vision_packet: terminalPacket}).agent_vision; const obligation = {obligation_id: proposal.reference_obligation_id, - triggers: [{kind: "periodic_review_due"}], + triggers: [{kind}], replan_context: {stride_strategy: "evidence_conditioned_v1", context_id: contextId}}; const observation_delta = {delta_kinds: ["coverage_backed_no_followup"]}; assert.equal(projectReplanSemantics({operation: "qualify", obligation, observation_delta, @@ -215,9 +222,9 @@ test("terminal due review can record stride without manufacturing a continuing p agent_vision: terminal}).accepted, true); }); -test("a successor Todo transition cannot silently settle an enabled due review", () => { +for (const kind of decisionReviewTriggers) test(`${kind} cannot be silently settled by a successor Todo transition`, () => { const obligation = {obligation_id: proposal.reference_obligation_id, - triggers: [{kind: "periodic_review_due"}]}; + triggers: [{kind}]}; const semantic_delta = {schema_version: "replan_semantic_delta_v0", accepted: true, obligation_id: proposal.reference_obligation_id, successor_todo_id: "todo_successor", outcomes: ["new_runnable_successor"], satisfying_outcomes: ["new_runnable_successor"]}; @@ -237,6 +244,23 @@ test("a successor Todo transition cannot silently settle an enabled due review", assert.equal((bound.obligation as Record).resolution_mode, undefined); }); +for (const kind of decisionReviewTriggers) test(`${kind} can retain useful work through an evidenced decision without inventing progress`, () => { + const obligation = {obligation_id: proposal.reference_obligation_id, triggers: [{kind}], + replan_context: {stride_strategy: "evidence_conditioned_v1", context_id: contextId}}; + const requirements = projectReplanSemantics({operation: "requirements", obligation}); + assert.ok((requirements.required_any_of as string[]).includes("fresh_vision_path_outcome")); + assert.ok(((requirements.writeback_contract as Record).required_fields as string[]) + .includes("agent_vision.stride_decision")); + for (const action of ["grow", "switch", "verify"]) { + const vision = buildVisionCheckpoint({...request, agent_vision_packet: {...packet, + path_delta: {...packet.path_delta, outcome: "continue"}, + stride_decision: {...proposal, action}}}).agent_vision; + const result = projectReplanSemantics({operation: "qualify", obligation, agent_vision: vision}); + assert.equal(result.accepted, true, action); + assert.deepEqual(result.satisfying_outcomes, ["fresh_vision_path_outcome"]); + } +}); + test("all three actions preserve independently planned complete work and validation", () => { const axes = {...proposal.axes, work_slice: {plan: "Finish the expanded engine and paired independent repetitions.", reason: "The complete experiment is needed to resolve uncertainty."},