Skip to content

benchmark(runtime): opt-in owner-locked acceptance gate for planned entry - #6585

Closed
songoow wants to merge 6 commits into
mainfrom
codex/lhtb-closure-acceptance
Closed

songoow wants to merge 6 commits into
mainfrom
codex/lhtb-closure-acceptance

Conversation

@songoow

@songoow songoow commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Current disposition

Maintainer feedback on this exact head requests keeping this strategy as a branch experiment or considering integration with LoopX. Mainline adoption is on hold. The branch is retained; this maintenance does not choose a new product integration design, rebase it as a merge candidate, run an experiment, close/retarget the PR or grant merge authority. The validated snapshot correctness repair below remains branch evidence. Current hosted failures and the shared validation repair #6607 are separate from that design decision.

Problem and result

A planned benchmark run must retain the artifact checks the host first accepted. Re-reading the model-writable draft after native configuration could save different criteria in the host copy; a later Todo edit and reconfirmation could then complete a missing artifact.

The host now reads and validates one document, configures a unique temporary snapshot of that document through the existing owner CLI/CAS, and saves the same in-memory document with the returned native revision/digest. Initial configuration and reconfirmation share this path. No second acceptance decision owner or shared control-plane vocabulary is added.

Scope and behavior

  • The optional acceptance_contract=planned remains limited to loopx-planned; its default is off. Planning, worker/Harbor handoff and LHTB configuration use the same explicit switch.
  • Selected-work coverage, leases, task completion checks and owner authority remain with the existing product owner. The runner adapter only configures and reconfirms accepted associations.
  • Explicit off, including a stale document locator, emits no acceptance instructions. Normal wake reconfirmation uses the host copy and refuses an externally changed contract digest.
  • This host copy relies on the existing trusted local filesystem/process boundary. It does not authenticate or isolate processes running as the same OS user.

Current validation

Repair source: 6c60727af; Python 3.12.13, Node 22.22.3 / SQLite 3.51.3.

  • Planned acceptance: 11 passed, including real isolated bootstrap, canonical SQLite, leases, edit, reconfirm and completion. Deterministic barriers modify only the model draft immediately before and after the real native configure effect. Both counterexamples completed a missing artifact before the fix; both now reject completion until the original artifact check passes.
  • Checkpoint capture, heartbeat envelope transport, Turn driver and goal-acceptance CLI regression: 198 passed, 11 skipped. Skips require explicit real Host prerequisites and do not qualify those paths.
  • Ruff, changed-source semantic advisory, full semantic-vocabulary smoke and whitespace checks passed. The advisory does not scan benchmark vocabulary; no new shared vocabulary is claimed from its empty output.
  • An earlier broad test attempt was interrupted by local disk exhaustion and is retained as an invalid qualification attempt; the counts above are the completed rerun. No real model, hidden evaluator, live benchmark, scoring change or submission was executed.

Delivery boundary

This is the existing benchmark runtime provider adapter and focused regression. UI/Lark and generic product acceptance behavior are unchanged. Related refactor applied: initial and subsequent configuration use one snapshot path, replacing the reusable mutable reconfirm file. The independent source-staging repair remains in #6584 rather than being duplicated here.

Mainline convergence requires an accepted owner placement before renewed integration qualification; hosted CI, benchmark manual hold and explicit maintainer merge authority remain separate. A local artifact gate does not prove long-running benchmark quality or close the research RFC.

songoow and others added 2 commits October 16, 2026 07:54
…ntry

Four LHTB trials derived terminal no-follow-up at 43-60 min of a 90 min
budget because every Todo was marked done while the deliverable still
failed the verifier's own interface (CLI non-zero on the hidden wrapper
manifest, model unable to predict in the sandbox). The closure rule itself
is correct: it only sees Todo completeness.

Add `acceptance_contract: planned` (default `off`) to the shared benchmark
runtime. During the planning checkpoint the model also writes a Goal
acceptance document whose criteria exercise the deliverable through the
task's declared verifier interface on public inputs. The trial host, as the
trusted local owner inside the container, configures it through the
existing `goal-acceptance` contract scoped to the Todos planning returned,
so a selected Todo cannot complete until its criteria pass. On each wake
the host re-confirms associations that the model's own Todo edits made
stale without touching the criteria. A missing or invalid document fails
the planning phase. `off` keeps byte-identical prompts and behavior.

No scoring, hidden-verifier feedback or new control-plane vocabulary is
introduced; `ACCEPTANCE_CONTRACTS` is a runner-local closed set.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: song <liusongstep@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: song <liusongstep@gmail.com>
…ance failures in the wake receipt

The first live planning run wrote `validation_files` as a list of paths; the
owner contract needs `{path, sha256}` pins, and the TypeScript rejection only
reached the host as a truncated traceback. Tell the planner to omit the field,
fail with the criterion id named, and record the public-safe reason in the
wake receipt so a failed plan stage is diagnosable from downloaded logs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: song <liusongstep@gmail.com>
@songoow

songoow commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

Live-run readback (LHTB host, covshift, LOOPX_ACCEPTANCE_CONTRACT=planned)

First planning phase that reached the model (job …-084307, 429k input / 384k cached / 10.6k output tokens) wrote a well-shaped document: selected_work over the three planned Todos, four criteria including c_public_predict_interface (loads the submitted pickle and calls model.predict on public validation rows, asserts finite (n, 200) output) and c_clean_submission (leakage_flag is False, audit_gap <= 0.10). That is exactly the verifier-facing closure check the pilot runs lacked.

It failed configuration because validation_files was written as a list of paths; the owner contract requires {path, sha256} pins, and the TypeScript rejection reached the host only as a truncated traceback. Fixed in a10f988: the planner is told to omit validation_files, the host rejects bare pins naming the criterion, and the failure reason is kept in the wake receipt. Re-run in progress; results will be posted separately and are not part of this PR's claims.

Two deployment notes for anyone reproducing: the in-container install-local.sh needs a prepared loopx/web/chat bundle valid for the exact source (otherwise it attempts npm ci offline), and a stale cached mirasim route shows up as 502 Bad Gateway in the plan wake.

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=max

动机

运行多轮计划任务的维护者,需要模型真正产出并检查交付物后再结束任务,而不是仅把计划列表标成完成。

旧运行器没有配置产物验收门槛,任务可能过早结束;本提案在计划后配置既有验收规则,但下一轮重新确认时会重新信任可改写的原始文档,使缺失产物仍能被标记完成。

首次配置确实能拦住缺失产物,正常产物通过后也能完成;跨轮保持原验收条件,以及显式关闭时不发出该功能指令,尚未达成。

本次只核验计划、验收和继续执行的本地边界,不运行真实模型或隐藏评分器,也不把本地检查当作独立评分。

改动思路

运行器应复用既有 TypeScript 验收 owner 和 CLI 的版本/CAS/完成检查;Python 只负责本次基准的计划文档与调用顺序,不再建立第二套完成规则。

当前 PR 的边界是计划型 benchmark 的显式验收选项及 Harbor/worker 传递,不包含通用产品 UI、新的身份隔离或隐藏评分器集成。

正常路径为“计划创建实际 Todo → 模型提出验收文档 → host 以可信本机 owner 调用既有 configure → 每次完成运行原条件”。关联失效时只能重新确认当前 Todo;不能把可变草稿等同于已接受的规则。当前实现恰好在这一步重新载入全部条件。该边界沿用本机角色与文件权限,不提供同一系统用户进程间的身份隔离。

具体改动

关键代码讲解

  1. apply_planned_acceptance:Validates selected planning IDs and applies the document through the existing owner CLI.

  2. reconfirm_planned_acceptance:Reloads the mutable document on every wake, filters applicable tasks, then reconfigures all its criteria without matching the accepted semantics.

  3. render_planning_prompt:Emits mandatory acceptance instructions solely from a document locator, even with acceptance_contract=off.

  4. run_once:Enforces the explicit Execution flag for configuration and reconfirmation; this flag and prompt activation currently disagree.

规范按 docs/reference/goal-acceptance-observations.md 在提交 1fd9bbb 的文本判断。该文档没有逐条编号,以下以原文条款名称标识。

  • selected_work:implemented;Independent real CLI: selected missing artifact held; existing and newly-created unselected work complete; contract digest unchanged.
  • owner configuration:not_met;Automatic host reconfirmation replaces the accepted artifact check with mutable draft contents after a Todo edit.
  • fresh bound artifact checks:not_met;After rewriting the criterion to Python pass, the host configures revision 2 and missing outputs/result.txt completes.

对主干的风险

[P1] 重新确认会接受被改写的验收条件。 我通过真实隔离 CLI 先配置要求 outputs/result.txt 为 ok 的条件,缺失文件的完成请求按预期失败。随后修改同一 Todo 使关联 stale,再把原 JSON 中的 validation_argv 改为 python -c pass。下一次 reconfirm_planned_acceptance 以 owner 配置了 revision 2,摘要改变;同一 Todo 随即完成,产物始终不存在。安装路径让 control 文档保持模型可写;这不是同用户进程认证问题,而是 host 自己承诺“不改变条件”却重新采用草稿。固定首次采用的验收声明并校验其语义摘要;重新确认只能更新受影响 Todo 的关联,拒绝或显式交回 owner 审核条件变化。 需要把该反例加入真实 CLI 测试。

[P2] 显式关闭仍会发出验收指令。 给实际 planning worker 同时传 LOOPX_ACCEPTANCE_CONTRACT=off 和遗留文档 locator,worker 成功且 receipt 不含已启用验收,但发给模型的正文仍含强制的 Host acceptance checkpoint。Harbor 普通关闭路径会省略 locator;问题在共享 worker 对显式关闭与遗留输入的处理。请使提示词采用同一经过校验的开关,并验证 off/无 locator、off/有 locator、planned/有 locator 三路。

发布前检测到新提交,旧头结论未发布;以下按新头重新执行。新提交增加裸 validation_files 路径拒绝、要求规划省略 pins,并把验收失败原因保存到 wake receipt;这些新增正常/负向路径已通过。Python 3.12.15 + Harbor 0.20.0/SForge 1.1.0 下,验收/真实 CLI/运行时与共享 worker/planning/session 合并测试 196 passed / 1 skipped,跳过项要求真实 Codex。两个上述反例均在新头重新复现。独立范围测试确认:受选任务缺产物被拦、同容器原有与后建未选任务可完成,合同摘要不变。基线与当前头默认规划文本一致,但 off+locator 路径不一致;默认环境还新增 LOOPX_ACCEPTANCE_CONTRACT=off 字段,不能宣称所有环境字节完全相同。隔离 plain/native 导入在本 PR 与基线均失败,签名相同,归因于既有打包缺口;该缺口由 #6584 单独修复,不要求本 PR 复制修复。原生 premerge 4 direct + 19 selected 全过,保留 benchmark_sensitive 人工审查 hold。

语义与 CI 对齐

复用既有 selected_work、ready/stale、owner CAS 与完成验证,不新增共享 Python 决策 owner。off/planned 属于运行器本地显式选项。开发期 advisory 不扫描 benchmark 目录,因此零候选不是无新增语义的证据;真正的问题由跨轮可变条件和关闭提示词反例确定。未查询或等待远端 CI。

我的整体评价

初始产物门槛有价值,但跨轮可被重写后会再次过早结束;关闭功能却仍要求模型准备文档也增加无效工作。因此长期执行与操作体验都尚未保住,当前头请求修改。有限的未来改动检查应先固定已采用声明并复用同一开关,不扩大成另一套验收服务。

运行器应复用既有 TypeScript 验收 owner 和 CLI 的版本/CAS/完成检查;Python 只负责本次基准的计划文档与调用顺序,不再建立第二套完成规则。 当前 PR 的边界是计划型 benchmark 的显式验收选项及 Harbor/worker 传递,不包含通用产品 UI、新的身份隔离或隐藏评分器集成。 当前证据不包括真实模型采用、隐藏评分、长期得分或发布安装;重新评审需在新精确头修复上述反例并重跑原正常/负向路径。

English verdict: REQUEST_CHANGES

Comment thread benchmark/runtime/acceptance.py Outdated
"bindings": [row for row in bindings if row["todo_id"] in applicable]}
document_path.write_text(json.dumps(revised, ensure_ascii=False, indent=2) + "\n",
encoding="utf-8")
receipt = _configure(env, cli, document_path,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] 重新确认须保留首次接受的验收条件

这里重新采用模型仍可改写的文档全部 criteria,没有校验它与已接受合同的条件一致。真实隔离 CLI 复现:原条件要求 outputs/result.txt 为 ok,缺文件完成先失败;编辑 Todo 使关联 stale,把原 JSON 的 validation_argv 改为 python -c pass,再调用 reconfirm,host 以 owner 配置 revision 2,之后缺文件仍能完成。请固定或核对首次采用声明,只重新绑定当前 Todo 关联;条件变化应拒绝或回到明确的 owner 审核。新增真实 CLI 测试保证原产物通过前任务仍 OPEN。

Comment thread benchmark/runtime/planning.py Outdated
"runtime startup and settlement reserves also consume it.",
])
prefix = "$loopx\n\n"
if env.get(ACCEPTANCE_DOCUMENT_ENV):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 验收提示词需要使用同一显式开关

此处仅凭文档 locator 发出强制验收指令,而 worker 的配置和 reconfirm 按 Execution.acceptance_contract 决定。实际 planning worker 在 LOOPX_ACCEPTANCE_CONTRACT=off 且遗留 LOOPX_ACCEPTANCE_DOCUMENT 时成功运行、receipt 保持禁用,却向模型发送 Host acceptance checkpoint。普通 Harbor 关闭路径会省略 locator;共享 worker 仍需按同一经过验证的开关处理这种关闭输入。请覆盖 off/无 locator、off/遗留 locator、planned/locator,前两路应保留原提示词且无验收效果。

@songoow

songoow commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

Live re-run result: covshift with planned (one trial, not a causal claim)

Pilot 2026-10-15 (off) This run (planned, head a10f988) Five-arm study heartbeat
hidden-set reward 0.000 0.82 0.743
end AgentTimeoutError at 5400 s AgentTimeoutError at 5400 s —
cost $15.8 $15.3 (19.9M prompt / 18.9M cached / 190k completion, 10 sessions) —

Mechanism readback from downloaded wake receipts and the worker log:

  • Planning configured 3 criteria bound 1:1 to 3 Todos (public model.predict interface on validation rows, 25-experiment diversity/plateau receipt, submitted leakage_flag=false + predict through the submitted pickle). Contract revision 1; every execute wake read reconfirmed: [] (no stale associations, so the model never needed its Todo text edited under lease).
  • The first Todo's todo complete ran its bound predict-interface criterion and passed. In the pilot the submitted model could not predict in the verifier sandbox; that is the failure this criterion targets.
  • stop_until_explicit_resume still fired once, at ~38 min, after all three criteria had passed and the Goal derived no_followup. Harbor's continue-until-timeout then started phase 2: planning wrote a new document (revision 2) with a new_honest_submission_public_interface criterion, and two later todo complete attempts were rejected with goal_acceptance_validation_rejected (command_failed) until the artifact satisfied it. A third phase/plan started at ~83 min and was cut by the timeout.
  • Verifier: test_recovery_score 0.704, audit gap -0.0185, leak False, band 2 (pass threshold 2).

Reading: the gate changed what counts as closure evidence (verifier-shaped, owner-locked), not whether the agent may close; closure at 38 min with a passing honest submission is the intended behavior. Whether forced continuation after that point adds reward is a separate question this trial cannot answer (the submitted artifact timing was not retained by the harness). Two more paired runs (nrel-pysam-hybrid-renewables-audit, unison-paper-reproduction) are queued serially on the same host.

songoow and others added 2 commits October 16, 2026 11:13
…ment

Re-confirmation re-applied the model-writable planning draft, so a Todo edit
followed by a rewritten criterion let a selected Todo complete without its
artifact. The host now keeps the configured document in its runtime root,
re-applies only that copy, and refuses when the owner contract digest moved
outside the host. The planning prompt uses the same explicit
LOOPX_ACCEPTANCE_CONTRACT switch, so off with a leftover locator keeps the
original prompt. Also fixes the E702 lint failures.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: song <liusongstep@gmail.com>
…ceptance

Signed-off-by: song <liusongstep@gmail.com>
@songoow

songoow commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the two REQUEST_CHANGES findings on a10f988 (new head b41e8a7):

  • [P1] Re-confirmation now re-applies only the host-accepted document. apply_planned_acceptance/_configure keep the configured document in the trial runtime root, and reconfirm_planned_acceptance rebuilds the scope from that copy, never from the model-writable draft. It also refuses when the owner contract digest changed outside the host. The real-CLI test now rewrites the draft criterion to python -c pass after the Todo edit: after reconfirm, completion is still rejected until outputs/result.txt exists. A new test covers the out-of-band reconfigure refusal and the missing-record case.
  • [P2] The planning prompt uses the same explicit LOOPX_ACCEPTANCE_CONTRACT switch (planned_acceptance_document). Covered paths: off without a locator and off with a leftover locator both produce the original prompt and no acceptance effect; planned without a locator fails fast; planned with a locator adds the guidance. The worker-level test adds the off+legacy-locator variant.
  • Fixed the E702 lint errors that failed kernel-static-checks. Merged current main.

Local: planned-acceptance tests 9 passed; ruff (CI scope + benchmark/runtime) clean; mypy clean; benchmark/tests 363 passed. The 5 failures in test_checkpoint_archive_transport come from docker not being installed in this venv, not from this change. Scope note: a new planning phase still configures its own document (as in the covshift phase-2 run); only re-confirmation is pinned.

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent | gpt-6.1-sol | OpenAI | runtime_reported | reasoning_effort=max

动机

运行计划型基准任务的维护者,需要交付物满足首次接受的验收条件后才能结束,避免只把待办列表标成完成。旧运行器没有产物验收门;本提案能先拦住缺失产物,但配置完成与保存 host 副本之间改写草稿,下一轮仍能把原条件替换为恒通过命令,让缺失产物的任务完成。普通重确认已使用 host 副本,关闭但遗留 locator 的指令泄漏也已修复;配置 IO 窗口中的条件替换仍通过真实 CLI 复现。本次只评审计划型运行器对既有验收 owner 的本地适配,不运行真实模型或隐藏评分,也不认证长期得分或隔离同系统用户的进程。

改动思路

运行器复用既有 TS 验收与完成 owner;需要的是同一已接受快照跨 IO 保持一致,而不是第二套验收服务。 本 PR 只覆盖计划型 benchmark 的显式验收适配;修复配置到保存副本的快照衔接后再批准。

正常流程是模型创建 Todo 和验收草稿,trial host 以既有可信本机 owner 调用 configure,产品 owner 约束选中任务的完成。选择范围、绑定关联和实际产物检查是不同事实;不能以文件存在或关联 ready 代替产物合格。Python 适配属于 benchmark provider,不新增通用控制面决策。selected_work 继续限定本次规划明确返回的任务,原有与后来创建的未选任务保留自身检查。

新 host 副本消除了“下一轮直接重读草稿”的旧问题,也保留了外部重配置摘要变化时的拒绝。但副本必须就是原生 owner 已接受的内容;现在 _configure 在外部 CLI 完成之后才重读同一个可变路径,仍没有完成这条承诺。最小修复是同一不可变快照贯穿配置和副本保存,无需扩大成新的服务或跨进程认证体系。

具体改动

head b41e8a716456e77ed97738168b447e39d7be36a6,实际 merge base ebc01b2a46abb493f8a2300ce07bc54036cc7a29;独立规范 docs/reference/goal-acceptance-observations.md,spec_revision ebc01b2a46abb493f8a2300ce07bc54036cc7a29。规范与前次已读版本 blob 相同,未用本 PR 文案重新定义验收。

  • selected_work:implemented;真实 CLI 验证选中任务缺文件被拦,同容器原有及后建未选任务可完成,合同摘要保持。
  • owner configuration:not_met;后述 IO 窗口会将未被首次 owner 接受的草稿内容存成 host 副本。
  • fresh bound artifact checks:not_met;产品检查本身正确执行,但下一轮 host 能替换它,导致缺失产物最终完成。

关键代码讲解

  1. acceptance.py:135 _configure 使用 inspect/provider revision/CAS 配置并核对 association ready;152–154 行却在配置后重新读取草稿,产生摘要和文档内容不一致的副本。
  2. acceptance.py:181 reconfirm_planned_acceptance 读取副本,要求当前 owner digest 与存储 digest 相同,过滤完成/取消任务,并只为 stale 关联重新配置。普通编辑已安全,但不一致副本仍会被它作为已接受内容重放。
  3. planned_acceptance_document 与 render_planning_prompt 共用显式开关;实际 planning worker 的 off+遗留 locator 不再发送 Host acceptance checkpoint,planned 缺 locator 则明确失败。
  4. worker.run_once、Harbor plan readback 与 LHTB 配置传递一起保留启用范围、失败回执和 handoff 拒绝,不从安装可用性推断启用。整个提案为 10 paths +589/-2,含四处公共操作说明/配置、五处运行适配和一组 focused 测试,没有改 loopx/**。

对主干的风险

[P1] 保存副本时重读可变草稿,仍可替换首次接受的条件。 我在隔离 canonical runtime 通过真实 CLI 接受“outputs/result.txt 内容为 ok”的条件。在原生 configure 已返回 applied、但 host 读取副本前这个确定窗口,仅改写模型草稿的 validation_argv 为恒通过命令;没有修改 native receipt 或 host 副本。初次缺文件完成按原条件失败;修改 Todo 使关联 stale 后,reconfirm 比对的原 digest 仍相同,于是以副本中的弱条件配置 revision 2,随后缺文件任务实际变成 DONE。Python 3.11 和 3.12 均重现,真实接受/完成效果没有被 mock。

修复需先固定输入快照,并让 native configure 与 host 副本使用同一份内容,校验它与提交的 owner basis 相符。仅在失败后补异常或仅保护后续读取不足以消除该窗口。回归应在真实配置 effect 后设置屏障改草稿,继续经过真实 lease/edit/reconfirm/complete,原产物未通过前始终保持 OPEN。

当前相关测试为 207 passed /11 skipped,Python 3.11 与补齐 Harbor0.20/SForge1.1 的 3.12.15 两个环境均如此,按同一覆盖计数;跳过为显式真实 Host 条件。普通 host-copy、外部 digest 变更拒绝、缺失记录和显式关闭正常路径通过,不能覆盖这个不同时间窗。原生 premerge 4 direct +19 selected 全过,另有 benchmark_sensitive 人工 hold;DCO、公共边界与先 advisory 后完整语义检查通过,未查询 CI。

语义与 CI 对齐

复用现有 selected_work、ready/stale、owner CAS 和完成检查;off/planned 属 runner 本地闭集。开发 advisory 不覆盖 benchmark 载体,空结果不证明无语义。显式关闭的正文、产物状态和回执与基线一致;五种默认环境新增了 inert LOOPX_ACCEPTANCE_CONTRACT=off,Execution/YAML 也披露默认字段,因此不能宣称所有环境字节全等。isolated source archive 在 base/head 都有相同既存 plain/native import 缺口,由 #6584 单独修复,不要求本 PR 复制。

我的整体评价

REQUEST_CHANGES。普通重确认与关闭指令两个旧反例已修复;当前完整提案仍未保住首次 owner 条件,long_horizon=not_yet_proven、user_experience=not_yet_proven,缺失产物可误完成。规模和必要持久记录与问题相称,应在现有 owner 边界做有限快照修复。真实模型采用、公开反馈效果、隐藏得分和持续运行均未由本地证据证明;作者的 live 单次分数不作为本评审的因果证据。

English verdict: REQUEST_CHANGES - b41e8a7; a draft edit after real configure but before host-copy read still replaces the accepted criterion during reconfirm and completes a missing artifact. 207 tests pass with 11 Host skips; native configuration/completion race reproduced on Python3.11 and3.12.

Comment thread benchmark/runtime/acceptance.py Outdated
raise ValueError(f"acceptance association is not ready for {', '.join(not_ready)}")
_write_json(_accepted_record_path(env), {
"revision": contract["revision"], "digest": contract["digest"],
"document": json.loads(document_path.read_text(encoding="utf-8")),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Persist the exact document accepted before the configure IO

This reread occurs after the real owner configure succeeds, so it can store a later model draft together with the digest of the earlier accepted contract. I changed only that draft in the deterministic window after the native configure effect: the initial missing-artifact completion was rejected correctly, but after an ordinary Todo edit the host reconfirmed the stored weaker command as revision 2 and the task completed without outputs/result.txt. Reproduced on Python 3.11 and 3.12 with real configure/lease/edit/complete effects; no host record or native acceptance result was mocked. Freeze the input before configure and use the same accepted snapshot for both the native command and host record. Add the interleaving after real configure, then require the original artifact check to remain enforced through reconfirmation.

Signed-off-by: song <liusongstep@gmail.com>

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

感觉这个策略做进benchmark里有点重了,保留在分支里实验或者看怎么和loopx融合

@mergify

mergify Bot commented Oct 16, 2026

Copy link
Copy Markdown

This pull request has merge conflicts with main and cannot be merged
until they are resolved. Please rebase or merge the base branch, @songoow.

Choose the remote for the base repository, not an out-of-date fork.
For a fork clone, first inspect git remote -v; upstream must point
to https://github.com/loopx-project/loopx.git. If it is absent, add it
with git remote add upstream https://github.com/loopx-project/loopx.git.
Then run:

git fetch upstream
git rebase upstream/main
# Resolve each conflict, git add the resolved files, then git rebase --continue.
git push --force-with-lease origin HEAD

For a same-repository clone whose origin points to
https://github.com/loopx-project/loopx.git, use origin instead of
upstream for fetch/rebase. If you prefer merging the base, use
git merge <base-remote>/main and push normally.

Keep the DCO Signed-off-by trailer on every commit when you rebase.
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase Mergify: the pull request has merge conflicts with its base branch label Oct 16, 2026

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent | gpt-6.1-sol | OpenAI | runtime_reported | reasoning_effort=max

请求修改:[P1] 重确认的第二次读取会采用尚未校验的新 owner basis,覆盖合法的新验收条件。

动机

运行多轮计划型基准任务的维护者,需要任务完成时仍满足当前 owner 接受的产物条件,避免待办已完成但交付物缺失。

提案给计划型运行器增加默认关闭的验收适配:首次计划接受条件,后续只重新确认被编辑的任务关联;owner 在两次读取之间修改条件时,当前 host 却会覆盖这次修改。

首次缺产物拒绝、原有及后来未选任务不受门禁影响、正常恢复和关闭路径已核;真实 SQLite/CLI 交错仍让缺少 owner 新要求文件的任务变成 DONE。

本次不运行真实模型、隐藏评分器或 benchmark,不认证得分、长期净益、跨进程认证、主干采纳或安装发布。

改动思路

复用既有 Goal acceptance 的 TS owner、CAS 和完成检查,snapshot 修复属于同一 provider 适配;维护者当前明确要求保留分支实验,主干整合的 owner placement 尚未获得接受。 当前按已声明的分支实验边界核验完整运行器适配;先在既有 helper 中修复同一已核 basis 的重确认,不另建验收服务,也不替维护者决定主干融合方案。

正常流程是模型产出规划 Todo 和验收草稿,trial host 固定一份输入快照,以既有可信本机 owner 角色通过原生 CAS 配置,再保存同一份文档与回执 digest。每轮执行仅为 stale 的所选工作关联重新确认;真正完成仍运行原生检查。selected work、关联 ready 与产物通过是不同事实。这里的 owner 角色依赖现有本机文件/进程权限,不是同一 OS 用户进程间的认证,也不是独立 oracle。

最新单快照 helper 解决了前次 b41 在配置后重读模型草稿的问题,属于合理的有限重构。当前新问题出在时间有效性:第一次确认 digest 后,helper 再次 inspect,却直接把新 provider revision 配给旧条件;native CAS 本身正常工作,调用方用了不再代表已确认意图的 basis。

具体改动

head 6c60727af26256e0ba1eb4c49b7e5048a632420a,实际 merge base ebc01b2a46abb493f8a2300ce07bc54036cc7a29。spec_ref: docs/reference/goal-acceptance-observations.md;spec_revision: ebc01b2a46abb493f8a2300ce07bc54036cc7a29,见原生 owner 合同。同时完整读取了维护者当前意见和两次旧评审/inline;没有用新 README 重新定义既有权威。

原合同条款 当前判定
selected_work implemented:只拦显式所选任务;同容器已有和后来新增未选任务实际完成,合同 digest 不变;修改 class 逃离的请求被拒绝。
owner configuration not_met:已校验旧 digest 后 owner 修改合同,第二次 inspect 取得新 revision 却写回旧文档。
fresh bound artifact checks not_met:native 检查能正确执行,但 adapter 先替换 owner 的新条件,使缺失新产物的任务完成。

关键代码讲解

  1. _configure 的临时 snapshot 与保存 record 共用同一内存文档,修复了旧草稿窗口;139行重新 inspect、149行采用该 revision,和调用方已检查的 digest 没有绑定。
  2. reconfirm_planned_acceptance 先核 host copy 和 owner digest、筛当前 applicable/stale,再调用 helper。外部修改发生在首次读取前会拒绝,发生在之后却被第二次读取掩盖。
  3. render_planning_prompt 与 planned_acceptance_document 使用同一个显式开关;off+遗留 locator 已不再泄漏验收指令,planned 缺 locator 明确失败。
  4. run_once 在计划成功后配置、执行前重确认,保存有限失败信息;Harbor 的环境、上下文和 handoff 检查及 LHTB renderer/shell 传递同一开关。完整10路径 +622/-2,包括上述适配、两份说明及319行测试,未修改原生 scoring 或 loopx/** owner。

对主干的风险

[P1] 并发 owner 修改会被旧 host copy 覆盖。 独立隔离 SQLite/真实 CLI 步骤:配置 revision1,真实 lease 下修改 Todo 使关联 stale;让 reconfirm 首次真实 inspect 返回旧合同;在返回后、helper 第二次读取前,由合法 owner 通过当前 CAS 配置 revision2,新增要求 outputs/final.txt。host 随后用新 revision 作 expected basis,把旧 result-present 条件写成 revision3。只提供原 outputs/result.txt,final.txt 不存在,实际 todo complete 仍 exit0,独立 list 读回 DONE。

对照组把同样 owner 修改放在首次读取之前:明确拒绝重确认、保留 revision2,缺文件完成 exit1,Todo 保持 OPEN。两个窗口都通过真实 parser/preflight、bootstrap、canonical owner、lease/edit/configure/complete;包装只控制真实 CLI 结果返回的时机,未伪造 native 回执、accepted 状态或完成后置条件。把首次校验 digest 的 inspect 所得到的 provider revision 传到 configure;若重新读取,必须重新验证合同 digest/范围后才能使用新 revision。并发 owner 修改应保留并明确拒绝重确认。

原有11项当前测试全部通过,包含原生 configure 前后修改模型草稿两个窗口;它们不能覆盖这次不同的 owner-update 窗口。相关 checkpoint capture、heartbeat envelope、Turn driver 和 acceptance CLI 共198 passed/11 skipped;skip 都有真实 Host 前置条件,不当作其资格。独立五例 scope/recovery 证明:所选缺产物被拦、原有/后建未选工作正常完成、class 逃离被拒绝、正常重新确认并产出原文件后确实完成。

显式关闭对比:同一 packet 的四种 off/遗留 locator 输入在 base/head 生成相同正文;五种合法 Execution 模式除已披露的 inert acceptance_contract=off 字段外相同,turn 的既有独立验证命令前置条件保留。head 的实际 planning worker 验证 off 无验收副作用、planned 缺文档保存失败回执和 planned 正常配置。环境/YAML 新增 off 字段已披露,不能称所有环境字节全等。没有从 CLI 可用或 locator 存在推断启用。

语义与 CI 对齐

复用 selected_work、ready/stale、owner CAS、产物完成检查;off/planned 是 runner 本地闭集,benchmark 语境保留在 provider 内。advisory 对 benchmark 目录报告0载体,不能以空输出证明没有语义;完整 semantic、Ruff、DCO和公共边界通过。原生 premerge 4direct+19selected 可执行检查通过,整体仍 quality_stale_receipt,并有 benchmark_sensitive 人工 hold;未生成/豁免凭据、调预算或查询/等待 CI。

我的整体评价

REQUEST_CHANGES;long_horizon=regression、user_experience=regression 针对已证明的 enabled 并发 owner 修改丢失,不扩大成默认 off 的回归。旧草稿重读和关闭提示词发现已修,新的同 basis 承诺尚未保住。未来改动检查应把首次已核 revision/digest 贯穿同一 helper 和 effect,保留正确 snapshot seam,不增加第二套验收服务或更多状态开关。

当前按已声明的分支实验边界核验完整运行器适配;先在既有 helper 中修复同一已核 basis 的重确认,不另建验收服务,也不替维护者决定主干融合方案。 维护者在当前 head 要求“保留在分支里实验或者看怎么和loopx融合”的 hold 继续有效;当前 adapter 的本地测试不会选择融合方案或授予主干采纳。单次公开 covshift 结果是作者历史观察,不作本评审因果证据。重新评审需在新精确 head 保留外部 owner 修改,验证缺新文件仍 OPEN,重跑原全部正常/负向、scope/off 和原生检查;真实 Host、得分、长期使用及合并授权各自保留。

English verdict: REQUEST_CHANGES - 6c60727; after the first digest-checked inspect, a real owner update to revision2 is overwritten using a newly inspected basis at revision3, and completion succeeds without the owner's new artifact. Original11 tests and198 related tests pass with11 real-Host skips; preserve the explicit branch-experiment/mainline-placement hold.

def _configure(env, cli, document: dict[str, Any], *, operation_id: str,
expected_todo_ids: list[str]) -> dict[str, Any]:
goal = ["--goal-id", env["LOOPX_GOAL_ID"]]
basis = _run_cli(env, cli, ["goal-acceptance", "inspect", *goal])

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Reconfirm against the same owner basis whose digest was checked

reconfirm_planned_acceptance checks the first inspect digest at lines208–214, but _configure inspects again here and uses that fresh revision with the old host document. In an isolated SQLite Goal using real native CLI/lease/configure/complete calls, an owner update after the first inspect strengthens revision2 to require outputs/final.txt; this second inspect adopts revision2 and writes the old result-present criterion as revision3. final.txt is absent, yet actual completion exits0 and the Todo is DONE. The same update before the first inspect is correctly rejected and preserves revision2. Pass the provider revision from the digest-checked inspect into configure, or revalidate the digest/scope before adopting a refreshed revision; the interleaving must preserve the owner update and reject completion missing its new artifact.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-rebase Mergify: the pull request has merge conflicts with its base branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants