Skip to content

fix(benchmark): start checkpoint replan in a fresh conversation - #6622

Merged
huangruiteng merged 2 commits into
codex/edgebench-checkpoint-integration-20261016from
codex/checkpoint-fresh-replan-20261016
Oct 16, 2026
Merged

huangruiteng merged 2 commits into
codex/edgebench-checkpoint-integration-20261016from
codex/checkpoint-fresh-replan-20261016

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Restored checkpoint experiments previously could clear the native conversation only after the destination had already replanned. That leaves the first decision conditioned on inherited native history.

Add an explicit --restore-fresh-replan treatment that starts the first destination model call in a fresh conversation, before preparation and replan. It requires the sealed pre-decision packet, reuses the typed artifact selection and native admission owners, retains checkpoint/research lineage, and preserves the original budget and native opportunities. The existing post-review option remains explicit and mutually exclusive for compatibility; defaults are unchanged.

Validation: focused session/warm-start/restore tests 62 passed, 9 environment skips; shared Codex runtime and checkpoint handoff 92 passed, 34 environment skips. A non-root isolated Linux receiver consumed a real mature sealed point, verified retained research and unchanged source history, then reached the actual fresh native command before any destination review. It made no remote model or grading calls, so research adoption, a subsequent recapture and outcome benefit remain unproven. Compilation, semantic advisory and public-boundary diff checks passed.

Placement: existing benchmark provider composition and Codex session IO; the TypeScript artifact owner still selects the pending replan. Reuses the once-only session replacement/recovery path rather than creating another authority owner. No general frontend restore journey is claimed. This PR is based on the current experimental integration branch; it does not merge or publish that branch to main.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · reasoning_effort=max · observation_id=b03ff20e1b97e1566d2adbf1d42f9e0fcedd546825a3c8466e93a9093f157504

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

依赖与版本更正: 父 #6618 的非法 stub 输入 P2 已撤回、同头更正为 APPROVE(见其原评审链接)。本条 native30秒问题仍仅指旧7b8头。当前8548头因合入 #6630 已变化,须重新审阅全 diff;不沿用旧结论。

动机

研究操作者希望从同一决策前 checkpoint 重新思考路线,保留任务、研究文件和原预算,同时舍弃旧原生消息。原选项先在继承历史中完成 replan 再清空上下文;本提案新增首轮准备和 replan 就进入 fresh 会话的显式选项,避免把旧决定当成新决定的输入。新参数、原始 pending packet 校验和一次性会话替换已接通,但首次准备再次完整准入时仍硬编码 30 秒,成熟历史失败后的完整用户路径尚未证明可用。本次评审不授予合并、评分或执行权限,不证明科学收益、完整持续运行或已安装产品资格。

[P2] 首次 fresh 准备重新完整准入时仍限制为 30 秒 — benchmark/runtime/checkpoint_session_refresh.py:102–103。已通过初始 warm native admission 的成熟 checkpoint,首次使用 --restore-fresh-replan,尚无 pre-replan.json;完整历史复核需要超过 30 秒。新分支再次调用 admit_native_checkpoint_session(timeout=30),后者对现有绑定重新读取完整 thread/turns/list;超时发生在 fresh 会话准备和首次有效研究工作之前,无法完成承诺的首轮路径。新处理不能只以较小成功点代替成熟失败点。资格化这个第二次完整复核及其预算,在原累计截止时间内支持已接收的成熟点;可复用既有读取/会话 owner,但必须保留拒绝 advanced/foreign source、原结果恢复与未知启动不得重发。不要仅删除复核或修改历史实验阈值。

改动思路

精确 head 7b8b08f8a85dd7f0b87398d838af09f1b6a1bf03,实际 parent/merge base 072178167e8e3389986502922495f2cc9a32e919;目标是 codex/edgebench-checkpoint-integration-20261016,不等于 main。已核对真实祖先而非只比 tree;父 PR #6618 的原文件入口 P2 已撤回,并在同一 head 上完成更正批准;其结论也不能代替本 PR 的独立审阅。

捕获或上下文处理留在既有 benchmark/native adapter,准入、原始来源和会话替换仍由各自 owner 决定。当前 PR 只资格化显式观察/上下文处理;不扩大原预算、原生机会或执行权限,下面的入口缺口修好前不能批准。新的 pre_replan_basis 复用原 artifact replan selector 与原 session CAS。重复完整复核有防止 source 已推进的实际职责,不能简单删除;需要让该职责在原累计预算内支持已接收的完整历史。更大的统一读取预算整理可以归原 native adapter 继续,本次必须先交付这个首轮路径。

处理 首次准备/replan 的 native 输入 保留的边界
两选项均省略 原 retained history 原来源、原预算和机会
旧 after-replan 先完成当前复盘,再 fresh 原决定作为历史证据
新 fresh-replan 首轮即 fresh,重新准备和决定 原任务/研究文件,当前 guard;不得预置旧决定

表格来自源代码阅读,模型是否实际采用这些义务没有因表格或提示文本而获得证明。

具体改动

完整审阅 8 文件 +133/-15、一个带 DCO 的提交:CLI 新显式选项与互斥校验,WarmStart 的布尔校验和声明字段,新首轮 helper,以及既有会话刷新中恰一边界 basis、原 attempt/output 和 binding CAS 的复用。README 提供中英文处理方式与关闭方法;省略两选项维持历史接续,不自动安装、启动或恢复权限。

关键代码:prepare_session_refresh 先检查声明、原 checkpoint 及 Goal/Agent,再进入 _prepare_fresh_replan。后者验证 sealed held packet 原始字节和 TS owner 的原 pending replan,不能把归档命令当作执行授权。未见 attempt 时它再次调用完整 admit_native_checkpoint_session(...timeout=30);初始 prepare_context 已调用同一个 owner,第二次走的是已存在绑定的全量 history readback。只有通过后,CodexAgentSession.prepare_checkpoint_refresh 才落一次 attempt,清空本次输入 session_id,并由原 stdout 及当前 lifetime 下 CAS 读回新 ID。retained_native_history 明确保留来源而不声称保留已丢弃的 native prefix。

规范依据为不可变 base 中 checkpoint RFC 7.7 Research-state branching and matched continuation 及 benchmark README 的 warm composition、源历史完整性、原预算和未知结果恢复约束;本处理的关联验收是 partial,不是一个提示字符串或字段存在即可结案。没有另建 replan 决策 owner;通用会话模块保持 native IO 职责,没有加入 benchmark-specific quota 判定。

对主干的风险

独立四模块 89 passed / 12 skipped。另用既有 Codex 二进制、隔离 home 和本地静态响应运行原生 owner 场景:10 passed / 28 deselected,覆盖全历史/cursor、真实 fork/resume、profile/scope 与丢响应;这些没有执行新选项的成熟首轮处理。新入口的独立 35 秒完整读取适配器反例为 1 failed / 2 passed,只在私有预算反事实中改为 90 秒则 3 passed;artifact 选择和读取延迟是显式 stub,真实文件/attempt/session owner 被执行,不能把它包装成成熟原生端到端资格。

收到的独立 receiver 失败记录指向同一新 helper→已有绑定准入→全量 thread/turns/list,错误为 app_server_response_timeout:thread/turns/list。日志本身未携带 commit ID,因此仅作报告来源,与本次 current-source 因果分析分开;较小成功点不能抹掉该失败。请保留成熟失败样本,完成真实 warm admission→新首轮→fresh input→binding readback 的资格,并覆盖 advanced source 与 lost reply。

实际 base/head 的关闭路径、无效布尔和旧 refresh/原输出恢复结果一致;旧开启路径有一条错误诊断从 post-replan 改成 checkpoint,已逐字记录,没有归一化隐藏。Ruff、diff 和完整 semantic smoke 通过;完整 canary 的失败/跳过仍单列,未查询或等待 CI。共享 bank 并发导入的 exists→mkdir 竞态所在 checkpoint_controller.py 在实际 base/head 字节完全相同,作为既有运营资格缺口保留,不把它误报成本 diff 引入的回归;串行启动只是临时缓解,不能称并发资格通过。

本次风险 canary:5 项 direct 无失败,19 项 selected 中 18 项通过,1 项 maintainability advisory 失败;整体为 quality_receipt_missing,未借用作者 receipt。实际 base→6618 head→6622 head 的完整 ratchet JSON 只归一化 checkout 根目录后完全相同,保留 DSH 93 statements / 51 decisions 的既有失败,没有调阈值或扫描范围。

我的整体评价

REQUEST_CHANGES,绑定此未变 head。整体交付、long_horizon 与 user_experience 均为 not_yet_proven:预期首轮获得新的决定,当前却可能在第二次完整准入中耗尽局部 30 秒,产生准备成本而没有研究接续。最小修复与验收是 资格化这个第二次完整复核及其预算,在原累计截止时间内支持已接收的成熟点;可复用既有读取/会话 owner,但必须保留拒绝 advanced/foreign source、原结果恢复与未知启动不得重发。不要仅删除复核或修改历史实验阈值。 复现完整 warm admission→首轮 prepare_session_refresh→fresh 首次输入→原会话绑定读回,保留超过30秒成熟完整历史失败样本,并同时验证 lost reply/advanced source/default-off。。有界 future-facing pass 应继续共用 native/session owner 的截止时间和原结果恢复,不增加平行状态机。旧7b8头尚缺真实新首轮资格,不能用现有绿测或单个成功点替代。当前评审不批准 merge,也不把 integration 分支当成 main 发布。

English verdict: REQUEST_CHANGES — 6622@7b8b08f8a85dd7f0b87398d838af09f1b6a1bf03: first fresh-before-replan preparation repeats full native admission with a hard-coded30s allowance. Current-source/adapter counterfactual and a separately reported mature receiver timeout identify the uncovered route;10 real-native owner tests do not exercise that new complete path. Preserve advanced-source refusal and original lost-output recovery while qualifying it under the original cumulative deadline. The shared-bank race is unchanged at7b8; parent6618 is now independently approved, while the new-head first-wake qualification remains unreviewed here. No merge authority.

Specification source: docs/architecture/rfcs/hierarchical-agent-stride-control-v0.md#77-research-state-branching-and-matched-continuation, immutable revision 072178167e8e3389986502922495f2cc9a32e919. Related accepted benchmark/README.md section: Explicit warm worker composition (qualification in progress). The original source/authority invariants are implemented; the complete changed caller acceptance above is not_met.

后继修复 #6630 已由外部合入此分支,新 head 为 8548a5b5d352c2c11ce3ac66c22a20bf01041630。该新 head 尚未由本评审独立审阅;此 REQUEST_CHANGES 只绑定旧 7b8b08f8a85dd7f0b87398d838af09f1b6a1bf03,不能移用为新 head 的批准或缺陷结论。后续 Core adoption 修复由原 owner 继续新提案,作者报告不得替代独立资格。

@mergify

mergify Bot commented Oct 16, 2026

Copy link
Copy Markdown

Hi @loopx-agent, the DCO Sign-off check did not pass. Please inspect
its details first: checkout, fetch, timeout or infrastructure errors
need their own recovery, not a rewrite of otherwise signed commits.

If the log confirms a missing Signed-off-by trailer, amend the
affected commit with git commit --amend -s; for multiple commits,
use an interactive rebase against the current base from the correct
base-repository remote and sign off each affected commit. Push the
rewritten PR branch with git push --force-with-lease origin HEAD.

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · reasoning_effort=max · observation_id=93709c62b40bc6afa5cc5ee555f900164e54eab5aabc888fd499c776c37410ce

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

动机

研究操作者从同一决策前 checkpoint 重新准备并作出当前目标的首次路线决定。
此前先继承旧会话完成 replan 再清空;新选项让首次准备就使用 fresh 会话,同一 bank 的并发接收端也能完成各自的原生准入。
已独立验证真实 native 首次 fresh 输入、原输出恢复和真实 HTTP bank 并发导入;新头的提交签署仍有阻塞。
成熟点的 Core 完整上下文读取由后继 #6631 负责;真实新决策、后续封存及持续容量仍须各自验收。
本评审不证明科研收益、完整持续运行、通用恢复 UI 或 main 发布,也不授予合并或实验启动权限。

[P2] 补齐当前 squash 提交的 DCO sign-off。 当前提交 的完整 Git message 只有 subject,没有 Signed-off-by。规范依据为 AGENTS.md 的 DCO Sign-Off:每个 PR commit 都须签署。由作者沿原分支补签该提交并推送更正后的 head;保留原成果,再按新 exact head 重新评审和检查 readiness。不要仅重试 CI 或改写规则。 这是直接读 Git 对象得到的当前仓库契约缺口,不是等待 CI 的理由。该 finding 位于提交元数据,不伪造源码 inline 行。

改动思路

精确 head 8548a5b5d352c2c11ce3ac66c22a20bf01041630,真实 merge base / parent stack 072178167e8e3389986502922495f2cc9a32e919;已核对祖先与完整三点 diff,目标为 codex/edgebench-checkpoint-integration-20261016。这是实验整合分支,不等于 main;当前 main 的 #6618 是另一个 squash 提交,tree 同源不能代替真实祖先和新头评审。

保留现有 typed artifact 选择、native 准入与 session CAS,新增处理只组合这些 owner;锁仅保护 bank 字节发布。 当前 PR 的边界是显式 fresh-before-replan 与接收端导入可靠性;Core 读取后继和科研效用不在本头关闭。 实际入口是 CLI flags -> WarmStart.stage -> prepare_context/admit_warm_session -> prepare_session_refresh/_prepare_fresh_replan -> CodexAgentSession.prepare_checkpoint_refresh -> native_command -> original stdout -> session CAS。完整读取与过期拒绝是机器校验;给模型的原目标/反例/原预算说明只负责输入,不证明模型已采用。相关 #6630 确实已外部合入本头,非本评审执行;#6631 是 exact child。另已读 #6627:它仍 OPEN,移除旧 after-replan provider 路线,当前 PR 尚未采纳,整合时应按真实新头复核;不能因提出删除就提前删除原 persisted 结果恢复。

情况 预期路径与读回
未选 fresh treatment 原 retained conversation;冷入口没有 warm 声明或新 attempt
显式 fresh-before-replan typed 原 pending basis → 完整 native 复核 → 新首轮输入 → 原 stdout/CAS
到期或丢响应 到期拒绝下一新操作;未知结果只查原 identity,不重置机会或 deadline

此表是精确源代码解释,不是独立执行证据。

具体改动

完整阅读 10 文件 +392/-48,包含所有新增测试与 README,不以只看上轮修复代替全 PR 判断。规范依据:docs/architecture/rfcs/hierarchical-agent-stride-control-v0.md#77-research-state-branching-and-matched-continuation,不可变 revision 072178167e8e3389986502922495f2cc9a32e919。7.7 Research-state branching and matched continuation 对应完整历史、源证据与当前权限,已由真实 native/SQLite 检查实现;Explicit warm worker composition (qualification in progress) 对应原 phase、机会及准备成本,已实现;持续科研与通用 UI 属于保留的父边界。 DCO Sign-Off 对应本头缺签,判为 not_met。

关键代码讲解

  • _prepare_fresh_replan:Select sealed pending boundary and prepare fresh input。New explicit pre-replan variant; old completed-review variant remains. Sealed packet exact bytes and typed select_replan equality; attempt must be original.
  • admit_warm_session:Shared full-history admission allowance。Both initial and fresh revalidation now use min(90, original remaining phase). Finite original integer deadline; late success rejects.
  • _bank_import_lock:Serialize bank byte publication。Native HTTP two-run concurrent import base fails/head passes. Physical owner-only regular lock inode; bounded nonblocking flock; no unlink.
  • CodexAgentSession.prepare_checkpoint_refresh:Existing once-only native replacement and output recovery。Exactly one completed-review or pending-replan provenance basis. Original attempt/output and current binding CAS; missing/ambiguous output refuses redispatch.

当前 owner 分工仍是 typed artifact/continuation 负责选择与权限,native/session 负责完整会话与原结果恢复,Python provider 负责 IO、锁与 deadline 组合。fresh_replan is explicit irreducible experiment intent produced by CLI->WarmStart.stage, absent by default and mutually exclusive. pre_replan_basis derives from sealed packet bytes and the typed selector; it cannot be authored as work authority. Original attempt/binding owns replay and retirement. Bank lock is local concurrency state, not a judge admission receipt. 没有加入 substring 状态判断、通用 scheduler 规则或新的能力目录;类型化 advisory 无受支持新 carrier,完整 semantic smoke 通过,但空报告本身不证明语义安全。

对主干的风险

实际 base/head 的相同原生 HTTP 并发导入用例,base 为失败、head 为通过:修复前第二个发布者越过第一个的预留点,修复后两条独立 run/token 都保留,原 grading 次数不增加。新锁的争用超时与 symlink 拒绝也通过。共享 runtime/SForge/worker 三模块在同 Harbor0.20.0 环境下 base/head 各282 passed、31 optional skips;首次缺 Harbor 的 base 结果保留为无效环境对照,未归责产品。

本头另行通过真实 native+SQLite+artifact/session CAS 的 synthetic 首次 fresh 输入及原 stdout 恢复。生产 prepare_context、typed selector、native_command、当前绑定都实际执行;首次新会话请求不含源消息,丢响应后只恢复原输出,没有第三次模型请求。模型为本地静态响应;pending packet 是明确合成输入,不冒充真实生产 guard 或成熟私有点,未执行真实策略推理或 grading。两个初版私有测试的环境字段/调用签名错误已在执行前置层纠正,原日志保留,不把它们当产品缺陷。

本轮 scoped 验证:173 passed/19 optional skips; 282 passed/31 optional skips; synthetic real-native fresh path1 passed; same actual native HTTP bank workload base fails/head passes. Ruff、diff/compile 与完整 semantic smoke 通过。风险 canary 两头各5 direct通过、19 selected全部执行,18通过、1项既有 maintainability advisory failed。实际 base→6622→6631 三份完整 ratchet JSON 仅替换 checkout 根路径后完全一致:原 DSH _run_dsh_operation_host 93 statements /51 decisions 失败保留,未调阈值。当地精确 scope 的 premerge gate 仍为 quality_receipt_missing,benchmark-sensitive 人工合并 hold 保留,不能声称 premerge 全绿或有合并权限。

语义与 CI 对齐

这是显式新增的 pre-replan 输入语义:先读原目标和保留负证据,再在当前 guard 下准备与 replan;原 phase、submission、cooldown 和来源仍约束。关闭路径和旧 post-review 输入保留,新增输入的实际模型理解仍未测试。

按本 capability 的 wait_for_ci=false 未查询、轮询或等待 CI。收到的 managed-goal-results 失败报告不计作独立测试:已核 source fixture 与实际相关 frontend 在 base/head 字节相同,原 read-count 断言的主干修复 f94619f 已存在;未把未复跑的浏览器或其它手机失败说成已修。未运行的 Docker/Linux 联合 barrier 与作者成熟 guest 记录均明确保留,作者记录不代替本方验收。

我的整体评价

REQUEST_CHANGES,只绑定 6622@8548a5b5d352c2c11ce3ac66c22a20bf01041630。整体交付为 justified_increment;long_horizon 为 preserved,user_experience 为 improved,理由是完整历史与当前权限仍被核验、原结果能恢复,并减少在首个有效输入前失败或重复人工准备的机会。更长读取占用原预算,是明确成本;不把它或单个成功点说成持续性能资格。成熟点的 Core 完整上下文读取由后继 #6631 负责;真实新决策、后续封存及持续容量仍须各自验收。

保留现有 typed artifact 选择、native 准入与 session CAS,新增处理只组合这些 owner;锁仅保护 bank 字节发布。 当前 PR 的边界是显式 fresh-before-replan 与接收端导入可靠性;Core 读取后继和科研效用不在本头关闭。 有界 future-facing pass 已复用现有完整读取与原结果 owner;保留必要 persisted 兼容,#6627 的 provider 删除由后续真实整合处理,不增加平行状态机。源码路径与上轮30秒问题已重新核验,不再把旧7b8的结论移用到8548;当前具体阻塞是缺签提交,作者补签后必须重新绑定新头。

English verdict: REQUEST_CHANGES — 6622@8548a5b5d352c2c11ce3ac66c22a20bf01041630. The fresh first-input/recovery, bounded phase and concurrent bank paths are independently exercised; the new8548 squash commit lacks the repository-required DCO trailer. Correct that commit and review the new head. No merge authority.

@huangruiteng
huangruiteng merged commit e45cda8 into codex/edgebench-checkpoint-integration-20261016 Oct 16, 2026
25 of 38 checks passed
@huangruiteng
huangruiteng deleted the codex/checkpoint-fresh-replan-20261016 branch October 16, 2026 10:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants