Skip to content

fix(bench): SHIM_TIMEOUT below the openswarm budget so budget-exhausted cells keep their executor events (#213) - #214

Merged
surlymochan merged 1 commit into
mainfrom
m16s2-shim-timeout-213
Nov 17, 2026
Merged

surlymochan merged 1 commit into
mainfrom
m16s2-shim-timeout-213

Conversation

@surlymochan

Copy link
Copy Markdown
Owner

缺陷

arm_openswarm 传给 shim 的 SHIM_TIMEOUT=budget_min*60+30,而 openswarm 自己的预算只有 budget_min*60 秒。执行器跑满预算时 openswarm 先杀掉后端 shim(attempt rc=124),shim 在 run_pi 返回前就死了,来不及写 executor-*-record.json、也来不及把 pi 事件拷贝为 executor-*-events.jsonl;openswarm 自己捕获的 events.1.pi.jsonl 也是 0 字节。未修复的验证格(sympy__sympy-13878 r1,run 20261117-013602-da5999):840 秒预算 vs 870 秒超时,843 秒执行器工作 0 diff、executor usage 全 0。

修复

新增纯函数 shim_timeout_s(budget_min) = max(30, budget_min*60-60),arm_openswarm 改用它设置 SHIM_TIMEOUT。840 秒预算 → 780 秒,留 60 秒 flush 余量:run_pi.py 超时路径会 SIGTERM 整个 pi 进程组、最多等 10 秒后 SIGKILL,随后仍打印含 events_file 的 record,shim 有时间写 record.json、拷贝并 cat 事件给 openswarm。只改这一处,不做 SIGTERM 落盘、不做流式输出。

单测

新增 bench/swebench/test_shim_timeout.py(stdlib-only,与 test_planner_prompt.py 同款 ast 模式):对全部 budget_min 1–60 断言超时严格小于预算;验证格预算(14 分钟)断言 780 秒且 ≤800 秒;小预算下限 30 秒;调用点使用 helper 且旧公式已消除。python3 -m pytest test_shim_timeout.py test_planner_prompt.py test_diff_filter.py test_cost_field.py:23 passed。

复验(同一单格,修复后代码重跑)

  • 格:openswarm / sympy__sympy-13878 / r1;planner 95.4 秒 → budget_min=17 → 预算 1020 秒,SHIM_TIMEOUT=960 秒(旧公式会是 1050 秒)
  • run_pi record:timed_out=true、wall 961.4 秒——超时先触发、正常返回、shim 完成 finalization
  • executor-1-events.jsonl:56,613,606 字节、3046 行,已落盘(未修复格缺失);openswarm 自捕的 events.1.pi.jsonl 同样 56.6MB(未修复格 0 字节)
  • executor usage:44 turns / 47 tools,input 115449 / output 77109 / cache_read 2429824,cost $0.114937(未修复格全 0、仅计 planner 的 $0.021012)
  • grade:f2p 0/1、p2p 19/19,resolved=false——与未修复格相同;本复验标准是事件落盘而非解出
  • 复验在独立克隆(本分支 @ 1e901b5)跑,--out/--raw-root 独立,OPENSWARM_DIR 指验证员的 swrm @ 5aa074f;证据见工作区 ownrepo-1117-m16s2-shimfix-validate/VALIDATION.md

注意:本修复不能热生效,合入跑批须重启 run.py

SHIM_TIMEOUT 的数值是每格现算的,但公式在 arms.py 模块代码里;run.py 在 main() 启动时一次性 import arms,全 bench 代码无 importlib.reload;在跑的 run.py PID 2092 自 2026-11-17 05:48 起一直在跑,盘上改 arms.py 不会被它采纳。是否、何时合入跑批检出并重启跑批,由主轮决定;本 PR 不自行合并。

…ed cells keep their executor events (#213)

arm_openswarm passed SHIM_TIMEOUT=budget_min*60+30 while openswarm's own
budget is budget_min*60s, so whenever the executor ran the budget out
openswarm killed the backend shim (rc=124) before run_pi returned and the
shim never wrote executor-*-record.json / executor-*-events.jsonl. The
840s-budget / 870s-timeout validation cell (sympy__sympy-13878 r1, run
20261117-013602-da5999) lost 843s of executor work: no events file,
executor usage all 0, solution.diff 0 bytes.

SHIM_TIMEOUT is now shim_timeout_s(budget_min) = max(30, budget_min*60-60):
840s budget -> 780s, leaving 60s for run_pi's timeout path (SIGTERM + up
to 10s grace, then the record print) plus the shim's copy + cat of the
events file. Covered by bench/swebench/test_shim_timeout.py (stdlib-only,
same ast pattern as test_planner_prompt.py).
@surlymochan
surlymochan merged commit 14e8015 into main Nov 17, 2026
2 checks passed
@surlymochan
surlymochan deleted the m16s2-shim-timeout-213 branch November 17, 2026 02:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant