Repository navigation
fix(bench): SHIM_TIMEOUT below the openswarm budget so budget-exhausted cells keep their executor events (#213) - #214
Merged
Conversation
…ed cells keep their executor events (#213) arm_openswarm passed SHIM_TIMEOUT=budget_min*60+30 while openswarm's own budget is budget_min*60s, so whenever the executor ran the budget out openswarm killed the backend shim (rc=124) before run_pi returned and the shim never wrote executor-*-record.json / executor-*-events.jsonl. The 840s-budget / 870s-timeout validation cell (sympy__sympy-13878 r1, run 20261117-013602-da5999) lost 843s of executor work: no events file, executor usage all 0, solution.diff 0 bytes. SHIM_TIMEOUT is now shim_timeout_s(budget_min) = max(30, budget_min*60-60): 840s budget -> 780s, leaving 60s for run_pi's timeout path (SIGTERM + up to 10s grace, then the record print) plus the shim's copy + cat of the events file. Covered by bench/swebench/test_shim_timeout.py (stdlib-only, same ast pattern as test_planner_prompt.py).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
缺陷
arm_openswarm传给 shim 的SHIM_TIMEOUT=budget_min*60+30,而 openswarm 自己的预算只有budget_min*60秒。执行器跑满预算时 openswarm 先杀掉后端 shim(attempt rc=124),shim 在run_pi返回前就死了,来不及写executor-*-record.json、也来不及把 pi 事件拷贝为executor-*-events.jsonl;openswarm 自己捕获的events.1.pi.jsonl也是 0 字节。未修复的验证格(sympy__sympy-13878 r1,run 20261117-013602-da5999):840 秒预算 vs 870 秒超时,843 秒执行器工作 0 diff、executor usage 全 0。修复
新增纯函数
shim_timeout_s(budget_min) = max(30, budget_min*60-60),arm_openswarm改用它设置 SHIM_TIMEOUT。840 秒预算 → 780 秒,留 60 秒 flush 余量:run_pi.py超时路径会 SIGTERM 整个 pi 进程组、最多等 10 秒后 SIGKILL,随后仍打印含events_file的 record,shim 有时间写 record.json、拷贝并 cat 事件给 openswarm。只改这一处,不做 SIGTERM 落盘、不做流式输出。单测
新增
bench/swebench/test_shim_timeout.py(stdlib-only,与 test_planner_prompt.py 同款 ast 模式):对全部 budget_min 1–60 断言超时严格小于预算;验证格预算(14 分钟)断言 780 秒且 ≤800 秒;小预算下限 30 秒;调用点使用 helper 且旧公式已消除。python3 -m pytest test_shim_timeout.py test_planner_prompt.py test_diff_filter.py test_cost_field.py:23 passed。复验(同一单格,修复后代码重跑)
timed_out=true、wall 961.4 秒——超时先触发、正常返回、shim 完成 finalizationownrepo-1117-m16s2-shimfix-validate/VALIDATION.md注意:本修复不能热生效,合入跑批须重启 run.py
SHIM_TIMEOUT 的数值是每格现算的,但公式在
arms.py模块代码里;run.py在main()启动时一次性import arms,全 bench 代码无importlib.reload;在跑的 run.py PID 2092 自 2026-11-17 05:48 起一直在跑,盘上改arms.py不会被它采纳。是否、何时合入跑批检出并重启跑批,由主轮决定;本 PR 不自行合并。