Repository navigation
#649 [scripts][eval] ローカル LLM・実 provider を使う live 評価の入口がマシン共通の heavy-lock の枠を取る - #660
Merged
Merged
Conversation
Live Python evals now take the same mkdir slot as the with-heavy-lock wrapper, so two live runs serialize and exclude cargo tests and any other holder. scripts/eval_lib/heavy_lock.py provides the shared context manager: atomic mkdir, pid/cmd in the wrapper shape, HEAVY_LOCK_DIR/HEAVY_LOCK_HELD compatibility, stale reclaim of a confirmed-dead owner guarded by an atomic reclaim mutex, nested-call skip, child inheritance, release of only the lock this process owns, and append-only wait records in <run>/heavy-lock-waits.jsonl with nested calls distinguished. Wired: goal_verify live campaign (v0/v3/v4 and contract smoke), blind review, recovery-v4 smoke, the non-dry-run eval-run matrix (wait linked through the existing extras_json, no TSV column or schema change), and preflight with an explicit --live-provider-smoke. Dry-run, plain preflight, deterministic evaluation, and model-list GETs take no slot. Evidence: - python3 -m pytest tests/eval/test_heavy_lock.py -q -> 26 passed, 2 subtests passed - ruff check --isolated --select E4,E7,E9,F,I --ignore E402 <changed python> -> All checks passed - Two-point: mutating HeavyLock._release_own to ignore ownership makes test_release_leaves_a_slot_it_no_longer_owns fail (1 failed); restoring it returns to 26 passed. 判断: added an atomic sibling reclaim mutex and HEAVY_LOCK_POLL_SECONDS because a bare rmtree reclaim raced with a freshly acquired slot when several waiters detected the same dead owner (observed as a FileNotFoundError on pid write). 判断: wait time is recorded in a new append-only <run>/heavy-lock-waits.jsonl and only linked to the existing extras_json so frozen manifest, raw records, TSV header, and schema_version stay unchanged. 読み替え: the acceptance criterion "every generation of live campaign" is covered by a parametrized test that drives the live CLI with v3 and v4 contracts. 本文に無い指摘: pre-existing failures in tests/eval/test_goal_verify_live_v2.py, test_goal_verify_recovery_a15.py, and test_goal_verify_v3.py reproduce on the unmodified baseline and are not part of this issue's gates. Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概要
ローカル LLM や実 provider を使う live 評価の入口が、
with-heavy-lockと同じ「マシン共通の heavy-lock の枠」を取るようにしました。cargo などの重い処理と live 評価が同時に走らないようにするためです。Refs #649
変更（7 ファイル、+1081 −77、1 commit）
scripts/eval_lib/heavy_lock.py（新規）：with-heavy-lockと同じ枠を使います。slot.d、その中のpid・cmdの file を使います。HEAVY_LOCK_HELDがあれば入れ子として取得を省き、HEAVY_LOCK_DIRで置き場所を変えられます。.reclaimmutex の下で確かめ直してから回収します。pid が無い・壊れている・負の値・生きている、のどれかに当たる枠は回収しません。heavy-lock-waits.jsonlに追記します。再開しても上書きしません。with heavy_lock(...)で囲みました。eval-goal-verify-live.py：全世代の campaigneval-goal-verify-blind.py：旧 blind revieweval-goal-verify-recovery-v4.py：recovery smoke。実 product の子プロセスはHEAVY_LOCK_HELDを受け継ぎます。eval-run.py：dry-run でない matrix。dry-run の return の後で取ります。待ち時間と入れ子かどうかは events の extras（heavy_lock_wait_sec・heavy_lock_nested）に残します。TSV の header は変えていません。import の並び替え（I001）も入っています。eval-preflight.py：--live-provider-smokeのときだけ取ります。tests/eval/test_heavy_lock.py（新規）：26 本。v3・v4 は subTest で分けています。検証
93ff4469でcommandmate verifyを実行し、15/15 passed でした。clippy、cargo test --all-targets、CI の形の ruff、tests/evalの既存テスト、npm ci を含みます。python3 tests/eval/test_heavy_lock.py -v→ 26 ok。tests/は package ではありません。そのため既存のtests/evalと同じく、直接実行の形で流します。ruff check --isolated --select E4,E7,E9,F,I --ignore E402を 7 ファイルにかけ、All checks passed でした。9047eb35の入口 5 本に、新しいheavy_lock.pyとテストを置いて流しました。入口が枠を取ることを確かめるテストが 7 本 FAIL しました（live、live の v3・v4、blind、recovery、eval-run、preflight の smoke）。残る 20 本は ok です。HEAVY_LOCK_DIRを使います。/Volumes/SSD_NX/tmp/heavy-lock/slot.dには触れていません。smoke は mock で、実 provider には接続していません。確かめてほしい点
with-heavy-lockの wrapper にだけあり、Python の枠にはありません。本文の範囲どおり、この PR は枠の排他だけを扱います。HEAVY_LOCK_HELDがあると、持ち主が生きているかを確かめずに入れ子として扱います。wrapper と同じ挙動です。🤖 Generated with Claude Code