Repository navigation
fix(parsec): wait for simulator quiescence before seeding, and raise the agent ceiling - #694
Merged
Merged
Conversation
…the agent ceiling Two independent failure modes found while running the h4 held-out suite, both of which produce results that look valid. **Seeding races an in-flight tool call.** `PUT /api/v1/simulation/database` is refused with HTTP 409 while a service still has a call in flight. A trial ends when the *agent* stops, which does not wait for the simulators it stopped talking to, so an abandoned call outlives the trial that made it — usually on a service the *next* task never touches. Arms that seeded only each task's own declared services never saw this: the service the current agent used is quiet by construction, because the agent waited for its answer. Seeding every service exposes it, and it cost 69 errored trials in a single sweep. `_await_quiescent_sims()` polls every service in `MCP_PORTS` to `queue_depth == 0` before seeding. It lives here rather than inside `install_seeds.py` because that script runs under the `timeout=120` subprocess cap below: a drain budget inside it races its own caller, which is how an earlier attempt converted a recoverable wait into a dead trial. A service that cannot be polled counts as busy, never as idle — seeding into a service whose state is unknown is the thing being prevented. **Trials outlive the task's agent timeout.** Degraded gateway latency pushed 10-18 call tasks past the task's own `[agent] timeout_sec` (600s in h4). Harbor cancels the agent, the trial still writes a `reward.json`, the verifier's completion gate multiplies it by zero, and it lands in the results as a **0.0 indistinguishable from a wrong answer** — with `errored_trials` still reading **0**, because from the harness's point of view the trial completed. 15% of one arm's trials died that way against 0.7% in its neighbours. Both knobs are needed: `AGENT_TIMEOUT_MULTIPLIER` raises the ceiling Harbor enforces, and `TRIAL_TIMEOUT_SEC` must stay well above it or this subprocess cap preempts the ceiling it exists to contain. A test asserts that ordering, since getting it backwards is the same nesting mistake as the drain budget above. Revert both to `20 * 60` / `1.0` once the gateway slowdown is understood: a raised timeout makes a *failing* trial cost 2.5x more wall clock, and it did not rescue the one task that thrashes on a deliberately unmodelled tool. Ports come from `parsec_paths.MCP_PORTS` rather than a second table, with service names derived from its keys. Note: `test_adapter.py` has two pre-existing failures on `origin/main` (`TASK_ID` assertions left behind when that support was removed). They are untouched here and fail identically before and after this change. Signed-off-by: boazc <boazc@il.ibm.com>
Contributor
|
🏷️ Automatic Labeling I've analyzed this pull request and added the following labels:
These labels were selected based on the PR title, description, and changed files. If you believe any labels are incorrect or missing, feel free to adjust them manually. |
Re-expanded during the port onto main's diverged adapter. The instruction to keep this to a single line was given on the earlier edit, and I re-authored the hunk from scratch instead of carrying over the approved text. Signed-off-by: boazc <boazc@il.ibm.com>
This was referenced Oct 8, 2026
OsherElhadad
approved these changes
Oct 12, 2026
AIMikav
pushed a commit
to AIMikav/cap-evolve
that referenced
this pull request
Oct 12, 2026
…e shared pieces
scripts/parsec/ -> examples/parsec/, where the other benchmarks live. All moves
are git mv, so history follows.
The shared pieces come up to the example root, which both matches the
examples/<name>/{adapters,optimizer}/ convention and drops an indirection that
was never meaningful:
v4_t2_e1/common/adapters/ -> adapters/
v4_t2_e1/common/optimizer/ -> optimizer/
v4_t2_e1/common/parsec_paths.py -> parsec_paths.py
v4_t2_e1/tests/ -> tests/
The adapter and optimizer were already shared rather than copied — both arms
symlink the same files into every generated project — so keeping them nested
inside one arm's directory misrepresented the dependency. h4's constant was
literally named V4T2_DIR while pointing at files it co-owns; it is now
EXAMPLE_DIR.
Path fixes:
* adapter.py's _REPO_ROOT: parents[5] -> parents[3], two levels shallower.
_COMMON_DIR stays parents[1] — it resolved to common/ and now resolves to
the example root, which is where parsec_paths.py went.
* every arm driver's REPO_ROOT stays parents[3]: scripts/parsec/<arm>/ and
examples/parsec/<arm>/ are the same depth.
* tests/ moved up one level, so parents[1]/"common"/"adapters" -> parents[1]/
"adapters", parents[4]/"core" -> parents[3]/"core", and the three driver
modules are now reached through parents[1]/"v4_t2_e1".
* COMMON_ADAPTERS_SRC/COMMON_OPTIMIZER_SRC in both scaffolds now resolve from
the example root instead of a common/ child.
Also updated outside the moved tree: RUN_H4.md, docs/how-to/ccc/
CCC_PODMAN_SETUP.md, scripts/ccc/parsec_stack.sh and .gitignore all named the
old path. Repo-wide grep for "scripts/parsec" is now empty.
Adds examples/parsec/README.md — short, in the shape of toy_calc's and
toy_skill's rather than the three PROMPT.md files, because those are
paste-to-your-agent intake prompts and parsec was onboarded months ago by a
different route; writing one would describe a flow that never happened. It
covers the layout, the two env vars and why neither has a default, that the
bundles and results live on the parsec-history branch, and the three
trial-integrity detectors under h4/audit/.
Tests: 69 pass. The 2 failures in test_adapter.py are pre-existing on main
(stale TASK_ID assertions, flagged in skillberry-ai#694) and fail identically before this
change; the next commit addresses them separately.
Signed-off-by: boazc <boazc@il.ibm.com>
AIMikav
pushed a commit
to AIMikav/cap-evolve
that referenced
this pull request
Oct 12, 2026
Both have failed on main since TASK_ID support was removed from the adapter (flagged in skillberry-ai#694, unrelated to the move in the previous commit). Separate commit so it can be dropped on its own. test_raises_clearly_when_task_id_unset asserted "TASK_ID" appears in the construction error. The adapter now raises about PARSEC_V4N instead, so the assertion was stale while its intent — an unconfigured checkout must stop construction rather than default to one — still holds. Renamed and re-pointed at the message actually raised. test_raises_clearly_when_task_dir_missing expected FileNotFoundError from resolving a single task that no longer exists as a concept. Verified what the multi-task adapter actually does with an empty tasks dir: it constructs fine and tasks("val") returns []. That is worth pinning precisely BECAUSE it does not raise — a run against the wrong checkout would otherwise "succeed" having measured nothing, and a zero task count is the only signal it happened. Suite is now 71 passed, 0 failed. Signed-off-by: boazc <boazc@il.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two failure modes found while running the h4 held-out suite. Both produce results that look valid, which is why they went unnoticed for several arms.
1. Seeding races an in-flight tool call
PUT /api/v1/simulation/databaseis refused with HTTP 409 while a service still has a call in flight. A trial ends when the agent stops — which does not wait for the simulators it stopped talking to — so an abandoned call outlives the trial that made it, usually on a service the next task never touches.Arms that seeded only each task's declared services never hit this: the service the current agent used is quiet by construction, because the agent waited for its answer. Seeding every service exposes it, and it cost 69 errored trials in one sweep.
_await_quiescent_sims()polls every service inMCP_PORTStoqueue_depth == 0first. It lives in the adapter, not ininstall_seeds.py, because that script runs under thetimeout=120subprocess cap — a drain budget inside it races its own caller, which is how an earlier attempt turned a recoverable wait into a dead trial. An unreadable service counts as busy, never idle: seeding into a service whose state is unknown is the thing being prevented.2. Trials outlive the task's agent timeout
Degraded gateway latency pushed 10-18 call tasks past the task's own
[agent] timeout_sec(600s in h4). Harbor cancels the agent, the trial still writes areward.json, the verifier's completion gate multiplies it by zero, and it lands in the results as a 0.0 indistinguishable from a wrong answer — witherrored_trialsstill reading 0, because the trial completed as far as the harness can tell.15% of one arm's trials died that way, against 0.7% in its neighbours.
Both knobs are needed: the multiplier raises the ceiling Harbor enforces, and
TRIAL_TIMEOUT_SECmust stay above it or this subprocess cap preempts the ceiling it exists to contain. A test asserts that ordering, because getting it backwards is the same nesting mistake as the drain budget in §1.These are tuning values, not permanent: revert to
20 * 60/1.0once the gateway slowdown is understood. A raised timeout makes a failing trial cost 2.5x more wall clock, and it did not rescue the one task that thrashes on a deliberately unmodelled tool.Notes for review
parsec_paths.MCP_PORTSrather than a second table; service names are derived from its keys.test_adapter.pyhas two pre-existing failures onorigin/main(TASK_IDassertions left behind when that support was removed). Untouched here; they fail identically before and after. Happy to fix them in a separate PR if wanted.