Skip to content

fix(parsec): wait for simulator quiescence before seeding, and raise the agent ceiling - #694

Merged
OsherElhadad merged 2 commits into
mainfrom
parsec/h4-sim-quiescence
Oct 12, 2026
Merged

OsherElhadad merged 2 commits into
mainfrom
parsec/h4-sim-quiescence

Conversation

@bcarmeli

@bcarmeli bcarmeli commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Two failure modes found while running the h4 held-out suite. Both produce results that look valid, which is why they went unnoticed for several arms.

1. Seeding races an in-flight tool call

PUT /api/v1/simulation/database is refused with HTTP 409 while a service still has a call in flight. A trial ends when the agent stops — which does not wait for the simulators it stopped talking to — so an abandoned call outlives the trial that made it, usually on a service the next task never touches.

Arms that seeded only each task's declared services never hit this: the service the current agent used is quiet by construction, because the agent waited for its answer. Seeding every service exposes it, and it cost 69 errored trials in one sweep.

_await_quiescent_sims() polls every service in MCP_PORTS to queue_depth == 0 first. It lives in the adapter, not in install_seeds.py, because that script runs under the timeout=120 subprocess cap — a drain budget inside it races its own caller, which is how an earlier attempt turned a recoverable wait into a dead trial. An unreadable service counts as busy, never idle: seeding into a service whose state is unknown is the thing being prevented.

2. Trials outlive the task's agent timeout

Degraded gateway latency pushed 10-18 call tasks past the task's own [agent] timeout_sec (600s in h4). Harbor cancels the agent, the trial still writes a reward.json, the verifier's completion gate multiplies it by zero, and it lands in the results as a 0.0 indistinguishable from a wrong answer — with errored_trials still reading 0, because the trial completed as far as the harness can tell.

15% of one arm's trials died that way, against 0.7% in its neighbours.

Both knobs are needed: the multiplier raises the ceiling Harbor enforces, and TRIAL_TIMEOUT_SEC must stay above it or this subprocess cap preempts the ceiling it exists to contain. A test asserts that ordering, because getting it backwards is the same nesting mistake as the drain budget in §1.

These are tuning values, not permanent: revert to 20 * 60 / 1.0 once the gateway slowdown is understood. A raised timeout makes a failing trial cost 2.5x more wall clock, and it did not rescue the one task that thrashes on a deliberately unmodelled tool.

Notes for review

  • Ports come from parsec_paths.MCP_PORTS rather than a second table; service names are derived from its keys.
  • 5 new tests cover: all-idle, naming the busy service, unreadable-is-not-idle, draining mid-wait, and the timeout ordering invariant.
  • test_adapter.py has two pre-existing failures on origin/main (TASK_ID assertions left behind when that support was removed). Untouched here; they fail identically before and after. Happy to fix them in a separate PR if wanted.

…the agent ceiling

Two independent failure modes found while running the h4 held-out suite, both
of which produce results that look valid.

**Seeding races an in-flight tool call.** `PUT /api/v1/simulation/database` is
refused with HTTP 409 while a service still has a call in flight. A trial ends
when the *agent* stops, which does not wait for the simulators it stopped
talking to, so an abandoned call outlives the trial that made it — usually on a
service the *next* task never touches. Arms that seeded only each task's own
declared services never saw this: the service the current agent used is quiet
by construction, because the agent waited for its answer. Seeding every service
exposes it, and it cost 69 errored trials in a single sweep.

`_await_quiescent_sims()` polls every service in `MCP_PORTS` to
`queue_depth == 0` before seeding. It lives here rather than inside
`install_seeds.py` because that script runs under the `timeout=120` subprocess
cap below: a drain budget inside it races its own caller, which is how an
earlier attempt converted a recoverable wait into a dead trial. A service that
cannot be polled counts as busy, never as idle — seeding into a service whose
state is unknown is the thing being prevented.

**Trials outlive the task's agent timeout.** Degraded gateway latency pushed
10-18 call tasks past the task's own `[agent] timeout_sec` (600s in h4). Harbor
cancels the agent, the trial still writes a `reward.json`, the verifier's
completion gate multiplies it by zero, and it lands in the results as a **0.0
indistinguishable from a wrong answer** — with `errored_trials` still reading
**0**, because from the harness's point of view the trial completed. 15% of one
arm's trials died that way against 0.7% in its neighbours.

Both knobs are needed: `AGENT_TIMEOUT_MULTIPLIER` raises the ceiling Harbor
enforces, and `TRIAL_TIMEOUT_SEC` must stay well above it or this subprocess cap
preempts the ceiling it exists to contain. A test asserts that ordering, since
getting it backwards is the same nesting mistake as the drain budget above.

Revert both to `20 * 60` / `1.0` once the gateway slowdown is understood: a
raised timeout makes a *failing* trial cost 2.5x more wall clock, and it did not
rescue the one task that thrashes on a deliberately unmodelled tool.

Ports come from `parsec_paths.MCP_PORTS` rather than a second table, with
service names derived from its keys.

Note: `test_adapter.py` has two pre-existing failures on `origin/main`
(`TASK_ID` assertions left behind when that support was removed). They are
untouched here and fail identically before and after this change.

Signed-off-by: boazc <boazc@il.ibm.com>
@skillberry-bot skillberry-bot added bug Something isn't working honesty Honesty/consistency of reported results (brand-critical) priority-p1 High impact labels Oct 8, 2026
@skillberry-bot

Copy link
Copy Markdown
Contributor

🏷️ Automatic Labeling

I've analyzed this pull request and added the following labels:

  • bug - honesty - bug - honesty - priority-p1

These labels were selected based on the PR title, description, and changed files. If you believe any labels are incorrect or missing, feel free to adjust them manually.

Re-expanded during the port onto main's diverged adapter. The instruction to
keep this to a single line was given on the earlier edit, and I re-authored the
hunk from scratch instead of carrying over the approved text.

Signed-off-by: boazc <boazc@il.ibm.com>
@OsherElhadad
OsherElhadad merged commit 74ccf95 into main Oct 12, 2026
16 checks passed
@OsherElhadad
OsherElhadad deleted the parsec/h4-sim-quiescence branch October 12, 2026 07:52
AIMikav pushed a commit to AIMikav/cap-evolve that referenced this pull request Oct 12, 2026
…e shared pieces

scripts/parsec/ -> examples/parsec/, where the other benchmarks live. All moves
are git mv, so history follows.

The shared pieces come up to the example root, which both matches the
examples/<name>/{adapters,optimizer}/ convention and drops an indirection that
was never meaningful:

    v4_t2_e1/common/adapters/  -> adapters/
    v4_t2_e1/common/optimizer/ -> optimizer/
    v4_t2_e1/common/parsec_paths.py -> parsec_paths.py
    v4_t2_e1/tests/            -> tests/

The adapter and optimizer were already shared rather than copied — both arms
symlink the same files into every generated project — so keeping them nested
inside one arm's directory misrepresented the dependency. h4's constant was
literally named V4T2_DIR while pointing at files it co-owns; it is now
EXAMPLE_DIR.

Path fixes:

  * adapter.py's _REPO_ROOT: parents[5] -> parents[3], two levels shallower.
    _COMMON_DIR stays parents[1] — it resolved to common/ and now resolves to
    the example root, which is where parsec_paths.py went.
  * every arm driver's REPO_ROOT stays parents[3]: scripts/parsec/<arm>/ and
    examples/parsec/<arm>/ are the same depth.
  * tests/ moved up one level, so parents[1]/"common"/"adapters" -> parents[1]/
    "adapters", parents[4]/"core" -> parents[3]/"core", and the three driver
    modules are now reached through parents[1]/"v4_t2_e1".
  * COMMON_ADAPTERS_SRC/COMMON_OPTIMIZER_SRC in both scaffolds now resolve from
    the example root instead of a common/ child.

Also updated outside the moved tree: RUN_H4.md, docs/how-to/ccc/
CCC_PODMAN_SETUP.md, scripts/ccc/parsec_stack.sh and .gitignore all named the
old path. Repo-wide grep for "scripts/parsec" is now empty.

Adds examples/parsec/README.md — short, in the shape of toy_calc's and
toy_skill's rather than the three PROMPT.md files, because those are
paste-to-your-agent intake prompts and parsec was onboarded months ago by a
different route; writing one would describe a flow that never happened. It
covers the layout, the two env vars and why neither has a default, that the
bundles and results live on the parsec-history branch, and the three
trial-integrity detectors under h4/audit/.

Tests: 69 pass. The 2 failures in test_adapter.py are pre-existing on main
(stale TASK_ID assertions, flagged in skillberry-ai#694) and fail identically before this
change; the next commit addresses them separately.

Signed-off-by: boazc <boazc@il.ibm.com>
AIMikav pushed a commit to AIMikav/cap-evolve that referenced this pull request Oct 12, 2026
Both have failed on main since TASK_ID support was removed from the adapter
(flagged in skillberry-ai#694, unrelated to the move in the previous commit). Separate
commit so it can be dropped on its own.

test_raises_clearly_when_task_id_unset asserted "TASK_ID" appears in the
construction error. The adapter now raises about PARSEC_V4N instead, so the
assertion was stale while its intent — an unconfigured checkout must stop
construction rather than default to one — still holds. Renamed and re-pointed
at the message actually raised.

test_raises_clearly_when_task_dir_missing expected FileNotFoundError from
resolving a single task that no longer exists as a concept. Verified what the
multi-task adapter actually does with an empty tasks dir: it constructs fine
and tasks("val") returns []. That is worth pinning precisely BECAUSE it does
not raise — a run against the wrong checkout would otherwise "succeed" having
measured nothing, and a zero task count is the only signal it happened.

Suite is now 71 passed, 0 failed.

Signed-off-by: boazc <boazc@il.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working honesty Honesty/consistency of reported results (brand-critical) priority-p1 High impact

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants