Problem
Helix Agent Lab uses an asynchronous local MCP server, deterministic prompt-case scoring, hidden simulator state, runtime capability descriptors, bounded logs, reproducible seeds, and explicit separation between stateful system benchmarks and prompt-only evaluations. It also keeps provider authentication inside official installed CLIs instead of turning consumer credentials into a general API.
Perseus has strong memory/context/evidence benchmarks, but its cross-runtime evaluation contract should make provider/auth mode, runtime capability, asynchronous terminal state, and benchmark-family parity equally explicit.
Proposed change
Add a reusable local runtime evaluation protocol in the appropriate Perseus benchmark/flywheel surface. Define:
- versioned observation/action envelopes;
- capability descriptors distinguishing persistent state, seeded memory, filesystem, tools, and feedback memory;
- account/environment auth mode and provider provenance without reading credentials;
- asynchronous run lifecycle and bounded artifact/log retention;
- deterministic assertions and hidden-state scenarios;
- separate score families for stateful system evaluations, prompt-case evaluations, and wrappers.
Acceptance criteria
- Every run emits a self-describing manifest containing suite digest, repo revision, runtime, model/provider/version, auth mode, seed, scorer version, scope, timestamps, and artifact digests.
- Run status distinguishes queued/running/passed/failed/cancelled/interrupted/failed_to_start.
- Account mode never reads or returns credential contents and uses explicit provider CLI boundaries where applicable.
- Stateful and prompt-only results cannot appear on one undifferentiated leaderboard.
- The offline harness runs without credentials/network; live-provider mode is explicit and reports authoritative provider usage separately from estimates.
- Cancellation, crash/restart, timeout, malformed output, and partial results are covered.
Grounding
Problem
Helix Agent Lab uses an asynchronous local MCP server, deterministic prompt-case scoring, hidden simulator state, runtime capability descriptors, bounded logs, reproducible seeds, and explicit separation between stateful system benchmarks and prompt-only evaluations. It also keeps provider authentication inside official installed CLIs instead of turning consumer credentials into a general API.
Perseus has strong memory/context/evidence benchmarks, but its cross-runtime evaluation contract should make provider/auth mode, runtime capability, asynchronous terminal state, and benchmark-family parity equally explicit.
Proposed change
Add a reusable local runtime evaluation protocol in the appropriate Perseus benchmark/flywheel surface. Define:
Acceptance criteria
Grounding