LoopArena evaluates models in the Controller role as they guide a separate, fixed coding Worker through long-horizon tasks. Controller models use the same Worker, task, starting workspace, public conversation, coding tools, budget, runtime, and terminal evaluator. Type II and Type III also report shared no-control and fixed-control reference policies; these provide context for the Controller results and are excluded from the Controller-model ranking.
Under the no-control reference policy, the Worker receives one instruction to complete the task and runs until it finishes or exhausts its budget.
Under the fixed-control reference policy, the Worker repeatedly receives the same task-level goal at loop handoffs. No model Controller reviews the run or adapts the instruction; the policy stops after the Worker explicitly reports that the goal is complete.
Under a Controller-model policy:
- the Worker receives a bounded assignment;
- when that assignment ends, a fresh Reporter reads the observable Worker conversation and inspects the current workspace with read-only tools;
- the Reporter produces a structured progress report with citations to the Worker conversation;
- the deterministic packet compiler combines that report with the cited Worker turns and remaining budget;
- the Controller returns
advance,verify, orstop; - an
advanceorverifyinstruction continues the same Worker conversation, whilestopends the episode.
No control has no Reporter, Controller, packet compilation, or periodic
interruption. The CLI retains --arm controlled|no-control as an
implementation-level execution-mode flag. Fixed control uses the controlled
execution mode with the deterministic fixed-control policy.
The Worker sees the public task, public conversation, current Controller instruction, ordinary coding tools, and the results of its own tool calls. It does not see private evaluator inputs or results.
The Reporter sees the public task, the complete observable Worker conversation, and a read-only view of the current workspace. It cannot edit files or execute commands.
The Controller sees the Reporter report, the Worker turns cited by that report, the remaining budget, and its own earlier reports and decisions. It cannot inspect the repository or private evaluator directly.
advanceassigns a bounded implementation, debugging, or cleanup step.verifyassigns evidence collection. It may use the ordinary coding tools but must not intentionally change production behavior.stopends the episode when the available evidence supports completion.
Controller output is a JSON object containing the decision, rationale, Worker instruction, protected invariants, and verification acceptance condition. Invalid output is recorded as a model outcome rather than retried into a different decision.
The main Worker has 600 model turns under every execution policy. Each Reporter call has at most 50 turns. A Controller-model run has at most 128 control cycles and 24 hours of Controller-channel wall time. Dataset runners may impose a shorter overall wall-time limit when the source task requires one.
One Worker response may make at most one tool call. Tool output and model input are bounded by the same deterministic context policy under every execution policy. Context compaction changes representation only when a provider input would exceed the declared capacity; the run records when it occurs.
Model access ends before terminal evaluation. SCBench evaluates a sealed copy of the final workspace after the solve container stops. BeyondSWE runs its source-native verifier at the final model boundary in the live source container, then attaches the evaluator receipt to the sealed solve result.
The terminal evaluator determines task success. A valid evaluator failure is a model result. Provider, runner, container, or evaluator infrastructure failures are recorded separately and are not converted into task failures.
Type II restores its frozen starting workspace and runs one declared task increment. Type III starts from the official problem origin; SCBench executes every native checkpoint in order, while BeyondSWE executes its complete task.
Each run records the observable Worker transcript, per-slice activity,
Reporter outputs, Controller inputs and outputs, token usage, runtime identity,
and terminal evaluator receipt. solve_manifest.json seals the state produced
by the models; run_manifest.json adds the terminal evaluation result.
Provider-interrupted runs may resume only from the last durable model boundary. Completed model work is not replayed. The resume record binds the task, workspace, execution policy, seed, model configuration, and previous progress needed to continue the same experiment.