[sharareh-y]

Time budget mode for CLI agents and per-task agent cost records

Summary

This PR adds two agent-level features:

1. Time budget mode for Claude Code, Cursor, Codex and DeepSeek Harness. With agent.budget_mode: time, timeout_seconds is the agent's wall-clock budget for a task. If the agent ends its session early, Arena resumes the same session with a prompt that states the time remaining, until the original deadline. The default iterations mode is unchanged.
2. Per-task cost records for all agents. After each task's agent run, main.py writes <run_dir>/agent_costs/<task_workspace>.json with the dollars and tokens the provider reports. Values that are not reported are null. Arena never estimates dollars from tokens.

agent:
  template: claude_code
  budget_mode: time        # iterations (default) | time
  timeout_seconds: 21600

Time budget mode

• The deadline is fixed when the first session starts, after setup and baseline measurement and before candidate evaluation. Those phases do not use the budget.
• A continuation starts only while at least 10 minutes remain (MIN_RESUME_SECONDS = 600). At most 20 sessions run per task (MAX_SESSIONS).
• Each continuation's timeout is the time left to the original deadline, so the agent can't run past it by more than the time it takes to stop the process.
• A continuation stopped at the deadline is not an agent failure. A first session that times out, a nonzero exit, or a failed result still is. Arena evaluates the files at the declared candidate paths in every case.
• To resume, time mode keeps the provider's session state:
  • Claude Code runs with --session-id instead of --no-session-persistence;
  • Codex runs without --ephemeral;
  • DeepSeek Harness reuses one task-local DSH_HOME and passes the first run's session ID through --session-id, with separate prompt/log artifacts for each continuation.
• main.py rejects an unknown budget_mode for every agent when the run starts. It also rejects time for agents other than Claude Code, Cursor, Codex and DeepSeek Harness: GEAK and Forge keep their own budget settings.
• In iterations mode, the command, the prompt text (byte for byte), the single run with the full timeout, and the failure behavior are the same as before.

Agent cost records

Agent	Dollars	Tokens
claude_code	result.total_cost_usd	result.usage
codex	null (not reported by the CLI)	turn.completed.usage
cursor	null (not reported by the CLI)	result.usage, when the CLI version emits it
geak	Claude Agent SDK ResultMessage.total_cost_usd	ResultMessage.usage
forge	KernelForge llm_usage.total_cost_usd, unless cost_available is false	KernelForge llm_usage counters
deepseek_harness	null	final --json event usage, when emitted
task_validator	null	null

• Each record contains:
  • total_cost_usd and usage, totalled over all attempts and sessions (or null);
  • cost_status and usage_status, each one of complete, partial, not_reported or ambiguous;
  • attempts, one per launch of the task: its run settings, agent seconds, any launcher error, and the raw provider reports.
• Resumed sessions are not counted twice. Codex token usage and the dollar totals from Claude Code 2.1.277 and later are running totals for the session, so only the increase is added. Values from older Claude Code versions are summed. If the version can't be parsed and a session was resumed, the dollar total is null with status ambiguous.
• A later --resume-run adds an attempt to the same record.
• A failed cost record never changes the task outcome. The record is written in a finally block, never raises, and leaves no temporary file behind.

Where the code lives

agents/ keeps only agent integrations, and agents/run_budget.py is unchanged.

Path	Contents
src/tools/agent_time_budget.py	budget_mode validation and the shared session loop
src/prompts/agent_budget.py	Budget prompt text (the iterations sentence is unchanged)
src/tools/agent_cost.py	Cost collection, normalization and the per-task record; called from main.py
agents/<name>/	Each CLI's resume arguments and output parsing
main.py	budget_mode check at run start; wraps each agent launcher with the cost collector and writes the record after the task

Docs

• docs/how-to/agents.md: new sections "Iteration and time budgets" and "Agent cost records".
• Reference pages: docs/reference/agent-model-defaults.md and docs/reference/api-reference.md are updated.
• README.md: a short paragraph that links to both sections.
• Release notes: not updated in this PR.

Testing

• New tests:
  • tests/test_agent_time_budget.py and tests/test_agent_cost.py;
  • additions to tests/test_cli_agents.py, tests/test_deepseek_harness.py, tests/test_process_control.py and tests/test_main.py.
• What they cover:
  • the session loop and the deadline, continuation and failure rules;
  • iterations mode unchanged;
  • cost normalization and running-total accounting, and parsing each provider's report;
  • the budget_mode check at run start;
  • a null record for an agent that reports nothing;
  • a write failure that leaves no temporary file.
• Before the rebase onto main:
  • the targeted files passed (145 tests);
  • the full tests/ suite failed on exactly the same tests as on main. Those failures come from torch not being installed in the test environment.
• After rebasing onto 099e44cf:
  • the Codex launcher was updated for StreamLog and src.process_control. Each Codex session gets its own StreamLog, because a stream log file is created once and time mode resumes Codex. Recording errors from any session are raised after the sessions end.
  • The targeted files plus tests/test_agent_stream_log.py pass on this commit: 149 tests and 9 subtests.

Earlier end-to-end runs on MI355X (Claude Code and Cursor)

One task (hip2hip/GELU), budget_mode: time, timeout_seconds: 1800, Docker runner, one GPU per run:

	Claude Code 2.1.294 (claude-opus-5-5, effort high)	Cursor Agent 2026.10.16 (composer-2.5)
Sessions	20, all in one session (--session-id, then --resume <id>); ended at the MAX_SESSIONS cap	3; ended with 494 s left, below MIN_RESUME_SECONDS
Agent time	1,161 s	1,306 s
Result	correct, accepted, speedup 2.50×, score 370.3	correct, accepted, speedup 1.83×, score 303.3
Cost record	$2.64 (complete); 44,655 output tokens	dollars null (not_reported); 51,239 output tokens (complete)

• Claude Code dollars: 2.1.294 reports total_cost_usd as a running total for the session. The 20 reported values only increase, from 0.606 to 2.644, so the record keeps the last value instead of summing them; summing would give about $37. Tokens are reported per invocation and are summed.
• After a run Claude Code considers finished, later continuations can be short replies such as "every case is at its practical limit". The loop keeps resuming until the deadline, the minimum remaining time or the session cap, which keeps the budget fixed. In this run, sessions 10–20 took 5–9 s each.

DeepSeek Harness continuation validation

DeepSeek Harness now uses the same fixed-deadline loop. The initial prompt includes total seconds; subsequent prompts state remaining minutes. Continuations retain the same task directory, isolated home and exact session ID. Each task launch or retry gets a fresh home. Missing IDs stop continuation; failed adoption and mismatched identities never fall back to a new conversation. Default iterations-mode prompt text, single invocation and artifact paths are retained.

• Focused CPU/mock tests: 188 passed, 9 subtests passed on macOS with Python 3.12.11. Coverage includes history isolation, decreasing timeouts, independent round logs, missing/lost/mismatched session IDs, nonzero exits, deadline exhaustion, token-record aggregation and tool-child cleanup.
• Real published @deepseek-ai/dsh@0.1.6-alpha.2 and @deepseek-ai/dsh-headless@0.1.6-alpha.2, using a loopback mock provider: two CLI invocations reused one persisted session; the continuation's provider request contained the original user message and assistant reply. The timeout decreased from 60 seconds to approximately 59.24 seconds. This smoke test used MIN_RESUME_SECONDS=0 and MAX_SESSIONS=2 only within its test process; production limits remain 600 seconds and 20 invocations.
• DSH uses shared process-group cleanup. On macOS, an EPERM for a disappearing group is ignored only when a fixed system process-table query proves the group absent; live groups, failed/malformed queries and Linux permission errors remain failures.
• make check-docker-runner, make check-evaluator (9 tests), make check-perf-helpers, focused source compilation and git diff --check passed. The Docker runner check was offline and used GNU coreutils plus a canonical temporary directory on macOS.
• The DSH addition has not been validated against a real DeepSeek model or on GPU hardware. The earlier Claude Code/Cursor GPU results above are separate evidence.