When an AI agent fails, was it the model — or the harness around it?
Code and tasks for Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution
English · 简体中文
Overview · Results · Ecosystem impact · Quick start · Benchmark · Citation
A coding/ops agent is a model plus a harness. The harness runs tools, manages context, asks for approval, spawns sub-agents and decides when the job is done. Claude Code, Codex, OpenClaw and DeepSeek Harness are the four harnesses evaluated in the paper.
Task success alone hides the source of a failure. A model can finish despite a silently dropped error; it can also fail when the harness behaves correctly. TraceTrial-Bench examines the execution trace and scores these two layers separately.
Framework overview: mechanism-focused tasks, multi-source evidence and run-level attribution. Click to view at full size. · Technical report
| Stage | What TraceTrial-Bench does |
|---|---|
| Run | Executes 95 mechanism-focused tasks across 4 harnesses × 9 models, giving 36 combinations and 3,420 Docker runs. |
| Inspect | Normalizes traces and extracts evidence about tool calls, context, execution and task outcomes. |
| Score | Evaluates Harness behavior and Agent task performance independently. |
| Attribute | Identifies harness-caused failures, agent-caused failures and harness defects that the agent worked around. |
31.1% of runs succeed at the task despite failing at least one harness criterion. Only 8.4% pass every criterion in both layers.
Claude Code has the highest Agent Pass rate; OpenClaw has the highest Harness Macro score. Harness Macro spans 12.2 percentage points, from 59.8% to 72.0%.
Overall performance: model scores average over four harnesses; harness scores average over nine models. · Technical report
| Harness | Agent Pass | Agent Score | Harness Macro | Clean | Overall |
|---|---|---|---|---|---|
| Claude Code | 44.0 | 75.9 | 71.2 | 9.1 | 73.5 |
| Codex | 35.8 | 70.7 | 59.8 | 2.8 | 65.2 |
| DeepSeek Harness | 40.0 | 73.3 | 68.8 | 9.8 | 71.0 |
| OpenClaw | 38.2 | 71.7 | 72.0 | 11.7 | 71.8 |
All values are percentages, averaged over 9 models × 95 tasks (855 cells per harness). Bold marks the best value in each column.
How to read the metrics
- Agent Pass: every Agent criterion passes.
- Clean: every criterion in both layers passes.
- Overall: mean of Agent Score and Harness Macro.
Per-model results and case studies are available in the technical report and on the leaderboard.
Beyond scoring existing releases, TraceTrial-Bench's mechanism probes (C, E, G, L, O, T) are used to audit production open-source agent harnesses. To date, mechanism-guided testing has surfaced 17 real-world defects and runtime gaps across OpenHands SDK, OpenCode and Aider, backed by 14 upstream pull requests and independent maintainer reproductions — including five issues codified into OpenHands's official verify-sdk and verify-agent-server regression suites (095af36).
Holding the model, task container and rubric fixed while patching only the implicated harness mechanism isolates the causal effect of harness engineering on end-to-end scores:
| Target harness | Node / Task | Harness defect or gap addressed | Upstream reference | Score delta (Before → After) |
|---|---|---|---|---|
| OpenCode | E9 (E9-ptyrepl-002) |
Native stateful pty session tool with EOF (Ctrl+D) and pattern wait |
#65017 · interactive-pty-tool |
Overall 37.5 → 62.5 (+25.0; 6/6 PTY checks 0 → 1) |
| OpenCode | L4 / O / C2 |
Missing background <subagent> / <shell> inbox delivery and compaction events in run --format json |
#64851 · PR #64852 | Overall 45.0 → 60.0 (+15.0; Harness 20.0 → 35.0) |
| OpenHands SDK | T3 (T3-reserved-tool-name-001) |
Single-server MCP tool colliding with built-in terminal raises Duplicate tool names found on startup |
#6992 · PR #6988 | Overall 0.0 → 33.3 (Harness 0.0 → 42.9) |
| Aider | G / E1 (E1-drainexit-001) |
Non-interactive --yes-shell-commands execution and shell output reflection in --message mode |
PR #6002 | Overall 25.0 → 37.5 (Harness 14.3 → 28.6) |
Full upstream bug and feature tracker (14 PRs across OpenHands, OpenCode and Aider)
| Harness | Node | Issue & upstream triage | Pull request | Defect / capability summary |
|---|---|---|---|---|
| OpenHands SDK | G2 × L |
#6999 (priority:high, ready-for-dev, reproduced in S11/task_fail_closed) |
PR #7001 | Sub-agents fail open on WAITING_FOR_CONFIRMATION when confirmation_handler=None and drop parent security_analyzer |
| OpenHands SDK | G4 × C2 × L |
#7002 (priority:medium, ready-for-dev) |
PR #7003 | SecretRegistry.update_secrets() leaves unexported and rotated secret values unmasked across parent/sub-agent observations |
| OpenHands SDK | L |
#6990 (priority:medium, ready-for-dev, reproduced in S11/task_resume_session) |
PR #6986 | Persisted sub-agent task index and original subagent_type are lost when reopening a parent conversation |
| OpenHands SDK | O × L |
#3907 (priority:medium, ready-for-dev) |
PR #6997 | Real-time forwarding of sub-agent events with parent_tool_use_id attribution |
| OpenHands SDK | C2 |
#6995 (priority:medium, ready-for-dev) |
PR #6998 | Entity-anchor-preserving context condensation (<preserved_facts>) and compression telemetry |
| OpenHands SDK | E2 |
#6991 (priority:low, ready-for-dev, reproduced in S14/terminal_flags) |
PR #6987 | TerminalObservation.timeout remains False on both hard-timeout and no-change-timeout branches |
| OpenHands SDK | T3 |
#6992 (priority:low, ready-for-dev, reproduced in S09/conversation) |
PR #6988 | Namespace colliding single-server MCP tools as <server>_<tool> instead of aborting initialization |
| OpenHands SDK | T7 |
#6993 (priority:low, ready-for-dev, reproduced in S09/conversation) |
PR #6989 | Preserve MCP EmbeddedResource text and structuredContent-only payloads in MCPToolObservation |
| OpenHands SDK | E3 / E6 |
#7017 (priority:low, ready-for-dev) |
PR #7025 | Opt-in hard_timeout foreground process-group reaping (SIGINT → SIGTERM → SIGKILL) and error signaling |
| OpenHands SDK | C1 |
#6994 (priority:low) |
PR #7024 | Machine-readable TruncationMetadata with line-range accounting and secret-masked spillover persistence |
| OpenCode | T7 |
#64837 (type/bug, status/repro-simulated) |
PR #64838 | Direct MCP tools drop structuredContent, structured errors and textual EmbeddedResource metadata |
| OpenCode | G2 × L |
#64840 (type/bug, status/repro-simulated) |
PR #64841 | SubagentTool ignores session.permissions and fails to cascade parent permission updates across descendants |
| OpenCode | L4 / O / C2 |
#64851 (type/feature, area/cli) |
PR #64852 | Stream and reconcile synthetic background notifications and compaction completions in opencode run --format json |
| Aider | G / E1 |
#6003 (related headless telemetry RFC) | PR #6002 | Headless --yes-shell-commands execution and shell output reflection in --message runs |
SophiaLabs: Nomos × TraceTrial-Bench compares stock Claude Code with Claude Code + Nomos on 95 tasks (190 selected runs, one retained per arm per task). The release includes traces, configurations, and scorecards.
We verified the published trace-package checksums and recalculated the headline averages from the scorecards as stored. We did not re-score or rerun the tasks. Two task pairs used different time budgets between arms. Two scorecards record an Agent item score of 2 or 3 where the rubric requires 0 or 1. These community-reported results are not an official leaderboard entry or an independent reproduction.
See SophiaLabs’ technical report for their experimental analysis.
Ran TraceTrial-Bench with your own harness, model, or configuration? Open an issue titled Community run: <name> and include:
- a public link pinned to a commit or release, containing your traces, resolved run configurations, and
combined_score.jsonscorecards; - checksums for any trace packages;
- the task manifest you used and how many runs you kept per task;
- for paired comparisons, confirmation that both arms used the same tasks, time budgets, and judge model;
- a link to your write-up, if you have one.
We check what the published files allow (for example, package checksums and headline averages recalculated from the scorecards) and list the run here as community-reported. A community run is not an official leaderboard entry, and listing it does not mean we have reproduced it.
Prerequisites: Python ≥ 3.11 and Docker. Commands below use a Linux shell; on Windows, use WSL.
git clone https://github.com/TraceLite-AI/TraceTrial-Bench.git
cd TraceTrial-Bench
python3 -m venv .venv
.venv/bin/pip install -e .
cp configs/models.example.toml model.toml
chmod 600 model.tomlEdit model.toml with your provider endpoint, model identifier and API key. The commands below use its deepseek alias. Keep this credential file private.
.venv/bin/tracetrial_bench image build --harness openclaw \
--model-registry ./model.toml --output-dir outputs/image-builds
.venv/bin/tracetrial_bench run --task Task/C1-escalation-export-001 --harness openclaw --model deepseek \
--mode docker --model-registry ./model.toml --output-dir outputs/runs.venv/bin/tracetrial_bench run --task-manifest Task/final.manifest \
--harness openclaw --model deepseek --judge-model deepseek \
--mode docker --concurrency 4 --model-registry ./model.toml --output-dir outputs/runsThe DeepSeek Harness image now installs dsh 0.2.2-rc.5 and runs its official sdk profile. The paper's DeepSeek Harness results used 0.1.0-rc.6; to reproduce them, check out the paper-v1 branch. dsh 0.2 calls models through the Anthropic Messages API, so give it a model alias whose base_url is https://api.deepseek.com/anthropic (see configs/models.example.toml).
.venv/bin/tracetrial_bench report outputs/runs/<run-directory>| Output | Contents |
|---|---|
result.json |
Status, scores, tokens, cost and timing |
| Normalized trace | Model and harness execution evidence |
artifacts/verifier_output/score.json |
Rule-check results |
artifacts/llm_judge/combined_score.json |
Harness and Agent layer scores |
Each task targets one of 7 reliability dimensions. Prompts provide a goal, materials and a deliverable without revealing the mechanism being tested.
| Code | Dimension | What it covers | Tasks | Nodes |
|---|---|---|---|---|
| C | Context | Information preservation, isolation and recovery | 15 | 6 |
| E | Execution | Process semantics and faithful result delivery | 27 | 9 |
| G | Governance | Permissions, policies and security boundaries | 9 | 4 |
| L | Lifecycle | Session transitions, delegation and state recovery | 19 | 8 |
| O | Observability | Faithful diagnostic and failure channels | 7 | 2 |
| T | Tools | Tool interfaces, discovery and transport fidelity | 17 | 10 |
| V | Verification | Completion claims versus authoritative outcomes | 1 | 1 |
| Total | 95 | 40 |
Start with the 95-task manifest. The manifest follows the top-to-bottom task order in the paper heatmap on page 45.
Task IDs follow <node>-<description>-<NNN>, for example C1-escalation-export-001. A leading L3- denotes Lifecycle node 3; difficulty metadata is unchanged. The historical lean96.manifest retains the additional hostrecord task excluded from the paper results.
| Path | Purpose |
|---|---|
Task/ |
Benchmark tasks and published manifests |
tracetrial_bench/ |
CLI, Docker runner, scoring, attribution and reports |
harness_tracer/ |
Harness adapters and normalized traces |
verifier/ |
Evidence layer (trace_ir) and API traffic capture |
docker/ |
Pinned harness images |
configs/ |
Model registry example |
Please cite our paper on arXiv:
@misc{tracelite2026trialbytrace,
title = {Trial by Trace: Benchmarking AI Agent Harnesses
for Reliability and Failure Attribution},
author = {Cao, Zhiyong and Zhu, Hengjie and Wang, Yongxin and Li, Xuchen and
Li, Jintao and Liu, Shuxia and Liu, Dunqiang and Wang, Bo and
Zhang, Kairan and Lin, Cong and Yu, JiangDi and Liu, Siyuan and
Yao, Sijia and Wei, Yi-Liang and Ma, Lingxu and Wang, Aijia and
Chen, Anxin and Wei, Zixian and Ruan, Xiang and Xu, Ruizhi and
Shi, Ruqian and Xiao, Baihua and Zhou, Sicheng},
year = {2026},
eprint = {2610.32285},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2610.32285}
}Built by TraceLite AI · Contact · MIT License
Open research on agent evaluation and post-training: forensics, attribution, environments, rewards and data generation.