Skip to content

About

TraceTrial-Bench: A benchmark for AI agent harness reliability and failure attribution, featuring 95 tasks across 7 components, execution tracing, and evidence-based evaluation that separates task success from harness reliability.

Resources

Stars

28 stars

Watchers

0 watching

Forks

Repository files navigation

TraceTrial-Bench

When an AI agent fails, was it the model — or the harness around it?

Code and tasks for Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution

Hugging Face: #2 Paper of the Day, October 27, 2026

Paper Report Leaderboard Hugging Face Dataset License: MIT

English · 简体中文

Overview · Results · Ecosystem impact · Quick start · Benchmark · Citation

95 tasks, 40 mechanism nodes, 7 reliability dimensions, and 3,420 paper runs

Overview

A coding/ops agent is a model plus a harness. The harness runs tools, manages context, asks for approval, spawns sub-agents and decides when the job is done. Claude Code, Codex, OpenClaw and DeepSeek Harness are the four harnesses evaluated in the paper.

Task success alone hides the source of a failure. A model can finish despite a silently dropped error; it can also fail when the harness behaves correctly. TraceTrial-Bench examines the execution trace and scores these two layers separately.

TraceTrial-Bench framework: task design, native execution, evidence acquisition and failure attribution

Framework overview: mechanism-focused tasks, multi-source evidence and run-level attribution. Click to view at full size. · Technical report

Stage What TraceTrial-Bench does
Run Executes 95 mechanism-focused tasks across 4 harnesses × 9 models, giving 36 combinations and 3,420 Docker runs.
Inspect Normalizes traces and extracts evidence about tool calls, context, execution and task outcomes.
Score Evaluates Harness behavior and Agent task performance independently.
Attribute Identifies harness-caused failures, agent-caused failures and harness defects that the agent worked around.

Results

31.1% of runs succeed at the task despite failing at least one harness criterion. Only 8.4% pass every criterion in both layers.

Claude Code has the highest Agent Pass rate; OpenClaw has the highest Harness Macro score. Harness Macro spans 12.2 percentage points, from 59.8% to 72.0%.

Overall scores for nine models and four harnesses

Overall performance: model scores average over four harnesses; harness scores average over nine models. · Technical report

Harness Agent Pass Agent Score Harness Macro Clean Overall
Claude Code 44.0 75.9 71.2 9.1 73.5
Codex 35.8 70.7 59.8 2.8 65.2
DeepSeek Harness 40.0 73.3 68.8 9.8 71.0
OpenClaw 38.2 71.7 72.0 11.7 71.8

All values are percentages, averaged over 9 models × 95 tasks (855 cells per harness). Bold marks the best value in each column.

How to read the metrics
  • Agent Pass: every Agent criterion passes.
  • Clean: every criterion in both layers passes.
  • Overall: mean of Agent Score and Harness Macro.

Per-model results and case studies are available in the technical report and on the leaderboard.

Ecosystem impact

Beyond scoring existing releases, TraceTrial-Bench's mechanism probes (C, E, G, L, O, T) are used to audit production open-source agent harnesses. To date, mechanism-guided testing has surfaced 17 real-world defects and runtime gaps across OpenHands SDK, OpenCode and Aider, backed by 14 upstream pull requests and independent maintainer reproductions — including five issues codified into OpenHands's official verify-sdk and verify-agent-server regression suites (095af36).

Controlled harness interventions (same model, patched harness)

Holding the model, task container and rubric fixed while patching only the implicated harness mechanism isolates the causal effect of harness engineering on end-to-end scores:

Target harness Node / Task Harness defect or gap addressed Upstream reference Score delta (Before → After)
OpenCode E9 (E9-ptyrepl-002) Native stateful pty session tool with EOF (Ctrl+D) and pattern wait #65017 · interactive-pty-tool Overall 37.5 → 62.5 (+25.0; 6/6 PTY checks 0 → 1)
OpenCode L4 / O / C2 Missing background <subagent> / <shell> inbox delivery and compaction events in run --format json #64851 · PR #64852 Overall 45.0 → 60.0 (+15.0; Harness 20.0 → 35.0)
OpenHands SDK T3 (T3-reserved-tool-name-001) Single-server MCP tool colliding with built-in terminal raises Duplicate tool names found on startup #6992 · PR #6988 Overall 0.0 → 33.3 (Harness 0.0 → 42.9)
Aider G / E1 (E1-drainexit-001) Non-interactive --yes-shell-commands execution and shell output reflection in --message mode PR #6002 Overall 25.0 → 37.5 (Harness 14.3 → 28.6)
Full upstream bug and feature tracker (14 PRs across OpenHands, OpenCode and Aider)
Harness Node Issue & upstream triage Pull request Defect / capability summary
OpenHands SDK G2 × L #6999 (priority:high, ready-for-dev, reproduced in S11/task_fail_closed) PR #7001 Sub-agents fail open on WAITING_FOR_CONFIRMATION when confirmation_handler=None and drop parent security_analyzer
OpenHands SDK G4 × C2 × L #7002 (priority:medium, ready-for-dev) PR #7003 SecretRegistry.update_secrets() leaves unexported and rotated secret values unmasked across parent/sub-agent observations
OpenHands SDK L #6990 (priority:medium, ready-for-dev, reproduced in S11/task_resume_session) PR #6986 Persisted sub-agent task index and original subagent_type are lost when reopening a parent conversation
OpenHands SDK O × L #3907 (priority:medium, ready-for-dev) PR #6997 Real-time forwarding of sub-agent events with parent_tool_use_id attribution
OpenHands SDK C2 #6995 (priority:medium, ready-for-dev) PR #6998 Entity-anchor-preserving context condensation (<preserved_facts>) and compression telemetry
OpenHands SDK E2 #6991 (priority:low, ready-for-dev, reproduced in S14/terminal_flags) PR #6987 TerminalObservation.timeout remains False on both hard-timeout and no-change-timeout branches
OpenHands SDK T3 #6992 (priority:low, ready-for-dev, reproduced in S09/conversation) PR #6988 Namespace colliding single-server MCP tools as <server>_<tool> instead of aborting initialization
OpenHands SDK T7 #6993 (priority:low, ready-for-dev, reproduced in S09/conversation) PR #6989 Preserve MCP EmbeddedResource text and structuredContent-only payloads in MCPToolObservation
OpenHands SDK E3 / E6 #7017 (priority:low, ready-for-dev) PR #7025 Opt-in hard_timeout foreground process-group reaping (SIGINT → SIGTERM → SIGKILL) and error signaling
OpenHands SDK C1 #6994 (priority:low) PR #7024 Machine-readable TruncationMetadata with line-range accounting and secret-masked spillover persistence
OpenCode T7 #64837 (type/bug, status/repro-simulated) PR #64838 Direct MCP tools drop structuredContent, structured errors and textual EmbeddedResource metadata
OpenCode G2 × L #64840 (type/bug, status/repro-simulated) PR #64841 SubagentTool ignores session.permissions and fails to cascade parent permission updates across descendants
OpenCode L4 / O / C2 #64851 (type/feature, area/cli) PR #64852 Stream and reconcile synthetic background notifications and compaction completions in opencode run --format json
Aider G / E1 #6003 (related headless telemetry RFC) PR #6002 Headless --yes-shell-commands execution and shell output reflection in --message runs

Community runs

SophiaLabs: Nomos × TraceTrial-Bench compares stock Claude Code with Claude Code + Nomos on 95 tasks (190 selected runs, one retained per arm per task). The release includes traces, configurations, and scorecards.

We verified the published trace-package checksums and recalculated the headline averages from the scorecards as stored. We did not re-score or rerun the tasks. Two task pairs used different time budgets between arms. Two scorecards record an Agent item score of 2 or 3 where the rubric requires 0 or 1. These community-reported results are not an official leaderboard entry or an independent reproduction.

See SophiaLabs’ technical report for their experimental analysis.

Submit a community run

Ran TraceTrial-Bench with your own harness, model, or configuration? Open an issue titled Community run: <name> and include:

  • a public link pinned to a commit or release, containing your traces, resolved run configurations, and combined_score.json scorecards;
  • checksums for any trace packages;
  • the task manifest you used and how many runs you kept per task;
  • for paired comparisons, confirmation that both arms used the same tasks, time budgets, and judge model;
  • a link to your write-up, if you have one.

We check what the published files allow (for example, package checksums and headline averages recalculated from the scorecards) and list the run here as community-reported. A community run is not an official leaderboard entry, and listing it does not mean we have reproduced it.

Quick start

Prerequisites: Python ≥ 3.11 and Docker. Commands below use a Linux shell; on Windows, use WSL.

1. Install and configure

git clone https://github.com/TraceLite-AI/TraceTrial-Bench.git
cd TraceTrial-Bench
python3 -m venv .venv
.venv/bin/pip install -e .
cp configs/models.example.toml model.toml
chmod 600 model.toml

Edit model.toml with your provider endpoint, model identifier and API key. The commands below use its deepseek alias. Keep this credential file private.

2. Build a harness and run one task

.venv/bin/tracetrial_bench image build --harness openclaw \
  --model-registry ./model.toml --output-dir outputs/image-builds

.venv/bin/tracetrial_bench run --task Task/C1-escalation-export-001 --harness openclaw --model deepseek \
  --mode docker --model-registry ./model.toml --output-dir outputs/runs

3. Run the paper benchmark

.venv/bin/tracetrial_bench run --task-manifest Task/final.manifest \
  --harness openclaw --model deepseek --judge-model deepseek \
  --mode docker --concurrency 4 --model-registry ./model.toml --output-dir outputs/runs

The DeepSeek Harness image now installs dsh 0.2.2-rc.5 and runs its official sdk profile. The paper's DeepSeek Harness results used 0.1.0-rc.6; to reproduce them, check out the paper-v1 branch. dsh 0.2 calls models through the Anthropic Messages API, so give it a model alias whose base_url is https://api.deepseek.com/anthropic (see configs/models.example.toml).

4. Inspect the report

.venv/bin/tracetrial_bench report outputs/runs/<run-directory>
Output Contents
result.json Status, scores, tokens, cost and timing
Normalized trace Model and harness execution evidence
artifacts/verifier_output/score.json Rule-check results
artifacts/llm_judge/combined_score.json Harness and Agent layer scores

Benchmark

Each task targets one of 7 reliability dimensions. Prompts provide a goal, materials and a deliverable without revealing the mechanism being tested.

Code Dimension What it covers Tasks Nodes
C Context Information preservation, isolation and recovery 15 6
E Execution Process semantics and faithful result delivery 27 9
G Governance Permissions, policies and security boundaries 9 4
L Lifecycle Session transitions, delegation and state recovery 19 8
O Observability Faithful diagnostic and failure channels 7 2
T Tools Tool interfaces, discovery and transport fidelity 17 10
V Verification Completion claims versus authoritative outcomes 1 1
Total 95 40

Start with the 95-task manifest. The manifest follows the top-to-bottom task order in the paper heatmap on page 45.

Task IDs follow <node>-<description>-<NNN>, for example C1-escalation-export-001. A leading L3- denotes Lifecycle node 3; difficulty metadata is unchanged. The historical lean96.manifest retains the additional hostrecord task excluded from the paper results.

Repository guide

Path Purpose
Task/ Benchmark tasks and published manifests
tracetrial_bench/ CLI, Docker runner, scoring, attribution and reports
harness_tracer/ Harness adapters and normalized traces
verifier/ Evidence layer (trace_ir) and API traffic capture
docker/ Pinned harness images
configs/ Model registry example

Citation

Please cite our paper on arXiv:

@misc{tracelite2026trialbytrace,
  title  = {Trial by Trace: Benchmarking AI Agent Harnesses
            for Reliability and Failure Attribution},
  author = {Cao, Zhiyong and Zhu, Hengjie and Wang, Yongxin and Li, Xuchen and
            Li, Jintao and Liu, Shuxia and Liu, Dunqiang and Wang, Bo and
            Zhang, Kairan and Lin, Cong and Yu, JiangDi and Liu, Siyuan and
            Yao, Sijia and Wei, Yi-Liang and Ma, Lingxu and Wang, Aijia and
            Chen, Anxin and Wei, Zixian and Ruan, Xiang and Xu, Ruizhi and
            Shi, Ruqian and Xiao, Baihua and Zhou, Sicheng},
  year   = {2026},
  eprint = {2610.32285},
  archivePrefix = {arXiv},
  primaryClass = {cs.AI},
  url    = {https://arxiv.org/abs/2610.32285}
}

Star History

TraceTrial-Bench GitHub Star history chart


Built by TraceLite AI · Contact · MIT License

Open research on agent evaluation and post-training: forensics, attribution, environments, rewards and data generation.

About

TraceTrial-Bench: A benchmark for AI agent harness reliability and failure attribution, featuring 95 tasks across 7 components, execution tracing, and evidence-based evaluation that separates task success from harness reliability.

Resources

Stars

28 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages