Skip to content

docs(evaluator): document the Gym and Fabric runners, fix the runner protocol #1420

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
SandyChapman wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs(evaluator): document the Gym and Fabric runners, fix the runner protocol #1420

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
269 changes: 269 additions & 0 deletions docs/evaluator/agent-eval/fabric-runner.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,269 @@
---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

title: "Evaluate with a NeMo Fabric Harness"
description: "Run a coding agent through NeMo Fabric and score it with agent-eval — one config selects the harness (Codex, Claude, Hermes), and Fabric returns an ATIF trajectory a metric can grade on."
---

[NeMo Fabric](https://github.com/nvidia/nemo-fabric) runs an agent **harness** rather than a single
agent. Which harness runs is selected entirely by `config["harness"]["adapter_id"]`, so one runtime
covers several agent frontends without changing your evaluation. Fabric returns an **ATIF
trajectory** alongside the final answer, so a metric can score *how* the agent worked, not just what
it answered.

The result and bundle are the same as the
[quickstart](/documentation/evaluate-models/agent-eval/quickstart) — only the runner changes.

## Harnesses

| `adapter_id` | Harness | Transport |
|---|---|---|
| `nvidia.fabric.codex` | Codex CLI | `cli` (subprocess) |
| `nvidia.fabric.claude` | Claude | `cli` |
| `nvidia.fabric.hermes` | Hermes SDK | `library` (in-process) |
| `nvidia.fabric.langchain.deepagents` | LangChain deepagents | `library` |

<Warning>

This runner is **not** zero-dependency. It needs:

- **The harness adapters**, from the `fabric` extra:

```bash
uv sync --frozen --package nemo-evaluator-sdk --extra fabric --inexact
```

- **The `nemo-relay` gateway binary**, which captures the trajectory for out-of-process harnesses.
The pip package ships bindings only, so the daemon comes from a GitHub release asset:

```bash
script/dev-install-fabric.sh
```

- **The harness's own CLI**, for `transport: cli` harnesses — `codex` on `PATH` and authenticated.

The `fabric` extra installs the Codex, Claude, and Hermes adapters. The deepagents adapter is
deliberately excluded from it, because that adapter does not support the Relay observability
configuration Fabric streaming generates; install its harness separately if you need it.

</Warning>

## The agent config

One mapping describes the whole agent — harness, runtime, environment, and model. This is a working
Codex configuration:

```python
config = {
"schema_version": "fabric.agent/v1alpha1",
"metadata": {"name": "eval-fabric"},
"harness": {
"adapter_id": "nvidia.fabric.codex",
"resolution": "preinstalled",
"settings": {"sandbox": "workspace-write"},
},
"runtime": {
"mode": "oneshot",
"transport": "cli",
"input_schema": "text",
"output_schema": "message",
"timeout_seconds": 180,
},
"environment": {"provider": "local"},
"models": {"default": {"provider": "openai", "model": "<provider-model>"}},
"telemetry": {"enabled": False},
}
```

An `environment.workspace` set here is **overridden per task** — the runtime gives every task its own
fresh workspace under `work_root`, so setting one in the config has no effect.

Across harnesses the shape differs mainly in `adapter_id`, `runtime.transport`, and any
harness-specific `harness.settings`. Codex runs as a subprocess (`transport: cli`) while the Hermes
SDK harness runs in-library (`transport: library`). For complete Codex-CLI and Hermes-SDK
configurations, see `examples/fabric_harness_runtimes.py` in the SDK.

<Warning>

**The Codex adapter requires an explicit model provider.** It does not fall back to the Codex CLI's
own configured default, and starting without `models.default` fails the adapter lifecycle with
`codex_invalid_configuration`.

</Warning>

Fold the complete configuration into this mapping. Fabric profile overlays are not used here.

## Run it

```python
from pathlib import Path

from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
from nemo_evaluator_sdk.agent_eval.runtimes.fabric.runtime import FabricAgentRuntime
from nemo_evaluator_sdk import StringCheckMetric
from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask

tasks = [
AgentEvalTask(
id="reply-ok",
intent="Reply with a fixed token.",
inputs={"instruction": "Reply with the word OK."},
metrics=[
StringCheckMetric(
operation="contains",
left_template="{{sample.output_text}}",
right_template="OK",
)
Comment thread
coderabbitai[bot] marked this conversation as resolved.
],
)
]

runtime = FabricAgentRuntime(
config=config,
work_root="fabric",
capture_trajectory=True,
)

result = AgentEvaluator().run_sync(
tasks=tasks,
target=runtime,
config=AgentEvalRunConfig(work_dir=Path("out"), parallelism=1),
)
```

### Configuration

| Field | Required | Notes |
|---|---|---|
| `config` | yes | the agent config mapping above |
| `model` | no | overrides `models.default.model` without editing the config |
| `base_dir` | no | base directory for relative paths in the config |
| `work_root` | no | where Fabric's per-run working directories are created |
| `timeout_s` | no | overall run timeout, default `600` |
| `capture_trajectory` | no | capture the ATIF trajectory as evidence, default `True` |
| `trajectory_extra` | no | extra fields merged into the captured trajectory |
| `runtime_name` | no | the name recorded on `runner_info`, default `fabric` |
| `skills` | no | skills injected into the agent's workspace |
| `task_hook` | no | a `FabricTaskRunHook` invoked around each task run |

## Seed files into the workspace

Every task runs in its own fresh workspace. Put files the agent should start from in the task's
`inputs["files"]`, and the runtime stages them before the harness runs:

```python
task = AgentEvalTask(
id="read-seed",
intent="Read a seeded file and echo its contents.",
inputs={
"instruction": "Read notes.txt in your workspace and reply with exactly its contents.",
"files": {"notes.txt": "MARKER-7f3a"},
},
metrics=[
StringCheckMetric(
operation="contains",
left_template="{{sample.output_text}}",
right_template="MARKER-7f3a",
)
],
)
```

With no `files` key this is a no-op, so tasks that need no starting state cost nothing.

## Read the results

The answer is on the trial, and scores come back under the metric's type:

```python
trial = result.trials[0]
print(trial.status) # AgentEvalTrialStatus.COMPLETED
assert trial.output is not None
print(trial.output.output_text) # MARKER-7f3a

for score in result.summary.scores.scores:
print(score.name)
# string-check.string-check
# string-check.string-check.pass@1
```

### Evidence

This is where Fabric differs from the other runners. Each trial carries several evidence streams, so
a metric can score *how* the agent worked rather than only its final answer:

| Key | Kind | What it is |
|---|---|---|
| `trace` | `trace` (format `atif`) | the ATIF trajectory — tool calls, reasoning, observations |
| `workspace` | `filesystem` | the workspace's final file tree, after the agent finished |
| `result` | `json` | the harness's structured result |
| `stdout` | `log` | harness stdout |
| `relay_atif` / `relay_atof` | `atif` / `atof` | the raw Relay artifacts the trajectory is promoted from |
| `relay_config` | `telemetry_config` | the Relay configuration used for the run |

```python
assert trial.evidence is not None
trace = trial.evidence.descriptors["trace"]
# trace.format == "atif"; trace.ref is a path to the trajectory JSON, which has a "steps" key

workspace = trial.evidence.descriptors["workspace"]
# workspace.ref is a directory — what the agent left behind, including any seeded files
```

The `workspace` tree is what lets a metric grade artifacts the agent produced on disk, and `trace`
is what lets it grade the process. See
[Writing Metrics](/documentation/evaluate-models/agent-eval/writing-metrics) for reading evidence,
and [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component) for combining
several signals into one reported score.

## Submit as a platform job

`FabricRunnerTarget` carries the same config into a durable job:

```python
from nemo_evaluator.jobs.agent_spec import FabricRunnerTarget

target = FabricRunnerTarget(config=config, model="<provider>/<model>", capture_trajectory=True)
```

| Field | Required | Notes |
|---|---|---|
| `config` | yes | the same agent config, as a JSON-shaped mapping |
| `model` | no | a `provider/model` slug applied as the config's default model |
| `timeout_s` | no | per-task timeout for the Fabric run |
| `capture_trajectory` | no | capture the ATIF trajectory and attach it to trial evidence |

Submit it with `nemo evaluator agent-evaluate submit`, or as the `target` of an
`AgentEvalInputSpec`. See
[Targets and Runners](/documentation/evaluate-models/agent-eval/targets-and-runners).

## Running in a sandbox

`FabricContainerRuntime` takes the same config and runs it inside a sandbox rather than on the host:

```python
from nemo_evaluator_sdk.agent_eval.runtimes.fabric.container_runtime import FabricContainerRuntime
from nemo_evaluator_sdk.agent_eval.runtimes.sandbox.providers.docker import DockerSandboxProvider

runtime = FabricContainerRuntime(config=config, provider=DockerSandboxProvider())
```

| Field | Required | Notes |
|---|---|---|
| `config` | yes | the same agent config, or a `FabricConfig` |
| `provider` | yes | a `SandboxProvider` — `DockerSandboxProvider` or `DockerComposeSandboxProvider` |
| `secrets` | no | secret references made available inside the sandbox |
| `image` | no | override the sandbox image |
| `skills` | no | skills injected into the agent's workspace |

Use it when the agent should not run on the host — untrusted tasks, or a workspace that must be
discarded per task. The host runtime is otherwise the simpler choice.

## Next steps

<Cards>
<Card title="Agent Evaluation (concepts)" href="/documentation/evaluate-models/agent-eval" />
<Card title="Targets and Runners" href="/documentation/evaluate-models/agent-eval/targets-and-runners" />
<Card title="Writing Metrics" href="/documentation/evaluate-models/agent-eval/writing-metrics" />
</Cards>
Loading
Loading