diff --git a/docs/evaluator/agent-eval/fabric-runner.mdx b/docs/evaluator/agent-eval/fabric-runner.mdx new file mode 100644 index 0000000000..9660034fc7 --- /dev/null +++ b/docs/evaluator/agent-eval/fabric-runner.mdx @@ -0,0 +1,269 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +title: "Evaluate with a NeMo Fabric Harness" +description: "Run a coding agent through NeMo Fabric and score it with agent-eval — one config selects the harness (Codex, Claude, Hermes), and Fabric returns an ATIF trajectory a metric can grade on." +--- + +[NeMo Fabric](https://github.com/nvidia/nemo-fabric) runs an agent **harness** rather than a single +agent. Which harness runs is selected entirely by `config["harness"]["adapter_id"]`, so one runtime +covers several agent frontends without changing your evaluation. Fabric returns an **ATIF +trajectory** alongside the final answer, so a metric can score *how* the agent worked, not just what +it answered. + +The result and bundle are the same as the +[quickstart](/documentation/evaluate-models/agent-eval/quickstart) — only the runner changes. + +## Harnesses + +| `adapter_id` | Harness | Transport | +|---|---|---| +| `nvidia.fabric.codex` | Codex CLI | `cli` (subprocess) | +| `nvidia.fabric.claude` | Claude | `cli` | +| `nvidia.fabric.hermes` | Hermes SDK | `library` (in-process) | +| `nvidia.fabric.langchain.deepagents` | LangChain deepagents | `library` | + + + +This runner is **not** zero-dependency. It needs: + +- **The harness adapters**, from the `fabric` extra: + + ```bash + uv sync --frozen --package nemo-evaluator-sdk --extra fabric --inexact + ``` + +- **The `nemo-relay` gateway binary**, which captures the trajectory for out-of-process harnesses. + The pip package ships bindings only, so the daemon comes from a GitHub release asset: + + ```bash + script/dev-install-fabric.sh + ``` + +- **The harness's own CLI**, for `transport: cli` harnesses — `codex` on `PATH` and authenticated. + +The `fabric` extra installs the Codex, Claude, and Hermes adapters. The deepagents adapter is +deliberately excluded from it, because that adapter does not support the Relay observability +configuration Fabric streaming generates; install its harness separately if you need it. + + + +## The agent config + +One mapping describes the whole agent — harness, runtime, environment, and model. This is a working +Codex configuration: + +```python +config = { + "schema_version": "fabric.agent/v1alpha1", + "metadata": {"name": "eval-fabric"}, + "harness": { + "adapter_id": "nvidia.fabric.codex", + "resolution": "preinstalled", + "settings": {"sandbox": "workspace-write"}, + }, + "runtime": { + "mode": "oneshot", + "transport": "cli", + "input_schema": "text", + "output_schema": "message", + "timeout_seconds": 180, + }, + "environment": {"provider": "local"}, + "models": {"default": {"provider": "openai", "model": ""}}, + "telemetry": {"enabled": False}, +} +``` + +An `environment.workspace` set here is **overridden per task** — the runtime gives every task its own +fresh workspace under `work_root`, so setting one in the config has no effect. + +Across harnesses the shape differs mainly in `adapter_id`, `runtime.transport`, and any +harness-specific `harness.settings`. Codex runs as a subprocess (`transport: cli`) while the Hermes +SDK harness runs in-library (`transport: library`). For complete Codex-CLI and Hermes-SDK +configurations, see `examples/fabric_harness_runtimes.py` in the SDK. + + + +**The Codex adapter requires an explicit model provider.** It does not fall back to the Codex CLI's +own configured default, and starting without `models.default` fails the adapter lifecycle with +`codex_invalid_configuration`. + + + +Fold the complete configuration into this mapping. Fabric profile overlays are not used here. + +## Run it + +```python +from pathlib import Path + +from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator +from nemo_evaluator_sdk.agent_eval.runtimes.fabric.runtime import FabricAgentRuntime +from nemo_evaluator_sdk import StringCheckMetric +from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask + +tasks = [ + AgentEvalTask( + id="reply-ok", + intent="Reply with a fixed token.", + inputs={"instruction": "Reply with the word OK."}, + metrics=[ + StringCheckMetric( + operation="contains", + left_template="{{sample.output_text}}", + right_template="OK", + ) + ], + ) +] + +runtime = FabricAgentRuntime( + config=config, + work_root="fabric", + capture_trajectory=True, +) + +result = AgentEvaluator().run_sync( + tasks=tasks, + target=runtime, + config=AgentEvalRunConfig(work_dir=Path("out"), parallelism=1), +) +``` + +### Configuration + +| Field | Required | Notes | +|---|---|---| +| `config` | yes | the agent config mapping above | +| `model` | no | overrides `models.default.model` without editing the config | +| `base_dir` | no | base directory for relative paths in the config | +| `work_root` | no | where Fabric's per-run working directories are created | +| `timeout_s` | no | overall run timeout, default `600` | +| `capture_trajectory` | no | capture the ATIF trajectory as evidence, default `True` | +| `trajectory_extra` | no | extra fields merged into the captured trajectory | +| `runtime_name` | no | the name recorded on `runner_info`, default `fabric` | +| `skills` | no | skills injected into the agent's workspace | +| `task_hook` | no | a `FabricTaskRunHook` invoked around each task run | + +## Seed files into the workspace + +Every task runs in its own fresh workspace. Put files the agent should start from in the task's +`inputs["files"]`, and the runtime stages them before the harness runs: + +```python +task = AgentEvalTask( + id="read-seed", + intent="Read a seeded file and echo its contents.", + inputs={ + "instruction": "Read notes.txt in your workspace and reply with exactly its contents.", + "files": {"notes.txt": "MARKER-7f3a"}, + }, + metrics=[ + StringCheckMetric( + operation="contains", + left_template="{{sample.output_text}}", + right_template="MARKER-7f3a", + ) + ], +) +``` + +With no `files` key this is a no-op, so tasks that need no starting state cost nothing. + +## Read the results + +The answer is on the trial, and scores come back under the metric's type: + +```python +trial = result.trials[0] +print(trial.status) # AgentEvalTrialStatus.COMPLETED +assert trial.output is not None +print(trial.output.output_text) # MARKER-7f3a + +for score in result.summary.scores.scores: + print(score.name) +# string-check.string-check +# string-check.string-check.pass@1 +``` + +### Evidence + +This is where Fabric differs from the other runners. Each trial carries several evidence streams, so +a metric can score *how* the agent worked rather than only its final answer: + +| Key | Kind | What it is | +|---|---|---| +| `trace` | `trace` (format `atif`) | the ATIF trajectory — tool calls, reasoning, observations | +| `workspace` | `filesystem` | the workspace's final file tree, after the agent finished | +| `result` | `json` | the harness's structured result | +| `stdout` | `log` | harness stdout | +| `relay_atif` / `relay_atof` | `atif` / `atof` | the raw Relay artifacts the trajectory is promoted from | +| `relay_config` | `telemetry_config` | the Relay configuration used for the run | + +```python +assert trial.evidence is not None +trace = trial.evidence.descriptors["trace"] +# trace.format == "atif"; trace.ref is a path to the trajectory JSON, which has a "steps" key + +workspace = trial.evidence.descriptors["workspace"] +# workspace.ref is a directory — what the agent left behind, including any seeded files +``` + +The `workspace` tree is what lets a metric grade artifacts the agent produced on disk, and `trace` +is what lets it grade the process. See +[Writing Metrics](/documentation/evaluate-models/agent-eval/writing-metrics) for reading evidence, +and [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component) for combining +several signals into one reported score. + +## Submit as a platform job + +`FabricRunnerTarget` carries the same config into a durable job: + +```python +from nemo_evaluator.jobs.agent_spec import FabricRunnerTarget + +target = FabricRunnerTarget(config=config, model="/", capture_trajectory=True) +``` + +| Field | Required | Notes | +|---|---|---| +| `config` | yes | the same agent config, as a JSON-shaped mapping | +| `model` | no | a `provider/model` slug applied as the config's default model | +| `timeout_s` | no | per-task timeout for the Fabric run | +| `capture_trajectory` | no | capture the ATIF trajectory and attach it to trial evidence | + +Submit it with `nemo evaluator agent-evaluate submit`, or as the `target` of an +`AgentEvalInputSpec`. See +[Targets and Runners](/documentation/evaluate-models/agent-eval/targets-and-runners). + +## Running in a sandbox + +`FabricContainerRuntime` takes the same config and runs it inside a sandbox rather than on the host: + +```python +from nemo_evaluator_sdk.agent_eval.runtimes.fabric.container_runtime import FabricContainerRuntime +from nemo_evaluator_sdk.agent_eval.runtimes.sandbox.providers.docker import DockerSandboxProvider + +runtime = FabricContainerRuntime(config=config, provider=DockerSandboxProvider()) +``` + +| Field | Required | Notes | +|---|---|---| +| `config` | yes | the same agent config, or a `FabricConfig` | +| `provider` | yes | a `SandboxProvider` — `DockerSandboxProvider` or `DockerComposeSandboxProvider` | +| `secrets` | no | secret references made available inside the sandbox | +| `image` | no | override the sandbox image | +| `skills` | no | skills injected into the agent's workspace | + +Use it when the agent should not run on the host — untrusted tasks, or a workspace that must be +discarded per task. The host runtime is otherwise the simpler choice. + +## Next steps + + + + + + diff --git a/docs/evaluator/agent-eval/gym-runner.mdx b/docs/evaluator/agent-eval/gym-runner.mdx new file mode 100644 index 0000000000..68539cfad1 --- /dev/null +++ b/docs/evaluator/agent-eval/gym-runner.mdx @@ -0,0 +1,321 @@ +--- +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +title: "Evaluate a NeMo Gym Environment" +description: "Run an existing NeMo Gym environment through NeMo Evaluator's Gym runner — Gym collects rollouts against its own resources-server and agent, and the SDK adapts them into trials scored on Gym's reward." +--- + +[NeMo Gym](https://github.com/NVIDIA-NeMo/Gym) is an environment framework for agentic rollouts: a +**resources-server** provides the environment, an **agent** acts in it, and each rollout carries a +**reward**. If you already have a Gym environment, the **Gym runner** runs it and scores its reward +through agent-eval — the same [`AgentEvaluator`](/documentation/evaluate-models/agent-eval) and the +same result and bundle as the [quickstart](/documentation/evaluate-models/agent-eval/quickstart). +Only the runner changes. + +Gym owns execution **and** scoring here. The runner shells out to the `gym` CLI and adapts the +rollout bundle into trials; `GymRewardMetric` surfaces Gym's per-attempt reward. + + + +Like the Harbor runner, this one is **not** zero-dependency — it shells out to the `gym` CLI: + +- **NeMo Gym**, installed into its own `uv` environment (below) +- **The target environment's own dependencies** — each resources-server ships its own + `requirements.txt` (the `mcqa` example needs `tiktoken`) +- **Model credentials** for the collector, in an `env.yaml` (below) + +```bash +uv venv ~/gym-env --python 3.12 +uv pip install --python ~/gym-env/bin/python nemo-gym tiktoken +export PATH="$HOME/gym-env/bin:$PATH" +``` + +Install Gym into **its own environment** and put that environment's `bin` on `PATH`. Gym imports Ray +at module load, and `nemo-platform` excludes Ray by constraint, so the two generally cannot share a +virtualenv. The runner resolves `gym` from `PATH` only — there is deliberately no setting pointing at +a checkout or another venv, because this config becomes a serialized job spec when Gym runs as a +platform job, and a local path means nothing on the other side of that boundary. In a job image, the +image owns `PATH` and this resolves normally. + + + +## Credentials + +Gym's collector calls your model endpoint directly. It reads the credentials from an `env.yaml` in +the directory you run from — **this SDK never reads or handles that file**: + +```yaml +policy_base_url: https:///v1 +policy_api_key: +policy_model_name: +``` + +Keep it out of version control. Gym searches the working directory first, then its install root. + +## The dataset + +A Gym dataset is a **jsonl file**, one row per case. Environments ship their example data inside the +`nemo-gym` wheel, so the bundled `mcqa` benchmark needs no checkout: + +``` +/resources_servers/mcqa/data/example.jsonl +``` + +`discover_gym_tasks` turns that file into tasks — one per **distinct** row: + +```python +from nemo_evaluator_sdk.agent_eval.runtimes.gym import GymRewardMetric, discover_gym_tasks + +tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()]) +``` + +Task identity is the row's **content hash**, which has two consequences worth knowing before you +build a dataset: + +- Duplicate rows collapse into a single task, and the runner warns. Duplicates usually mean a data + problem. +- Repeating a row is **not** how you ask for repeated attempts. Use `num_repeats` — attempts are a + run-level concern, not a dataset one. + +## Run it + +```python +import asyncio + +from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator +from nemo_evaluator_sdk.agent_eval.runtimes.gym import ( + GymAgentTaskRunner, + GymRewardMetric, + GymRuntimeConfig, + discover_gym_tasks, +) + +tasks = discover_gym_tasks("path/to/example.jsonl", metrics=[GymRewardMetric()]) + +runner = GymAgentTaskRunner( + config=GymRuntimeConfig( + agent="simple_agent", + agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml", + resources_server="mcqa", + num_repeats=2, + ) +) + +result = asyncio.run(AgentEvaluator().run(tasks=tasks, target=runner)) +print(result.summary) +``` + +Run it from the directory holding `env.yaml`. + +The mapping is: + +- one Gym dataset → one run +- each distinct row → one task +- each attempt → one trial + +So `num_repeats=2` over a 5-row dataset yields 5 tasks and 10 trials. + +### Configuration + +| Field | Required | Notes | +|---|---|---| +| `agent` | yes | agent name to collect rollouts with, e.g. `simple_agent` | +| `agent_config` | yes | agent config passed to `gym env start`, resolved relative to the Gym install, e.g. `responses_api_agents/simple_agent/configs/simple_agent.yaml` | +| `resources_server` | yes | resources-server (environment) name, e.g. `mcqa` | +| `model_type` | no | `inference_provider` for OpenAI-compatible **chat** endpoints; `openai_model` uses the OpenAI **Responses API** and fails against chat-only endpoints | +| `bind_resources_server` | no | auto-bind the agent's `resources_server.name` via a Hydra override, for a composable agent whose config leaves it unset (`simple_agent`). Set `False` for a self-contained agent that already binds its own | +| `num_repeats` | no | attempts per row; each attempt becomes one trial | +| `concurrency` | no | concurrent rollouts during collection — tune to your model endpoint's limits | +| `hydra_params` | no | parameters merged into Gym's Hydra config, e.g. `{"model": {"temperature": 0.7}}` | +| `env_vars` | no | environment variables set on the `gym` invocation | +| `reward_key` | no | key read from each rollout record (default `reward`) | +| `startup_timeout_s` | no | max wait for `gym env start` readiness | +| `collection_timeout_s` | no | max wait for collection; `None` is unbounded | +| `shutdown_grace_s` | no | grace period for the Gym subprocess group to exit on `SIGTERM`, letting Ray shut down cleanly, before escalating to `SIGKILL` | + +Anything `GymRuntimeConfig` does not expose can go through `hydra_params`, which is flattened to +Hydra's override grammar and applied to `gym env start`. For the full set of knobs, see the +[NeMo Gym documentation](https://github.com/NVIDIA-NeMo/Gym). + +## Read the results + +Gym's reward arrives as `gym_reward.reward`, and Gym's own aggregates are imported alongside the +SDK's under a `runner.gym.*` prefix — the prefix is what tells you which side computed a number: + +```python +for score in result.summary.scores.scores: + print(score.name) +# gym_reward.reward +# gym_reward.reward.pass@1 +# runner.gym.pass@1/accuracy +# runner.gym.input_tokens +``` + +Gym reports accuracy on a **0–100** scale where the SDK uses **0–1**, so `runner.gym.pass@1/accuracy` +of `50.0` corresponds to a `gym_reward.reward` mean of `0.5`. Trials, scores, and the run bundle are +otherwise read exactly as in +[Reading Results](/documentation/evaluate-models/agent-eval/reading-results). + +## Output directories + +Each run writes to a fresh temporary directory by default. To choose one, set `work_dir` on the run +config: + +```python +from pathlib import Path + +from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig + +result = asyncio.run( + AgentEvaluator().run( + tasks=tasks, target=runner, config=AgentEvalRunConfig(work_dir=Path("gym-run-1")) + ) +) +``` + +Give every run its own. The runner **refuses** to reuse a directory that already holds Gym rollout +output, raising `FileExistsError`: Gym appends to its failures sidecar, so reusing one would mix two +runs together, and the runner raises rather than clearing a previous run's results. + +Gym's own artifacts land in a `gym_run/` subdirectory — `rollouts.jsonl`, +`rollouts_failures.jsonl`, `rollouts_aggregate_metrics.json`, and the materialized +`gym_input.jsonl` handed to collection. + +## How it runs Gym + +The runner uses Gym's **two-step** flow, which reads a dataset file directly — no split-driven data +preparation and no HuggingFace downloads: + +1. `gym env start …` brings up the resources-server, agent, and model servers. +2. `gym eval run --no-serve --input …` collects rollouts against them. + +The dataset handed to step 2 is **not** your source file. The runner materializes a normalized one +into the run's work directory, one row per requested task, with `_ng_task_index` stamped explicitly. +Gym honors a caller-supplied `_ng_task_index` and echoes it back on every rollout record, so rollouts +join back to tasks through a map the runner owns rather than a guess about Gym's row ordering. That +is also what lets you run a **subset** of tasks and roll out only that subset. + +### Logs + +Gym's subprocess output is streamed to files in the run's work directory — `gym_env.log` for startup, +and `gym_eval.stdout.log` / `gym_eval.stderr.log` for collection — and mirrored to the +`nemo_evaluator_sdk.agent_eval.runtimes.gym` logger at `DEBUG`. Startup and collection failures name +the relevant file and inline its last lines. To watch Gym's output in your own terminal: + +```python +import logging + +logging.getLogger("nemo_evaluator_sdk.agent_eval.runtimes.gym").setLevel(logging.DEBUG) +``` + +## Submit as a platform job + +A Gym runner can be submitted as a durable platform job **from the live runner object**, rather than +described a second time as a job spec — the configuration you validated locally is the configuration +that runs. + +`submit` takes a stored taskset, so the Gym rows have to be stored first. A Gym taskset is not an +ordinary one: the job rebuilds the Gym dataset from each task, so every task must carry the row that +`discover_gym_tasks` split across `inputs['gym_row']` and `metadata['gym_row_extras']`. Build the +tasks with `discover_gym_tasks` and store both halves: + +```python +from nemo_evaluator.api.fields import MetadataItem, MetricInline +from nemo_evaluator.api.schemas import ( + EvaluatorTaskDefinition, + TaskInput, + TaskInputs, + TaskRef, + TasksetInput, + TasksetRef, +) +from nemo_evaluator.shared.metric_bundles.bundles import bundle_metric +from nemo_evaluator.shared.metric_bundles.cloudpickle import CloudpickleMetricBundlePackager +from nemo_evaluator_sdk.agent_eval.runtimes.gym import ( + GymAgentTaskRunner, + GymRewardMetric, + GymRuntimeConfig, + discover_gym_tasks, +) +from nemo_platform import NeMoPlatform + +client = NeMoPlatform(base_url="http://localhost:8080", workspace="default") + +runner = GymAgentTaskRunner( + config=GymRuntimeConfig( + agent="simple_agent", + agent_config="responses_api_agents/simple_agent/configs/simple_agent.yaml", + resources_server="mcqa", + ) +) + +tasks = discover_gym_tasks("path/to/example.jsonl") + +# GymRewardMetric is not a built-in metric type, so it needs the cloudpickle packager. +reward = MetricInline.model_validate( + bundle_metric(GymRewardMetric(), CloudpickleMetricBundlePackager()).model_dump(mode="json") +) + +names = [] +for index, task in enumerate(tasks): + name = f"mcqa-{index}" + names.append(name) + client.evaluator.tasks.create( + name, + task=TaskInput( + spec=EvaluatorTaskDefinition( + kind="evaluator", + intent=task.intent, + inputs=TaskInputs(**task.inputs), + metrics=[reward], + ), + metadata=[MetadataItem(key=key, value=value) for key, value in task.metadata.items()], + ), + ) + +client.evaluator.tasksets.create("mcqa-suite", taskset=TasksetInput(tasks=[TaskRef(n) for n in names])) + +job = client.evaluator.submit(tasks=TasksetRef("mcqa-suite"), target=runner) +job.wait_until_done() +``` + +Two things that bite here: + +- **Do not name the task after `task.id`.** It is a 64-character hex digest, already past the + 63-character cap on entity names, and it may begin with a digit where a name must start with a + letter. Derive a name, as above; the digest stays the task's own `id`. +- **`gym_row` rides on `inputs` and `gym_row_extras` on the task's `metadata`** — the field beside + `spec`, not inside it. A task missing either is rejected job-side with `task '' is missing + inputs['gym_row'] and/or metadata['gym_row_extras']`. + +A value with no JSON form — a callable in `hydra_params`, say — is refused with +`UnsubmittableRunnerError` at submit time, rather than failing inside the transport with an error +that names neither the runner nor the field. + +The returned handle is an `AgentEvaluatorJobResource`. Unlike a dataset-driven job it has no +`get_result()` or `download_artifacts()`, because an agent evaluation publishes agent-eval results +and a summary rather than row scores. Read the scores through +`client.evaluator.agent_eval_results`. + +Gym jobs run on their own `nmp-gym-tasks` container image rather than the shared CPU task image, +because Gym requires Ray. No configuration is needed — the target selects it. + +## Caveats + +- **Per-environment dependencies are heterogeneous.** `mcqa` needs only `tiktoken`; other Gym + environments pull in `torch`, COMET, a GPU, or Docker. Providing a Gym runtime with those installed + is the caller's responsibility. +- **`--no-serve --input` bypasses Gym's data-prep** — prompt templating and dataset materialization. + Rows that are already complete, like the bundled `example.jsonl`, are faithful; an environment + whose rows need templating would need that step run first. +- **Service-side execution** — Docker or Kubernetes, Ray provisioning — is out of scope for this SDK + path. That is the evaluator plugin's concern. + +## Next steps + + + + + + diff --git a/docs/evaluator/agent-eval/targets-and-runners.mdx b/docs/evaluator/agent-eval/targets-and-runners.mdx index 90f99998cc..19351d001b 100644 --- a/docs/evaluator/agent-eval/targets-and-runners.mdx +++ b/docs/evaluator/agent-eval/targets-and-runners.mdx @@ -3,7 +3,7 @@ # SPDX-License-Identifier: Apache-2.0 title: "Targets and Runners" -description: "Reference for what an agent-eval run can point at — a Model, a deployed Agent over HTTP, or an AgentTaskRunner (a callable, Harbor, or your own) — with each target's key fields and when to use it." +description: "Reference for what an agent-eval run can point at — a Model, a deployed Agent over HTTP, or an AgentTaskRunner (a callable, Harbor, Gym, or your own) — with each target's key fields and when to use it." --- `AgentEvaluator().run(target=...)` accepts one of three kinds of target. Whatever you pick, it produces @@ -19,6 +19,9 @@ description: "Reference for what an agent-eval run can point at — a Model, a d | `NemoAgentToolkitAgent` | a NeMo Agent Toolkit endpoint | a running NAT workflow | [Evaluate a Deployed Agent](/documentation/evaluate-models/agent-eval/evaluate-deployed-agent) | | `CallableAgentTaskRunner` | an in-process async function | none | [Quickstart](/documentation/evaluate-models/agent-eval/quickstart) | | `HarborAgentTaskRunner` | a Harbor task suite | `harbor` + Docker | [Harbor Task Suite](/documentation/evaluate-models/agent-eval/harbor-runner) | +| `GymAgentTaskRunner` | a NeMo Gym environment + agent | the `gym` CLI on `PATH` | [NeMo Gym Environment](/documentation/evaluate-models/agent-eval/gym-runner) | +| `FabricAgentRuntime` | a NeMo Fabric harness (Codex, Claude, Hermes, deepagents) | the `fabric` extra | [NeMo Fabric Harness](/documentation/evaluate-models/agent-eval/fabric-runner) | +| `FabricContainerRuntime` | the same harnesses, run inside a sandbox | the `fabric` extra + a sandbox provider | [NeMo Fabric Harness](/documentation/evaluate-models/agent-eval/fabric-runner) | | *your* `AgentTaskRunner` | anything that turns tasks into trials | up to you | *(this page)* | The union is `AgentEvalTarget = Model | Agent | AgentTaskRunner`, where `Agent = GenericAgent | @@ -36,6 +39,8 @@ bare model do before you wrap it in an agent?). The evaluator prompts it with ea | `name` | yes | model identifier, stamped on trials | | `format` | no | `ModelFormat.NVIDIA_NIM` (default), `ModelFormat.OPEN_AI`, or `ModelFormat.LLAMA_STACK` — serialized as `nim` / `openai` / `llama_stack` | | `api_key_secret` | no | credential reference — `workspace/secret_name` or `secret_name` | +| `default_headers` | no | non-auth headers applied to every request; authentication goes through `api_key_secret` | +| `host_url` | no | direct NIM endpoint (`http://host:port`), populated when the target resolves from a `ModelRef` | ```python from nemo_evaluator_sdk.enums import ModelFormat @@ -86,23 +91,27 @@ workflow's URL. | `nat` | no | `NatAgentConfig` — endpoint / query-param / response-path / aggregation overrides; defaults target `/generate/full` and `concat` token-delta frames into the full response | | `api_key_secret` | no | credential reference (env var locally, platform secret for a job); its value is sent as a bearer token | -## `AgentTaskRunner` (callable, Harbor, or your own) +## `AgentTaskRunner` (callable, Harbor, Gym, or your own) -The most general target: anything implementing the one-method protocol. +The most general target: anything implementing the two-method protocol. Both members are +required — a runner missing either is rejected with `NotImplementedError: unsupported +agent-eval target type`. ```python from collections.abc import Sequence from nemo_evaluator_sdk.agent_eval.tasks import AgentEvalRunConfig, AgentEvalTask -from nemo_evaluator_sdk.agent_eval.trials import AgentEvalTrial +from nemo_evaluator_sdk.agent_eval.trials import AgentEvalTrial, RunnerInfo class AgentTaskRunner: async def run_tasks( self, tasks: Sequence[AgentEvalTask], config: AgentEvalRunConfig | None = None ) -> Sequence[AgentEvalTrial]: ... + + def runner_info(self) -> RunnerInfo: ... ``` -The SDK ships two runners you'll usually reach for first: +The SDK ships three runners you'll usually reach for first: - **`CallableAgentTaskRunner`** wraps an `async def agent(task) -> str | AgentOutput | TrialDraft`. The smallest possible target — no Docker, no HTTP. See the @@ -110,12 +119,70 @@ The SDK ships two runners you'll usually reach for first: a trajectory or other evidence (see [Score by Component](/documentation/evaluate-models/agent-eval/score-by-component)). - **`HarborAgentTaskRunner`** runs a [Harbor](https://www.harborframework.com) task suite in Docker and scores its verifier reward. See [Harbor Task Suite](/documentation/evaluate-models/agent-eval/harbor-runner). +- **`GymAgentTaskRunner`** runs a [NeMo Gym](https://github.com/NVIDIA-NeMo/Gym) environment and agent, + and scores each rollout's reward. See [NeMo Gym Environment](/documentation/evaluate-models/agent-eval/gym-runner). + +### `GymAgentTaskRunner` + +Runs an existing [NeMo Gym](https://github.com/NVIDIA-NeMo/Gym) environment against your tasks and +adapts its rollouts into trials, scoring each rollout's reward. `GymRuntimeConfig` requires `agent`, +`agent_config`, and `resources_server`; `discover_gym_tasks` builds the tasks from a Gym jsonl +dataset. + +Gym is **not** a dependency of this SDK — it imports Ray at module load, which nemo-platform excludes +by constraint. Install it into its own environment and put its `bin` on `PATH`. + +It can also be submitted as a platform job from the live runner object, via +`client.evaluator.submit(tasks=..., target=runner)`, rather than described again as a spec. + +Full setup, configuration reference, and caveats: +[Evaluate a NeMo Gym Environment](/documentation/evaluate-models/agent-eval/gym-runner). + +### NeMo Fabric runtimes + +[NeMo Fabric](https://github.com/nvidia/nemo-fabric) drives an agent *harness* rather than a single +agent. Which harness runs is chosen entirely by `config["harness"]["adapter_id"]`, so one runtime +covers several agent frontends: + +| `adapter_id` | Harness | +|---|---| +| `nvidia.fabric.codex` | Codex CLI (`transport="cli"`) | +| `nvidia.fabric.claude` | Claude | +| `nvidia.fabric.hermes` | Hermes SDK (`transport="library"`) | +| `nvidia.fabric.langchain.deepagents` | LangChain deepagents — **not** installed by this SDK's `fabric` extra (see below) | + +Two runtimes share that config: + +- **`FabricAgentRuntime`** runs the harness directly, capturing an ATIF trajectory. +- **`FabricContainerRuntime`** runs the same configuration inside a sandbox, and additionally + accepts `skills` and `secrets`. + +Both need the `fabric` extra, which pulls the Codex, Claude, and Hermes adapters: + +```bash +uv sync --frozen --package nemo-evaluator-sdk --extra fabric --inexact +``` + +The deepagents adapter is deliberately excluded from that extra — it does not support the Relay +observability configuration Fabric streaming generates — so `nvidia.fabric.langchain.deepagents` +needs its harness installed separately. + +Full setup, the agent-config shape, and the trajectory evidence: +[Evaluate with a NeMo Fabric Harness](/documentation/evaluate-models/agent-eval/fabric-runner). + +### Writing your own Write your own when your agent doesn't fit those — a bespoke harness, a queue, a replay of stored runs. -Return one `AgentEvalTrial` per task; the evaluator scores them exactly like any other target: +Return one `AgentEvalTrial` per task and identify the runner with `runner_info`; the evaluator scores +the trials exactly like any other target: ```python -from nemo_evaluator_sdk.agent_eval.trials import AgentEvalTrial, AgentEvalTrialStatus, AgentOutput +from nemo_evaluator_sdk.agent_eval.trials import ( + AgentEvalTrial, + AgentEvalTrialStatus, + AgentOutput, + RunnerInfo, +) class EchoRunner: async def run_tasks(self, tasks, config=None): @@ -128,8 +195,16 @@ class EchoRunner: ) for task in tasks ] + + def runner_info(self) -> RunnerInfo: + return RunnerInfo(name="echo") ``` +`runner_info` is what records the producer of a run: the result carries it on +`AgentEvalResult.metadata.target`, so a stored run can be understood after the fact. Return a stable +short `name` (`"gym"`, `"harbor"`) rather than a class name, and keep secrets out of `config` — it is +persisted with the run bundle. + ## Choosing a target - Just trying the flow, or you already have the agent in Python → **`CallableAgentTaskRunner`**. @@ -137,6 +212,7 @@ class EchoRunner: (a NAT workflow). - You want a model baseline, no agent → **`Model`**. - You have Harbor task datasets → **`HarborAgentTaskRunner`**. +- You have a NeMo Gym environment → **`GymAgentTaskRunner`**. - None of the above fits → implement **`AgentTaskRunner`**. ## Related diff --git a/docs/fern/versions/latest.yml b/docs/fern/versions/latest.yml index 39bc7627c5..21acc2fa09 100644 --- a/docs/fern/versions/latest.yml +++ b/docs/fern/versions/latest.yml @@ -393,6 +393,12 @@ navigation: - page: Evaluate a Harbor Task Suite slug: harbor-runner path: ../../evaluator/agent-eval/harbor-runner.mdx + - page: Evaluate a NeMo Gym Environment + slug: gym-runner + path: ../../evaluator/agent-eval/gym-runner.mdx + - page: Evaluate with a NeMo Fabric Harness + slug: fabric-runner + path: ../../evaluator/agent-eval/fabric-runner.mdx - page: Score by Component slug: score-by-component path: ../../evaluator/agent-eval/score-by-component.mdx diff --git a/packages/nemo_evaluator_sdk/examples/gym/README.md b/packages/nemo_evaluator_sdk/examples/gym/README.md index b2b4c0c74d..5235c7b990 100644 --- a/packages/nemo_evaluator_sdk/examples/gym/README.md +++ b/packages/nemo_evaluator_sdk/examples/gym/README.md @@ -9,11 +9,13 @@ Mapping: one Gym dataset → one run; each distinct row → one `AgentEvalTask` ## Prerequisites -**1. NeMo Gym installed in the same environment as the SDK**, plus the target environment's own dependencies. Environments ship in the `nemo-gym` wheel — `resources_servers` and friends install beside `nemo_gym`, configs and `data/example.jsonl` included — so no checkout is needed: +**1. NeMo Gym installed in its own environment, with that environment's `bin` on `PATH`**, plus the target environment's own dependencies. Gym imports Ray at module load and nemo-platform excludes Ray by constraint, so the two generally cannot share a virtualenv; the runner resolves `gym` from `PATH` only. Environments ship in the `nemo-gym` wheel — `resources_servers` and friends install beside `nemo_gym`, configs and `data/example.jsonl` included — so no checkout is needed: ```bash -pip install nemo-gym -pip install tiktoken # mcqa's own dependency; each resources_server ships a requirements.txt +uv venv ~/gym-env --python 3.12 +# tiktoken is mcqa's own dependency; each resources_server ships a requirements.txt +uv pip install --python ~/gym-env/bin/python nemo-gym tiktoken +export PATH="$HOME/gym-env/bin:$PATH" ``` The runner shells out to whatever `gym` is on PATH. There is deliberately no setting for a checkout, another venv, or a search root: this config becomes a serialized job spec when Gym runs as a platform job, and a local path means nothing on the other side of that boundary. @@ -115,5 +117,6 @@ logging.getLogger("nemo_evaluator_sdk.agent_eval.runtimes.gym").setLevel(logging - [`../harbor/`](../harbor/) — the same `AgentEvaluator` seam driven by the Harbor runtime. - [`../fabric_harness_runtimes.py`](../fabric_harness_runtimes.py) — the Fabric runtime equivalent. -- [`gym_runtime.py`](../../src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py) — `GymRuntimeConfig` field reference plus the attribution and teardown rationale. +- [`gym/config.py`](../../src/nemo_evaluator_sdk/agent_eval/runtimes/gym/config.py) — `GymRuntimeConfig` field reference. +- [`gym/runtime.py`](../../src/nemo_evaluator_sdk/agent_eval/runtimes/gym/runtime.py) — the attribution and teardown rationale. - [NeMo Gym documentation](https://github.com/NVIDIA-NeMo/Gym) — authoring environments, agents, and datasets. diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym/results.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym/results.py index 1f14e6f45e..b1bd5740ea 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym/results.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym/results.py @@ -304,7 +304,7 @@ def _ensure_fresh_output(rollouts_path: Path) -> None: names = ", ".join(path.name for path in preexisting) raise FileExistsError( f"{rollouts_path.parent} already holds Gym rollout output ({names}); give each run a fresh " - "output_dir (the AgentEvaluator convention). Gym appends to the failures sidecar, so reusing a " + "work_dir (AgentEvalRunConfig.work_dir). Gym appends to the failures sidecar, so reusing a " "directory would mix runs and could obscure a prior run's results." ) diff --git a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym/results.py b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym/results.py index 6c5c9c09ff..5a42b3b09b 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym/results.py +++ b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym/results.py @@ -304,7 +304,7 @@ def _ensure_fresh_output(rollouts_path: Path) -> None: names = ", ".join(path.name for path in preexisting) raise FileExistsError( f"{rollouts_path.parent} already holds Gym rollout output ({names}); give each run a fresh " - "output_dir (the AgentEvaluator convention). Gym appends to the failures sidecar, so reusing a " + "work_dir (AgentEvalRunConfig.work_dir). Gym appends to the failures sidecar, so reusing a " "directory would mix runs and could obscure a prior run's results." )