refactor(evaluator)!: resolve the Gym CLI from PATH instead of a configured checkout

packages/nemo_evaluator_sdk/examples/gym/README.md

@@ -6,35 +6,38 @@ Mapping: one Gym dataset → one run; each distinct row → one `AgentEvalTask`
 
 ## Prerequisites
 
-A working **NeMo Gym checkout** is required — Gym resolves its environments from the repo, not from a package install. From the Gym checkout:
+**1. NeMo Gym installed in the same environment as the SDK**, plus the target environment's own dependencies. Environments ship in the `nemo-gym` wheel — `resources_servers` and friends install beside `nemo_gym`, configs and `data/example.jsonl` included — so no checkout is needed:
 
 ```bash
-# 1. Gym venv + framework (Gym pins uv >= 0.9.30; the workspace floor must satisfy that)
-uv venv --python 3.12 .venv
-uv pip install --no-config --python .venv/bin/python -e ".[dev]" "ray[default]>=2.55.1"
+pip install nemo-gym
+pip install tiktoken        # mcqa's own dependency; each resources_server ships a requirements.txt
+```
+
+The runner shells out to whatever `gym` is on PATH. There is deliberately no setting for a checkout, another venv, or a search root: this config becomes a serialized job spec when Gym runs as a platform job, and a local path means nothing on the other side of that boundary.
 
-# 2. The target env's own deps (each resources_server ships a requirements.txt)
-uv pip install --no-config --python .venv/bin/python tiktoken        # mcqa needs tiktoken
+**2. Model credentials for the collector** — a gitignored `env.yaml` in the directory you run from:
 
-# 3. Model credentials for the collector — a gitignored env.yaml at the Gym repo root:
+```bash
 cat > env.yaml <<'YAML'
 policy_base_url: https://<your-openai-compatible-endpoint>/v1
 policy_api_key: <key>
 policy_model_name: <model, e.g. nvidia/meta/llama-3.3-70b-instruct>
 YAML
 ```
 
-> `env.yaml` is gitignored by the Gym repo — the credentials stay local.
+> Gym searches the working directory, then its install root. This SDK never reads that file — the `gym` subprocess does.
 
 ## Run
 
-From the nemo-platform repo root (any Python with `nemo_evaluator_sdk` importable — the runner shells out to Gym's own venv):
+From the directory holding `env.yaml`:
 
 ```bash
-uv run python -m packages.nemo_evaluator_sdk.examples.gym.run_gym_eval --gym-root /path/to/Gym
+uv run python -m packages.nemo_evaluator_sdk.examples.gym.run_gym_eval
 ```
 
-Useful flags: `--resources-server`, `--agent`, `--model-type` (`inference_provider` for OpenAI-compatible **chat** endpoints; `openai_model` uses the OpenAI **Responses API** and 500s against chat-only endpoints), `--num-repeats`, `--output-dir`.
+Working from a Gym checkout also makes its components take precedence over the packaged ones, which is how you reach an environment whose data the wheel does not carry (bulk train/validation splits are not git-tracked, so they are not in it).
+
+Useful flags: `--resources-server`, `--agent`, `--model-type` (`inference_provider` for OpenAI-compatible **chat** endpoints; `openai_model` uses the OpenAI **Responses API** and 500s against chat-only endpoints), `--num-repeats`, `--dataset`, `--output-dir`.
 
 For the full set of knobs the underlying `gym env start` / `gym eval run` commands accept, see the [NeMo Gym documentation](https://github.com/NVIDIA-NeMo/Gym). Anything `GymRuntimeConfig` does not expose as a field can be passed through with its `env_overrides` escape hatch (Hydra `+key=value` overrides applied to `gym env start`).
 
@@ -60,7 +63,7 @@ No bundle is checked in; the run above produces one. Give it a stable `--output-
 
 ```bash
 uv run python -m packages.nemo_evaluator_sdk.examples.gym.run_gym_eval \
-    --gym-root /path/to/Gym --output-dir /tmp/gym-eval
+    --output-dir /tmp/gym-eval
 uv run python -m packages.nemo_evaluator_sdk.examples.gym.inspect_results --bundle /tmp/gym-eval
 ```
 
@@ -73,7 +76,7 @@ The runner uses Gym's **two-step** flow, which reads a dataset file directly (no
 1. `gym env start …` — brings up the resources-server + agent + model servers.
 2. `gym eval run --no-serve --input <dataset> …` — collects rollouts against them.
 
-It does **not** import `nemo_gym` and does **not** handle secrets — it invokes the `gym` executable in your checkout, and Gym reads credentials from that checkout's `env.yaml`.
+It does **not** import `nemo_gym` and does **not** handle secrets — it invokes whichever `gym` executable is on `PATH`, resolved once to an absolute path so the subprocesses cannot pick up a different one, and Gym reads credentials from the `env.yaml` in the directory you run from.
 
 The dataset handed to step 2 is not your source file. The runner **materializes** a normalized one into the run's work directory: one row per requested task, with `_ng_task_index` stamped explicitly. Gym honors a caller-supplied `_ng_task_index` (it only assigns one when a row lacks it) and echoes it back on every rollout record, so rollouts join back to tasks through a map the runner owns rather than a guess about Gym's internal row ordering. It also means running a *subset* of tasks only rolls out that subset.
 

packages/nemo_evaluator_sdk/examples/gym/inspect_results.py

@@ -17,7 +17,7 @@
 same path. From the repository root::
 
     uv run python -m packages.nemo_evaluator_sdk.examples.gym.run_gym_eval \\
-        --gym-root /path/to/Gym --output-dir /tmp/gym-eval
+        --output-dir /tmp/gym-eval
     uv run python -m packages.nemo_evaluator_sdk.examples.gym.inspect_results --bundle /tmp/gym-eval
 
 Any agent-eval bundle works, not just a Gym one: pass ``--metric-type``/``--output-name`` for the

packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py

@@ -16,15 +16,16 @@
 
 Prerequisites (see README.md):
 
-* A **NeMo Gym checkout** whose venv has the target env's deps installed — each env ships its own
-  ``requirements.txt`` (mcqa needs ``tiktoken``; Gym itself needs ``ray`` and ``uv >= 0.9.30``).
-* A **gitignored ``<gym_root>/env.yaml``** holding the model credentials the collector calls
-  (``policy_base_url`` / ``policy_api_key`` / ``policy_model_name``). This SDK never handles secrets —
-  the ``gym`` subprocess reads them from that file.
+* **NeMo Gym installed in this environment** (``pip install nemo-gym``), plus the target env's own
+  deps — each env ships its own ``requirements.txt`` (mcqa needs ``tiktoken``). Environments ship in
+  the wheel, so no checkout is needed for the bundled example data.
+* A **gitignored ``env.yaml``** holding the model credentials the collector calls
+  (``policy_base_url`` / ``policy_api_key`` / ``policy_model_name``) in the directory you run from.
+  This SDK never handles secrets — the ``gym`` subprocess reads them from that file.
 
-Run from the repository root::
+Run from the directory holding ``env.yaml``::
 
-    uv run python -m packages.nemo_evaluator_sdk.examples.gym.run_gym_eval --gym-root /path/to/Gym
+    uv run python -m packages.nemo_evaluator_sdk.examples.gym.run_gym_eval
 
 Pass ``--output-dir`` to write the bundle somewhere stable, then read it with ``inspect_results.py``.
 """
@@ -47,14 +48,12 @@
 
 def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
     parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
-    parser.add_argument(
-        "--gym-root", required=True, type=Path, help="NeMo Gym checkout; its venv + env.yaml provide deps/creds."
-    )
     parser.add_argument(
         "--dataset",
         type=Path,
         default=None,
-        help="Dataset jsonl (default: <gym-root>/resources_servers/<resources-server>/data/example.jsonl).",
+        help="Dataset jsonl (default: the installed package's "
+        "resources_servers/<resources-server>/data/example.jsonl).",
     )
     parser.add_argument("--resources-server", default="mcqa", help="Gym resources-server (environment) name.")
     parser.add_argument("--agent", default="simple_agent", help="Agent to collect rollouts with.")
@@ -79,15 +78,46 @@ def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
     return parser.parse_args(argv)
 
 
+def _packaged_dataset(resources_server: str) -> Path:
+    """The environment's bundled ``example.jsonl`` as installed by the ``nemo-gym`` wheel.
+
+    Gym ships ``resources_servers`` beside ``nemo_gym`` in site-packages, configs and example data
+    included, so this resolves without a checkout. It only works when Gym is installed in *this*
+    interpreter — with Gym in a separate venv there is nothing to import, and the caller passes
+    ``--dataset`` instead.
+    """
+    try:
+        from importlib import resources
+    except ImportError as exc:  # pragma: no cover - importlib.resources is stdlib
+        raise SystemExit(f"cannot resolve a packaged dataset: {exc}") from exc
+    try:
+        dataset = Path(str(resources.files(f"resources_servers.{resources_server}") / "data" / "example.jsonl"))
+    except ModuleNotFoundError as exc:
+        raise SystemExit(
+            f"resources_servers.{resources_server} is not importable here, so its bundled dataset cannot be "
+            "located. Install Gym in this environment, or pass --dataset with the path to the jsonl you "
+            "want to run."
+        ) from exc
+    # An importable environment does not guarantee bundled data: only git-tracked files ship in the
+    # wheel, so an environment whose splits are downloaded at runtime has no example.jsonl. Checking
+    # here keeps the guidance identical to the import failure, rather than letting
+    # `discover_gym_tasks` raise a bare FileNotFoundError on a path the caller never chose.
+    if not dataset.is_file():
+        raise SystemExit(
+            f"{resources_server} ships no bundled dataset at {dataset}. Pass --dataset with the path to "
+            "the jsonl you want to run."
+        )
+    return dataset
+
+
 async def _main(args: argparse.Namespace) -> int:
     output_dir = args.output_dir or Path(tempfile.mkdtemp(prefix="gym-eval-"))
-    dataset = args.dataset or (args.gym_root / "resources_servers" / args.resources_server / "data" / "example.jsonl")
+    dataset = args.dataset or _packaged_dataset(args.resources_server)
     tasks = discover_gym_tasks(dataset)
     print(f"discovered {len(tasks)} tasks from {dataset}")
 
     runner = GymAgentTaskRunner(
         config=GymRuntimeConfig(
-            gym_root=args.gym_root,
             agent=args.agent,
             agent_config=args.agent_config,
             resources_server=args.resources_server,

packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py

@@ -31,12 +31,28 @@
 without triggering Gym's split-driven data-prep): ``gym env start`` brings up the
 resources-server + agent + model servers, then ``gym eval run --no-serve --input
 <materialized dataset>`` collects rollouts against them. The runtime shells out
-to the ``gym`` executable in the caller-provided ``gym_root`` checkout — Gym
-resolves its environments from that repo and reads credentials from its
-(gitignored) ``env.yaml`` — so this SDK never imports ``nemo_gym`` and never
-handles secrets. Subprocess output is streamed to log files under the run's work
-dir *and* mirrored to this module's logger at ``DEBUG``, so callers choose
-terminal visibility through ordinary ``logging`` configuration.
+to the ``gym`` CLI on PATH, so this SDK never imports ``nemo_gym``. Subprocess
+output is streamed to log files under the run's work dir *and* mirrored to this
+module's logger at ``DEBUG``, so callers choose terminal visibility through
+ordinary ``logging`` configuration.
+
+**Where Gym finds things.** NeMo Gym must be installed and its ``gym`` on PATH,
+along with the target environment's own dependencies. Generally that means a
+*separate* environment: Gym imports Ray at module load, and nemo-platform
+excludes Ray by constraint over an unfixed CVE, so the two cannot share one. In a
+job image the image owns PATH and this is unremarkable. There is deliberately no
+config field naming a checkout, a venv, or a search root — these runner configs
+become serialized job specs, and a local filesystem path means nothing on the
+other side of that boundary. Environments themselves ship in the ``nemo-gym``
+wheel (``resources_servers`` and friends install beside ``nemo_gym``, configs and
+example data included), so no checkout is needed to reach them.
+
+The subprocesses inherit this process's working directory, which is where Gym
+looks for the gitignored ``env.yaml`` holding the collector's credentials before
+falling back to its install root — so credentials never pass through this SDK.
+Run from the directory holding that file; a Gym checkout there also has its
+components take precedence, which is how you reach an environment the wheel does
+not carry.
 
 **Boundaries**: the caller is responsible for a
 Gym runtime whose deps are installed (each Gym env ships its own
@@ -54,6 +70,7 @@
 import math
 import os
 import re
+import shutil
 import signal
 import tempfile
 from collections import deque
@@ -72,6 +89,12 @@
 
 #: Reward key read from each Gym rollout record.
 DEFAULT_REWARD_KEY = "reward"
+#: Gym's CLI, expected on PATH. Not configurable: these runner configs become serialized job specs,
+#: and a path into somebody's venv is meaningless on the other side of that boundary. Note this name
+#: is only ever *resolved*, never executed: :func:`_gym_executable` turns it into an absolute path
+#: once, and that path is what the subprocesses run — so a child whose PATH differs from ours cannot
+#: end up executing a different Gym.
+_GYM_CLI = "gym"
 #: Gym's index fields on each rollout record. ``_ng_task_index`` is the only join back to the input
 #: rows that survives a round-trip: Gym mutates ``responses_create_params`` (even the prompt) and
 #: copies only a fixed allowlist of row keys onto the result, so no field we invent comes back. Gym
@@ -205,13 +228,6 @@ class GymRuntimeConfig(BaseModel):
 
     model_config = ConfigDict(extra="forbid")
 
-    gym_root: Path = Field(
-        description="NeMo Gym checkout directory; the CLI resolves envs/agents/models from here and "
-        "reads credentials from its gitignored env.yaml.",
-    )
-    gym_bin: Path | None = Field(
-        default=None, description="Path to the `gym` executable; defaults to <gym_root>/.venv/bin/gym."
-    )
     agent: str = Field(description="Agent name to collect rollouts with, e.g. 'simple_agent'.")
     agent_config: str = Field(description="Repo-relative agent config passed to `gym env start` (--config).")
     resources_server: str = Field(description="Resources-server (environment) name, e.g. 'mcqa' (--resources-server).")
@@ -253,8 +269,24 @@ class GymRuntimeConfig(BaseModel):
     )
     reward_key: str = Field(default=DEFAULT_REWARD_KEY, description="Key read from each rollout record.")
 
-    def gym_executable(self) -> Path:
-        return self.gym_bin if self.gym_bin is not None else self.gym_root / ".venv" / "bin" / "gym"
+
+def _gym_executable() -> str:
+    """Locate the ``gym`` CLI on PATH, or fail saying what to do about it.
+
+    Gym is expected to be installed in the same environment as this SDK — there is deliberately no
+    config field pointing at a checkout or another venv, because these runner configs become
+    serialized job specs and a local filesystem path cannot cross that boundary. Resolving here
+    rather than at spawn time turns a missing Gym into one legible error instead of an ``ENOENT``
+    out of ``create_subprocess_exec`` after the run has already started.
+    """
+    resolved = shutil.which(_GYM_CLI)
+    if resolved is None:
+        raise RuntimeError(
+            f"The {_GYM_CLI!r} CLI was not found on PATH. NeMo Gym must be installed in the same Python "
+            "environment as this SDK: `pip install nemo-gym`, plus the target environment's own "
+            "dependencies (each resources-server ships a requirements.txt)."
+        )
+    return resolved
 
 
 class GymRewardMetric:
@@ -442,7 +474,7 @@ async def run_tasks(
     async def _run_two_step(self, input_path: Path, output_path: Path, work_dir: Path) -> None:
         """Start the Gym servers, collect against them with ``--no-serve``, then tear them down."""
         cfg = self._config
-        gym = str(cfg.gym_executable())
+        gym = _gym_executable()
         env_log = work_dir / "gym_env.log"
 
         # Gym launches each server from its own subdir with its own .venv. Ray (>=2.56) otherwise
@@ -477,7 +509,6 @@ async def _run_two_step(self, input_path: Path, output_path: Path, work_dir: Pat
         # to land on a particular stream.
         env_proc = await asyncio.create_subprocess_exec(
             *env_cmd,
-            cwd=str(cfg.gym_root),
             env=subprocess_env,
             stdout=asyncio.subprocess.PIPE,
             stderr=asyncio.subprocess.STDOUT,
@@ -526,7 +557,6 @@ async def _collect_rollouts(
         stderr_log = work_dir / "gym_eval.stderr.log"
         eval_proc = await asyncio.create_subprocess_exec(
             *eval_cmd,
-            cwd=str(cfg.gym_root),
             env=subprocess_env,
             stdout=asyncio.subprocess.PIPE,
             stderr=asyncio.subprocess.PIPE,

packages/nemo_evaluator_sdk/tests/agent_eval/test_run_metadata.py

@@ -87,9 +87,7 @@ class _Provider:
             {"provider", "image", "adapter_id", "skills"},
         ),
         (
-            GymAgentTaskRunner(
-                config=GymRuntimeConfig(gym_root=Path("/x"), agent="a", agent_config="c", resources_server="r")
-            ),
+            GymAgentTaskRunner(config=GymRuntimeConfig(agent="a", agent_config="c", resources_server="r")),
             "gym",
             {
                 "resources_server",
@@ -190,13 +188,10 @@ def _info(**kwargs):
 def test_gym_redacts_credential_looking_env_overrides() -> None:
     # env_overrides is a free-form Hydra escape hatch forwarded to `gym env start`, and RunnerInfo.config
     # is persisted into the run bundle — so a value that looks like a credential must not be written there.
-    from pathlib import Path
-
     from nemo_evaluator_sdk.agent_eval.runtimes.gym_runtime import GymAgentTaskRunner, GymRuntimeConfig
 
     runner = GymAgentTaskRunner(
         config=GymRuntimeConfig(
-            gym_root=Path("/x"),
             agent="a",
             agent_config="c",
             resources_server="r",