From b0262e7baceb6f69913a22fa118e25005841cb9e Mon Sep 17 00:00:00 2001 From: Sandy Chapman Date: Mon, 10 Aug 2026 09:43:37 -0300 Subject: [PATCH 1/3] refactor(evaluator)!: resolve the Gym CLI from PATH instead of a configured checkout MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `GymRuntimeConfig` no longer carries `gym_root` or a `gym_bin` path. The runner resolves `gym` from PATH, so NeMo Gym must be installed in the same environment as the SDK. Two reasons the old shape does not survive: * These runner configs become serialized job specs when Gym runs as a governed platform job. A path into somebody's checkout or venv means nothing on the other side of that boundary, so the fields would only ever be stripped or ignored server-side. * A checkout is no longer required. NeMo-Gym now ships its component trees in the wheel — `resources_servers` and friends install beside `nemo_gym`, configs and `data/example.jsonl` included — so a plain `pip install nemo-gym` resolves environments and their example data with nothing on disk. `gym_root` had exactly two jobs: defaulting `gym_bin` to `/.venv/bin/gym`, and setting the subprocess cwd. The first is replaced by `shutil.which`, matching how `CodexCliAgentRuntime` already resolves its own CLI. The second is dropped: the subprocesses inherit this process's working directory, which is where Gym looks for the gitignored `env.yaml` holding the collector's credentials before falling back to its install root. Running from a Gym checkout still makes its components take precedence, so the capability survives without a field to configure it. A missing CLI now fails with an actionable message before the run starts, instead of an ENOENT out of `create_subprocess_exec` partway through. The mcqa example drops `--gym-root`, defaults its dataset to the packaged `resources_servers//data/example.jsonl`, and its README leads with `pip install nemo-gym` rather than a checkout. BREAKING CHANGE: `GymRuntimeConfig.gym_root` and `GymRuntimeConfig.gym_bin` are removed. Install NeMo Gym in the same environment as the SDK; run from the directory holding `env.yaml`, or from a Gym checkout when you need components the wheel does not carry. Signed-off-by: Sandy Chapman --- .../nemo_evaluator_sdk/examples/gym/README.md | 27 +++++---- .../examples/gym/inspect_results.py | 2 +- .../examples/gym/run_gym_eval.py | 46 ++++++++++---- .../agent_eval/runtimes/gym_runtime.py | 60 +++++++++++++------ .../tests/agent_eval/test_run_metadata.py | 7 +-- .../agent_eval/runtimes/gym_runtime.py | 60 +++++++++++++------ 6 files changed, 134 insertions(+), 68 deletions(-) diff --git a/packages/nemo_evaluator_sdk/examples/gym/README.md b/packages/nemo_evaluator_sdk/examples/gym/README.md index 5ef48ba1a2..5a61602d09 100644 --- a/packages/nemo_evaluator_sdk/examples/gym/README.md +++ b/packages/nemo_evaluator_sdk/examples/gym/README.md @@ -6,17 +6,18 @@ Mapping: one Gym dataset → one run; each distinct row → one `AgentEvalTask` ## Prerequisites -A working **NeMo Gym checkout** is required — Gym resolves its environments from the repo, not from a package install. From the Gym checkout: +**1. NeMo Gym installed in the same environment as the SDK**, plus the target environment's own dependencies. Environments ship in the `nemo-gym` wheel — `resources_servers` and friends install beside `nemo_gym`, configs and `data/example.jsonl` included — so no checkout is needed: ```bash -# 1. Gym venv + framework (Gym pins uv >= 0.9.30; the workspace floor must satisfy that) -uv venv --python 3.12 .venv -uv pip install --no-config --python .venv/bin/python -e ".[dev]" "ray[default]>=2.55.1" +pip install nemo-gym +pip install tiktoken # mcqa's own dependency; each resources_server ships a requirements.txt +``` + +The runner shells out to whatever `gym` is on PATH. There is deliberately no setting for a checkout, another venv, or a search root: this config becomes a serialized job spec when Gym runs as a platform job, and a local path means nothing on the other side of that boundary. -# 2. The target env's own deps (each resources_server ships a requirements.txt) -uv pip install --no-config --python .venv/bin/python tiktoken # mcqa needs tiktoken +**2. Model credentials for the collector** — a gitignored `env.yaml` in the directory you run from: -# 3. Model credentials for the collector — a gitignored env.yaml at the Gym repo root: +```bash cat > env.yaml <<'YAML' policy_base_url: https:///v1 policy_api_key: @@ -24,17 +25,19 @@ policy_model_name: YAML ``` -> `env.yaml` is gitignored by the Gym repo — the credentials stay local. +> Gym searches the working directory, then its install root. This SDK never reads that file — the `gym` subprocess does. ## Run -From the nemo-platform repo root (any Python with `nemo_evaluator_sdk` importable — the runner shells out to Gym's own venv): +From the directory holding `env.yaml`: ```bash -uv run python -m packages.nemo_evaluator_sdk.examples.gym.run_gym_eval --gym-root /path/to/Gym +uv run python -m packages.nemo_evaluator_sdk.examples.gym.run_gym_eval ``` -Useful flags: `--resources-server`, `--agent`, `--model-type` (`inference_provider` for OpenAI-compatible **chat** endpoints; `openai_model` uses the OpenAI **Responses API** and 500s against chat-only endpoints), `--num-repeats`, `--output-dir`. +Working from a Gym checkout also makes its components take precedence over the packaged ones, which is how you reach an environment whose data the wheel does not carry (bulk train/validation splits are not git-tracked, so they are not in it). + +Useful flags: `--resources-server`, `--agent`, `--model-type` (`inference_provider` for OpenAI-compatible **chat** endpoints; `openai_model` uses the OpenAI **Responses API** and 500s against chat-only endpoints), `--num-repeats`, `--dataset`, `--output-dir`. For the full set of knobs the underlying `gym env start` / `gym eval run` commands accept, see the [NeMo Gym documentation](https://github.com/NVIDIA-NeMo/Gym). Anything `GymRuntimeConfig` does not expose as a field can be passed through with its `env_overrides` escape hatch (Hydra `+key=value` overrides applied to `gym env start`). @@ -60,7 +63,7 @@ No bundle is checked in; the run above produces one. Give it a stable `--output- ```bash uv run python -m packages.nemo_evaluator_sdk.examples.gym.run_gym_eval \ - --gym-root /path/to/Gym --output-dir /tmp/gym-eval + --output-dir /tmp/gym-eval uv run python -m packages.nemo_evaluator_sdk.examples.gym.inspect_results --bundle /tmp/gym-eval ``` diff --git a/packages/nemo_evaluator_sdk/examples/gym/inspect_results.py b/packages/nemo_evaluator_sdk/examples/gym/inspect_results.py index 834c403ddf..fae62e617b 100644 --- a/packages/nemo_evaluator_sdk/examples/gym/inspect_results.py +++ b/packages/nemo_evaluator_sdk/examples/gym/inspect_results.py @@ -17,7 +17,7 @@ same path. From the repository root:: uv run python -m packages.nemo_evaluator_sdk.examples.gym.run_gym_eval \\ - --gym-root /path/to/Gym --output-dir /tmp/gym-eval + --output-dir /tmp/gym-eval uv run python -m packages.nemo_evaluator_sdk.examples.gym.inspect_results --bundle /tmp/gym-eval Any agent-eval bundle works, not just a Gym one: pass ``--metric-type``/``--output-name`` for the diff --git a/packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py b/packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py index 72a856399e..971fdb0218 100644 --- a/packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py +++ b/packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py @@ -16,15 +16,16 @@ Prerequisites (see README.md): -* A **NeMo Gym checkout** whose venv has the target env's deps installed — each env ships its own - ``requirements.txt`` (mcqa needs ``tiktoken``; Gym itself needs ``ray`` and ``uv >= 0.9.30``). -* A **gitignored ``/env.yaml``** holding the model credentials the collector calls - (``policy_base_url`` / ``policy_api_key`` / ``policy_model_name``). This SDK never handles secrets — - the ``gym`` subprocess reads them from that file. +* **NeMo Gym installed in this environment** (``pip install nemo-gym``), plus the target env's own + deps — each env ships its own ``requirements.txt`` (mcqa needs ``tiktoken``). Environments ship in + the wheel, so no checkout is needed for the bundled example data. +* A **gitignored ``env.yaml``** holding the model credentials the collector calls + (``policy_base_url`` / ``policy_api_key`` / ``policy_model_name``) in the directory you run from. + This SDK never handles secrets — the ``gym`` subprocess reads them from that file. -Run from the repository root:: +Run from the directory holding ``env.yaml``:: - uv run python -m packages.nemo_evaluator_sdk.examples.gym.run_gym_eval --gym-root /path/to/Gym + uv run python -m packages.nemo_evaluator_sdk.examples.gym.run_gym_eval Pass ``--output-dir`` to write the bundle somewhere stable, then read it with ``inspect_results.py``. """ @@ -47,14 +48,12 @@ def _parse_args(argv: list[str] | None = None) -> argparse.Namespace: parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - parser.add_argument( - "--gym-root", required=True, type=Path, help="NeMo Gym checkout; its venv + env.yaml provide deps/creds." - ) parser.add_argument( "--dataset", type=Path, default=None, - help="Dataset jsonl (default: /resources_servers//data/example.jsonl).", + help="Dataset jsonl (default: the installed package's " + "resources_servers//data/example.jsonl).", ) parser.add_argument("--resources-server", default="mcqa", help="Gym resources-server (environment) name.") parser.add_argument("--agent", default="simple_agent", help="Agent to collect rollouts with.") @@ -79,15 +78,36 @@ def _parse_args(argv: list[str] | None = None) -> argparse.Namespace: return parser.parse_args(argv) +def _packaged_dataset(resources_server: str) -> Path: + """The environment's bundled ``example.jsonl`` as installed by the ``nemo-gym`` wheel. + + Gym ships ``resources_servers`` beside ``nemo_gym`` in site-packages, configs and example data + included, so this resolves without a checkout. It only works when Gym is installed in *this* + interpreter — with Gym in a separate venv there is nothing to import, and the caller passes + ``--dataset`` instead. + """ + try: + from importlib import resources + except ImportError as exc: # pragma: no cover - importlib.resources is stdlib + raise SystemExit(f"cannot resolve a packaged dataset: {exc}") from exc + try: + return Path(str(resources.files(f"resources_servers.{resources_server}") / "data" / "example.jsonl")) + except ModuleNotFoundError as exc: + raise SystemExit( + f"resources_servers.{resources_server} is not importable here, so its bundled dataset cannot be " + "located. Install Gym in this environment (`pip install nemo-gym`), or pass --dataset with the " + "path to the jsonl you want to run." + ) from exc + + async def _main(args: argparse.Namespace) -> int: output_dir = args.output_dir or Path(tempfile.mkdtemp(prefix="gym-eval-")) - dataset = args.dataset or (args.gym_root / "resources_servers" / args.resources_server / "data" / "example.jsonl") + dataset = args.dataset or _packaged_dataset(args.resources_server) tasks = discover_gym_tasks(dataset) print(f"discovered {len(tasks)} tasks from {dataset}") runner = GymAgentTaskRunner( config=GymRuntimeConfig( - gym_root=args.gym_root, agent=args.agent, agent_config=args.agent_config, resources_server=args.resources_server, diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py index 8a968dba8f..8316aa97a9 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py @@ -31,12 +31,25 @@ without triggering Gym's split-driven data-prep): ``gym env start`` brings up the resources-server + agent + model servers, then ``gym eval run --no-serve --input `` collects rollouts against them. The runtime shells out -to the ``gym`` executable in the caller-provided ``gym_root`` checkout — Gym -resolves its environments from that repo and reads credentials from its -(gitignored) ``env.yaml`` — so this SDK never imports ``nemo_gym`` and never -handles secrets. Subprocess output is streamed to log files under the run's work -dir *and* mirrored to this module's logger at ``DEBUG``, so callers choose -terminal visibility through ordinary ``logging`` configuration. +to the ``gym`` CLI on PATH, so this SDK never imports ``nemo_gym``. Subprocess +output is streamed to log files under the run's work dir *and* mirrored to this +module's logger at ``DEBUG``, so callers choose terminal visibility through +ordinary ``logging`` configuration. + +**Where Gym finds things.** NeMo Gym must be installed in the same environment as +this SDK, along with the target environment's own dependencies. There is no +config field naming a checkout, a venv, or a search root: these runner configs +become serialized job specs, and a local filesystem path means nothing on the +other side of that boundary. Environments ship in the ``nemo-gym`` wheel +(``resources_servers`` and friends install beside ``nemo_gym``, configs and +example data included), so an install is sufficient to run them. + +The subprocesses inherit this process's working directory, which is where Gym +looks for the gitignored ``env.yaml`` holding the collector's credentials before +falling back to its install root — so credentials never pass through this SDK. +Run from the directory holding that file; a Gym checkout there also has its +components take precedence, which is how you reach an environment the wheel does +not carry. **Boundaries**: the caller is responsible for a Gym runtime whose deps are installed (each Gym env ships its own @@ -54,6 +67,7 @@ import math import os import re +import shutil import signal import tempfile from collections import deque @@ -72,6 +86,9 @@ #: Reward key read from each Gym rollout record. DEFAULT_REWARD_KEY = "reward" +#: Gym's CLI, expected on PATH. Not configurable: these runner configs become serialized job specs, +#: and a path into somebody's venv is meaningless on the other side of that boundary. +_GYM_CLI = "gym" #: Gym's index fields on each rollout record. ``_ng_task_index`` is the only join back to the input #: rows that survives a round-trip: Gym mutates ``responses_create_params`` (even the prompt) and #: copies only a fixed allowlist of row keys onto the result, so no field we invent comes back. Gym @@ -205,13 +222,6 @@ class GymRuntimeConfig(BaseModel): model_config = ConfigDict(extra="forbid") - gym_root: Path = Field( - description="NeMo Gym checkout directory; the CLI resolves envs/agents/models from here and " - "reads credentials from its gitignored env.yaml.", - ) - gym_bin: Path | None = Field( - default=None, description="Path to the `gym` executable; defaults to /.venv/bin/gym." - ) agent: str = Field(description="Agent name to collect rollouts with, e.g. 'simple_agent'.") agent_config: str = Field(description="Repo-relative agent config passed to `gym env start` (--config).") resources_server: str = Field(description="Resources-server (environment) name, e.g. 'mcqa' (--resources-server).") @@ -253,8 +263,24 @@ class GymRuntimeConfig(BaseModel): ) reward_key: str = Field(default=DEFAULT_REWARD_KEY, description="Key read from each rollout record.") - def gym_executable(self) -> Path: - return self.gym_bin if self.gym_bin is not None else self.gym_root / ".venv" / "bin" / "gym" + +def _gym_executable() -> str: + """Locate the ``gym`` CLI on PATH, or fail saying what to do about it. + + Gym is expected to be installed in the same environment as this SDK — there is deliberately no + config field pointing at a checkout or another venv, because these runner configs become + serialized job specs and a local filesystem path cannot cross that boundary. Resolving here + rather than at spawn time turns a missing Gym into one legible error instead of an ``ENOENT`` + out of ``create_subprocess_exec`` after the run has already started. + """ + resolved = shutil.which(_GYM_CLI) + if resolved is None: + raise RuntimeError( + f"The {_GYM_CLI!r} CLI was not found on PATH. NeMo Gym must be installed in the same " + "environment as this SDK: `pip install nemo-gym`, plus the target environment's own " + "dependencies (each resources-server ships a requirements.txt)." + ) + return resolved class GymRewardMetric: @@ -442,7 +468,7 @@ async def run_tasks( async def _run_two_step(self, input_path: Path, output_path: Path, work_dir: Path) -> None: """Start the Gym servers, collect against them with ``--no-serve``, then tear them down.""" cfg = self._config - gym = str(cfg.gym_executable()) + gym = _gym_executable() env_log = work_dir / "gym_env.log" # Gym launches each server from its own subdir with its own .venv. Ray (>=2.56) otherwise @@ -477,7 +503,6 @@ async def _run_two_step(self, input_path: Path, output_path: Path, work_dir: Pat # to land on a particular stream. env_proc = await asyncio.create_subprocess_exec( *env_cmd, - cwd=str(cfg.gym_root), env=subprocess_env, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT, @@ -526,7 +551,6 @@ async def _collect_rollouts( stderr_log = work_dir / "gym_eval.stderr.log" eval_proc = await asyncio.create_subprocess_exec( *eval_cmd, - cwd=str(cfg.gym_root), env=subprocess_env, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, diff --git a/packages/nemo_evaluator_sdk/tests/agent_eval/test_run_metadata.py b/packages/nemo_evaluator_sdk/tests/agent_eval/test_run_metadata.py index cdea5624bc..326ede598e 100644 --- a/packages/nemo_evaluator_sdk/tests/agent_eval/test_run_metadata.py +++ b/packages/nemo_evaluator_sdk/tests/agent_eval/test_run_metadata.py @@ -87,9 +87,7 @@ class _Provider: {"provider", "image", "adapter_id", "skills"}, ), ( - GymAgentTaskRunner( - config=GymRuntimeConfig(gym_root=Path("/x"), agent="a", agent_config="c", resources_server="r") - ), + GymAgentTaskRunner(config=GymRuntimeConfig(agent="a", agent_config="c", resources_server="r")), "gym", { "resources_server", @@ -190,13 +188,10 @@ def _info(**kwargs): def test_gym_redacts_credential_looking_env_overrides() -> None: # env_overrides is a free-form Hydra escape hatch forwarded to `gym env start`, and RunnerInfo.config # is persisted into the run bundle — so a value that looks like a credential must not be written there. - from pathlib import Path - from nemo_evaluator_sdk.agent_eval.runtimes.gym_runtime import GymAgentTaskRunner, GymRuntimeConfig runner = GymAgentTaskRunner( config=GymRuntimeConfig( - gym_root=Path("/x"), agent="a", agent_config="c", resources_server="r", diff --git a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py index 6667fe93f9..549c7bece5 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py +++ b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py @@ -31,12 +31,25 @@ without triggering Gym's split-driven data-prep): ``gym env start`` brings up the resources-server + agent + model servers, then ``gym eval run --no-serve --input `` collects rollouts against them. The runtime shells out -to the ``gym`` executable in the caller-provided ``gym_root`` checkout — Gym -resolves its environments from that repo and reads credentials from its -(gitignored) ``env.yaml`` — so this SDK never imports ``nemo_gym`` and never -handles secrets. Subprocess output is streamed to log files under the run's work -dir *and* mirrored to this module's logger at ``DEBUG``, so callers choose -terminal visibility through ordinary ``logging`` configuration. +to the ``gym`` CLI on PATH, so this SDK never imports ``nemo_gym``. Subprocess +output is streamed to log files under the run's work dir *and* mirrored to this +module's logger at ``DEBUG``, so callers choose terminal visibility through +ordinary ``logging`` configuration. + +**Where Gym finds things.** NeMo Gym must be installed in the same environment as +this SDK, along with the target environment's own dependencies. There is no +config field naming a checkout, a venv, or a search root: these runner configs +become serialized job specs, and a local filesystem path means nothing on the +other side of that boundary. Environments ship in the ``nemo-gym`` wheel +(``resources_servers`` and friends install beside ``nemo_gym``, configs and +example data included), so an install is sufficient to run them. + +The subprocesses inherit this process's working directory, which is where Gym +looks for the gitignored ``env.yaml`` holding the collector's credentials before +falling back to its install root — so credentials never pass through this SDK. +Run from the directory holding that file; a Gym checkout there also has its +components take precedence, which is how you reach an environment the wheel does +not carry. **Boundaries**: the caller is responsible for a Gym runtime whose deps are installed (each Gym env ships its own @@ -54,6 +67,7 @@ import math import os import re +import shutil import signal import tempfile from collections import deque @@ -72,6 +86,9 @@ #: Reward key read from each Gym rollout record. DEFAULT_REWARD_KEY = "reward" +#: Gym's CLI, expected on PATH. Not configurable: these runner configs become serialized job specs, +#: and a path into somebody's venv is meaningless on the other side of that boundary. +_GYM_CLI = "gym" #: Gym's index fields on each rollout record. ``_ng_task_index`` is the only join back to the input #: rows that survives a round-trip: Gym mutates ``responses_create_params`` (even the prompt) and #: copies only a fixed allowlist of row keys onto the result, so no field we invent comes back. Gym @@ -205,13 +222,6 @@ class GymRuntimeConfig(BaseModel): model_config = ConfigDict(extra="forbid") - gym_root: Path = Field( - description="NeMo Gym checkout directory; the CLI resolves envs/agents/models from here and " - "reads credentials from its gitignored env.yaml.", - ) - gym_bin: Path | None = Field( - default=None, description="Path to the `gym` executable; defaults to /.venv/bin/gym." - ) agent: str = Field(description="Agent name to collect rollouts with, e.g. 'simple_agent'.") agent_config: str = Field(description="Repo-relative agent config passed to `gym env start` (--config).") resources_server: str = Field(description="Resources-server (environment) name, e.g. 'mcqa' (--resources-server).") @@ -253,8 +263,24 @@ class GymRuntimeConfig(BaseModel): ) reward_key: str = Field(default=DEFAULT_REWARD_KEY, description="Key read from each rollout record.") - def gym_executable(self) -> Path: - return self.gym_bin if self.gym_bin is not None else self.gym_root / ".venv" / "bin" / "gym" + +def _gym_executable() -> str: + """Locate the ``gym`` CLI on PATH, or fail saying what to do about it. + + Gym is expected to be installed in the same environment as this SDK — there is deliberately no + config field pointing at a checkout or another venv, because these runner configs become + serialized job specs and a local filesystem path cannot cross that boundary. Resolving here + rather than at spawn time turns a missing Gym into one legible error instead of an ``ENOENT`` + out of ``create_subprocess_exec`` after the run has already started. + """ + resolved = shutil.which(_GYM_CLI) + if resolved is None: + raise RuntimeError( + f"The {_GYM_CLI!r} CLI was not found on PATH. NeMo Gym must be installed in the same " + "environment as this SDK: `pip install nemo-gym`, plus the target environment's own " + "dependencies (each resources-server ships a requirements.txt)." + ) + return resolved class GymRewardMetric: @@ -442,7 +468,7 @@ async def run_tasks( async def _run_two_step(self, input_path: Path, output_path: Path, work_dir: Path) -> None: """Start the Gym servers, collect against them with ``--no-serve``, then tear them down.""" cfg = self._config - gym = str(cfg.gym_executable()) + gym = _gym_executable() env_log = work_dir / "gym_env.log" # Gym launches each server from its own subdir with its own .venv. Ray (>=2.56) otherwise @@ -477,7 +503,6 @@ async def _run_two_step(self, input_path: Path, output_path: Path, work_dir: Pat # to land on a particular stream. env_proc = await asyncio.create_subprocess_exec( *env_cmd, - cwd=str(cfg.gym_root), env=subprocess_env, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT, @@ -526,7 +551,6 @@ async def _collect_rollouts( stderr_log = work_dir / "gym_eval.stderr.log" eval_proc = await asyncio.create_subprocess_exec( *eval_cmd, - cwd=str(cfg.gym_root), env=subprocess_env, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, From e636cf6eb3888969be5b71d9505f8e07c92013b0 Mon Sep 17 00:00:00 2001 From: Sandy Chapman Date: Mon, 10 Aug 2026 13:24:35 -0300 Subject: [PATCH 2/3] fix(evaluator): address review on the Gym CLI resolution change MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three fixes from review of #1196. **README contradicted itself.** The "How it runs Gym" section still said the executable comes from "your checkout" and credentials from "that checkout's env.yaml", which the rewritten prerequisites had already replaced. It now says `gym` is resolved from PATH and `env.yaml` is read from the directory you run from. **`_packaged_dataset` returned paths it never checked.** It caught `ModuleNotFoundError` but not a missing file, so an importable environment without bundled data returned a path that does not exist and left `discover_gym_tasks` to raise a bare `FileNotFoundError` about a path the caller never chose. Only git-tracked files ship in the wheel, so an environment whose splits are downloaded at runtime has no `example.jsonl` — this is reachable, not theoretical. It now fails with the same `--dataset` guidance as the import case. **Clarified that `_GYM_CLI` is resolved, never executed.** Review asked whether the resolving and executing processes could disagree about PATH. They cannot: `shutil.which` returns an absolute path and that is what the subprocesses run. The constant reads like a bare name, so the reasoning is now written down next to it. Signed-off-by: Sandy Chapman --- .../nemo_evaluator_sdk/examples/gym/README.md | 2 +- .../examples/gym/run_gym_eval.py | 16 ++++++++++++--- .../agent_eval/runtimes/gym_runtime.py | 20 ++++++++++++------- .../agent_eval/runtimes/gym_runtime.py | 20 ++++++++++++------- 4 files changed, 40 insertions(+), 18 deletions(-) diff --git a/packages/nemo_evaluator_sdk/examples/gym/README.md b/packages/nemo_evaluator_sdk/examples/gym/README.md index 5a61602d09..ca04ba40b9 100644 --- a/packages/nemo_evaluator_sdk/examples/gym/README.md +++ b/packages/nemo_evaluator_sdk/examples/gym/README.md @@ -76,7 +76,7 @@ The runner uses Gym's **two-step** flow, which reads a dataset file directly (no 1. `gym env start …` — brings up the resources-server + agent + model servers. 2. `gym eval run --no-serve --input …` — collects rollouts against them. -It does **not** import `nemo_gym` and does **not** handle secrets — it invokes the `gym` executable in your checkout, and Gym reads credentials from that checkout's `env.yaml`. +It does **not** import `nemo_gym` and does **not** handle secrets — it invokes whichever `gym` executable is on `PATH`, resolved once to an absolute path so the subprocesses cannot pick up a different one, and Gym reads credentials from the `env.yaml` in the directory you run from. The dataset handed to step 2 is not your source file. The runner **materializes** a normalized one into the run's work directory: one row per requested task, with `_ng_task_index` stamped explicitly. Gym honors a caller-supplied `_ng_task_index` (it only assigns one when a row lacks it) and echoes it back on every rollout record, so rollouts join back to tasks through a map the runner owns rather than a guess about Gym's internal row ordering. It also means running a *subset* of tasks only rolls out that subset. diff --git a/packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py b/packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py index 971fdb0218..af12a94b1a 100644 --- a/packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py +++ b/packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py @@ -91,13 +91,23 @@ def _packaged_dataset(resources_server: str) -> Path: except ImportError as exc: # pragma: no cover - importlib.resources is stdlib raise SystemExit(f"cannot resolve a packaged dataset: {exc}") from exc try: - return Path(str(resources.files(f"resources_servers.{resources_server}") / "data" / "example.jsonl")) + dataset = Path(str(resources.files(f"resources_servers.{resources_server}") / "data" / "example.jsonl")) except ModuleNotFoundError as exc: raise SystemExit( f"resources_servers.{resources_server} is not importable here, so its bundled dataset cannot be " - "located. Install Gym in this environment (`pip install nemo-gym`), or pass --dataset with the " - "path to the jsonl you want to run." + "located. Install Gym in this environment, or pass --dataset with the path to the jsonl you " + "want to run." ) from exc + # An importable environment does not guarantee bundled data: only git-tracked files ship in the + # wheel, so an environment whose splits are downloaded at runtime has no example.jsonl. Checking + # here keeps the guidance identical to the import failure, rather than letting + # `discover_gym_tasks` raise a bare FileNotFoundError on a path the caller never chose. + if not dataset.is_file(): + raise SystemExit( + f"{resources_server} ships no bundled dataset at {dataset}. Pass --dataset with the path to " + "the jsonl you want to run." + ) + return dataset async def _main(args: argparse.Namespace) -> int: diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py index 8316aa97a9..90444b4cab 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py @@ -36,13 +36,16 @@ module's logger at ``DEBUG``, so callers choose terminal visibility through ordinary ``logging`` configuration. -**Where Gym finds things.** NeMo Gym must be installed in the same environment as -this SDK, along with the target environment's own dependencies. There is no -config field naming a checkout, a venv, or a search root: these runner configs +**Where Gym finds things.** NeMo Gym must be installed and its ``gym`` on PATH, +along with the target environment's own dependencies. Generally that means a +*separate* environment: Gym imports Ray at module load, and nemo-platform +excludes Ray by constraint over an unfixed CVE, so the two cannot share one. In a +job image the image owns PATH and this is unremarkable. There is deliberately no +config field naming a checkout, a venv, or a search root — these runner configs become serialized job specs, and a local filesystem path means nothing on the -other side of that boundary. Environments ship in the ``nemo-gym`` wheel -(``resources_servers`` and friends install beside ``nemo_gym``, configs and -example data included), so an install is sufficient to run them. +other side of that boundary. Environments themselves ship in the ``nemo-gym`` +wheel (``resources_servers`` and friends install beside ``nemo_gym``, configs and +example data included), so no checkout is needed to reach them. The subprocesses inherit this process's working directory, which is where Gym looks for the gitignored ``env.yaml`` holding the collector's credentials before @@ -87,7 +90,10 @@ #: Reward key read from each Gym rollout record. DEFAULT_REWARD_KEY = "reward" #: Gym's CLI, expected on PATH. Not configurable: these runner configs become serialized job specs, -#: and a path into somebody's venv is meaningless on the other side of that boundary. +#: and a path into somebody's venv is meaningless on the other side of that boundary. Note this name +#: is only ever *resolved*, never executed: :func:`_gym_executable` turns it into an absolute path +#: once, and that path is what the subprocesses run — so a child whose PATH differs from ours cannot +#: end up executing a different Gym. _GYM_CLI = "gym" #: Gym's index fields on each rollout record. ``_ng_task_index`` is the only join back to the input #: rows that survives a round-trip: Gym mutates ``responses_create_params`` (even the prompt) and diff --git a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py index 549c7bece5..1521869eff 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py +++ b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py @@ -36,13 +36,16 @@ module's logger at ``DEBUG``, so callers choose terminal visibility through ordinary ``logging`` configuration. -**Where Gym finds things.** NeMo Gym must be installed in the same environment as -this SDK, along with the target environment's own dependencies. There is no -config field naming a checkout, a venv, or a search root: these runner configs +**Where Gym finds things.** NeMo Gym must be installed and its ``gym`` on PATH, +along with the target environment's own dependencies. Generally that means a +*separate* environment: Gym imports Ray at module load, and nemo-platform +excludes Ray by constraint over an unfixed CVE, so the two cannot share one. In a +job image the image owns PATH and this is unremarkable. There is deliberately no +config field naming a checkout, a venv, or a search root — these runner configs become serialized job specs, and a local filesystem path means nothing on the -other side of that boundary. Environments ship in the ``nemo-gym`` wheel -(``resources_servers`` and friends install beside ``nemo_gym``, configs and -example data included), so an install is sufficient to run them. +other side of that boundary. Environments themselves ship in the ``nemo-gym`` +wheel (``resources_servers`` and friends install beside ``nemo_gym``, configs and +example data included), so no checkout is needed to reach them. The subprocesses inherit this process's working directory, which is where Gym looks for the gitignored ``env.yaml`` holding the collector's credentials before @@ -87,7 +90,10 @@ #: Reward key read from each Gym rollout record. DEFAULT_REWARD_KEY = "reward" #: Gym's CLI, expected on PATH. Not configurable: these runner configs become serialized job specs, -#: and a path into somebody's venv is meaningless on the other side of that boundary. +#: and a path into somebody's venv is meaningless on the other side of that boundary. Note this name +#: is only ever *resolved*, never executed: :func:`_gym_executable` turns it into an absolute path +#: once, and that path is what the subprocesses run — so a child whose PATH differs from ours cannot +#: end up executing a different Gym. _GYM_CLI = "gym" #: Gym's index fields on each rollout record. ``_ng_task_index`` is the only join back to the input #: rows that survives a round-trip: Gym mutates ``responses_create_params`` (even the prompt) and From d01b12ebb368ae7157fe0a5161329e2f5078584b Mon Sep 17 00:00:00 2001 From: Sandy Chapman Date: Mon, 10 Aug 2026 14:03:38 -0300 Subject: [PATCH 3/3] Apply suggestion from @ngoncharenko Co-authored-by: Nick Goncharenko <8766167+ngoncharenko@users.noreply.github.com> Signed-off-by: Sandy Chapman --- .../src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py | 2 +- .../beta/evaluator/agent_eval/runtimes/gym_runtime.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py index 90444b4cab..ce623ac894 100644 --- a/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py +++ b/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py @@ -282,7 +282,7 @@ def _gym_executable() -> str: resolved = shutil.which(_GYM_CLI) if resolved is None: raise RuntimeError( - f"The {_GYM_CLI!r} CLI was not found on PATH. NeMo Gym must be installed in the same " + f"The {_GYM_CLI!r} CLI was not found on PATH. NeMo Gym must be installed in the same Python " "environment as this SDK: `pip install nemo-gym`, plus the target environment's own " "dependencies (each resources-server ships a requirements.txt)." ) diff --git a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py index 1521869eff..912d6e8c30 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py +++ b/sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py @@ -282,7 +282,7 @@ def _gym_executable() -> str: resolved = shutil.which(_GYM_CLI) if resolved is None: raise RuntimeError( - f"The {_GYM_CLI!r} CLI was not found on PATH. NeMo Gym must be installed in the same " + f"The {_GYM_CLI!r} CLI was not found on PATH. NeMo Gym must be installed in the same Python " "environment as this SDK: `pip install nemo-gym`, plus the target environment's own " "dependencies (each resources-server ships a requirements.txt)." )