feat(evaluator): add convert-evals-to-harbor skill - #1181
Conversation
Port Harbor task-conversion guidance into a plugin-owned skill so agents can turn existing eval datasets into Harbor tasks NeMo Evaluator can run. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
📝 WalkthroughWalkthroughChangesHarbor conversion workflow
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (1)
plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/balance_splits.py (1)
57-63: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse concrete metadata types.
Both scripts use bare
dictannotations for metadata. Define a shared local metadata alias such asdict[str, object], then apply it to collection and row types.
plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/balance_splits.py#L57-L63: return a concrete metadata mapping type fromload_metadata.plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/plot_metadata_splits.py#L129-L142: use concrete row and metadata mapping types incollect_rows.As per coding guidelines, “Prefer concrete type hints over string-based annotations.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/balance_splits.py` around lines 57 - 63, Replace bare dict annotations with a shared concrete metadata mapping alias such as dict[str, object]. In plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/balance_splits.py:57-63, apply it to load_metadata’s return type; in plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/plot_metadata_splits.py:129-142, use concrete mapping types for collect_rows’ metadata and row annotations, reusing the shared alias where appropriate.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/balance_splits.py`:
- Around line 74-77: Update task discovery in balance_splits.py at lines 74-77
to collect every descendant directory containing task.toml by removing the early
break and preserving traversal of remaining entries. Apply the same complete
discovery behavior in plot_metadata_splits.py at lines 122-125; both functions
must include all nested task directories in grouped datasets.
In
`@plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/plot_metadata_splits.py`:
- Around line 287-289: Sanitize and validate each dimension before constructing
svg_name in the plotting flow, using a safe filename mapping that cannot
interpret metadata keys containing “/” or “..” as path components. Resolve the
resulting output path and verify it remains within output_dir before write_text,
while preserving the existing render_svg and svg_files behavior.
In
`@plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/SKILL.md`:
- Around line 783-827: Update the oracle template in the “Oracle solve.sh for
MCP-sidecar tasks” section so the Python replay script is executed through the
MCP sidecar’s /exec endpoint rather than locally in the main container. Preserve
the existing simulation setup and runtime-state generation, but route access to
/opt/tau2-bench and all task-environment commands through the sidecar execution
mechanism.
- Around line 131-132: Update the Harbor verifier contract documentation near
the references to test.sh and the sections around lines 230 and 569: state that
expected incorrect outputs must produce reward 0 and exit 0, while non-zero
exits are reserved for verifier or infrastructure failures such as missing or
malformed evidence.
- Around line 561-565: Update the task.toml checklist item to require
INFERENCE_API_KEY and INFERENCE_BASE_URL under the [environment].env section
instead of [agent].env. Keep the inference-only variable requirement and align
the wording with the Harbor AgentConfig schema example.
---
Nitpick comments:
In
`@plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/balance_splits.py`:
- Around line 57-63: Replace bare dict annotations with a shared concrete
metadata mapping alias such as dict[str, object]. In
plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/balance_splits.py:57-63,
apply it to load_metadata’s return type; in
plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/plot_metadata_splits.py:129-142,
use concrete mapping types for collect_rows’ metadata and row annotations,
reusing the shared alias where appropriate.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 840777c4-1ced-4ccb-bfed-e08e7a1b3539
📒 Files selected for processing (4)
plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/SKILL.mdplugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/balance_splits.pyplugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/plot_metadata_splits.pyplugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/tests.json
| for sub in sorted(entry.iterdir()): | ||
| if sub.is_dir() and (sub / "task.toml").exists(): | ||
| tasks.append(sub) | ||
| break |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Discover every nested task directory.
Both functions stop after the first nested task.toml. This omits tasks in grouped datasets and produces incorrect balance reports and plots.
plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/balance_splits.py#L74-L77: recurse through all descendant task directories withoutbreak.plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/plot_metadata_splits.py#L122-L125: use the same complete discovery behavior.
📍 Affects 2 files
plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/balance_splits.py#L74-L77(this comment)plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/plot_metadata_splits.py#L122-L125
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/balance_splits.py`
around lines 74 - 77, Update task discovery in balance_splits.py at lines 74-77
to collect every descendant directory containing task.toml by removing the early
break and preserving traversal of remaining entries. Apply the same complete
discovery behavior in plot_metadata_splits.py at lines 122-125; both functions
must include all nested task directories in grouped datasets.
| svg_name = f"{dimension}_by_split_pies.svg" | ||
| (output_dir / svg_name).write_text(render_svg(dataset, dimension, by_split, totals)) | ||
| svg_files.append(svg_name) |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Prevent output-path traversal through metadata keys.
dimension can contain / or .. from --dimensions or a task.toml key. svg_name then writes outside output_dir. Map dimensions to validated filenames and verify that each resolved output path remains under output_dir.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/scripts/plot_metadata_splits.py`
around lines 287 - 289, Sanitize and validate each dimension before constructing
svg_name in the plotting flow, using a safe filename mapping that cannot
interpret metadata keys containing “/” or “..” as path components. Resolve the
resulting output path and verify it remains within output_dir before write_text,
while preserving the existing render_svg and svg_files behavior.
| 4. `tests/test.sh` — verifier entry point (exit 0 always; failure = score 0) | ||
| 5. `tests/score.py` or `tests/evaluate.py` — scoring logic |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -e
file=$(fd --type f 'SKILL\.md$' plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor | head -n 1)
printf '%s\n' "FILE=$file"
printf '%s\n' '--- relevant sections ---'
sed -n '110,145p;210,245p;545,580p' "$file"
printf '%s\n' '--- verifier and exit-contract references ---'
rg -n -C 3 'exit 0|exit 1|non-zero|test\.sh|verifier|score 0|wrong output|failure' "$file"
printf '%s\n' '--- related files ---'
rg -n -C 2 'test\.sh|score\.py|evaluate\.py|exit 0|exit 1|Harbor verifier|verifier contract' plugins/nemo-evaluator/src/nemo_evaluator -g '*.py' -g '*.md' -g '*.sh' || trueRepository: NVIDIA-NeMo/nemo-platform
Length of output: 26612
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- repository Harbor verifier examples ---'
rg -n -C 5 'test\.sh|reward\.txt|reward\.json|score 0|retriable|non-zero|exit 0|exit 1' \
packages plugins examples -g '*.md' -g '*.sh' -g '*.py' -g '*.toml' -g '*.yaml' -g '*.yml' 2>/dev/null | head -n 800 || true
printf '%s\n' '--- Harbor references in manifests and docs ---'
rg -n -i -C 3 'harbor.*(verif|reward|test)|verif.*harbor|harbor' \
pyproject.toml uv.lock README.md packages plugins 2>/dev/null | head -n 500 || true
printf '%s\n' '--- available Harbor-related files ---'
git ls-files | rg -i 'harbor|test\.sh$|evaluate\.py$|score\.py$' | head -n 300Repository: NVIDIA-NeMo/nemo-platform
Length of output: 50384
🏁 Script executed:
#!/bin/bash
set -e
printf '%s\n' '--- local Harbor runtime and verifier status handling ---'
rg -n -C 5 'reward\.json|reward\.txt|verifier.*(exit|status|failed)|exit.*(zero|non.zero)|completed|failed|retriable|unmeasurable' \
packages/nemo_evaluator_sdk/src packages/nemo_evaluator_sdk/tests \
plugins/nemo-eval-author/src plugins/nemo-eval-author/tests \
-g '*.py' -g '*.md' -g '*.sh' | head -n 1200
printf '%s\n' '--- exact failing/compliant verifier contract test ---'
sed -n '370,435p;500,525p;575,600p' plugins/nemo-eval-author/tests/test_eval_author_repair_e2e.py
printf '%s\n' '--- Harbor runtime files ---'
git ls-files packages/nemo_evaluator_sdk | rg 'harbor_runtime|harbor|verifier' | head -n 100Repository: NVIDIA-NeMo/nemo-platform
Length of output: 50382
🌐 Web query:
Harbor benchmark task verifier test.sh reward file exit code expected failure score 0 retriable exception
💡 Result:
In the Harbor benchmark framework, the test.sh script is responsible for verifying agent performance and producing a reward signal [1][2]. The following technical requirements and practices apply: Reward File Requirements: - The script must write a numeric reward (integer or float) to /logs/verifier/reward.txt [3][4]. A secondary option exists to write to /logs/verifier/reward.json for structured data, but reward.txt is checked first [5][4]. - The value must be thresholded to a binary 0 or 1 for the final reward, even if internal sub-scores are calculated [6]. - Every reachable path in test.sh must write a reward value, including error paths [5][6][4]. Failing to write a reward file leads to a RewardFileNotFoundError, which is typically non-retriable by default [4][7]. Exit Codes and Scoring: - While test logic often uses exit codes (e.g., from pytest) to determine success or failure [3][8], the framework relies specifically on the content written to the reward file for scoring [2][9]. - It is standard practice to use an if/else block based on the test execution status (like $? -eq 0) to write 1 for success and 0 for failure [3][10][8]. Retriable Exceptions: - Retries are managed at the job-level configuration, not within test.sh [11]. - Certain exceptions are excluded from retries by default, meaning they will not trigger a re-run if they occur. These include AgentTimeoutError, VerifierTimeoutError, RewardFileNotFoundError, RewardFileEmptyError, and VerifierOutputParseError [7]. - Transient infrastructure failures can be configured for retries via the job's RetryConfig, provided the error does not fall into the excluded categories [12][7][13]. Infrastructure failures should not be masked by pre-writing 0 to the reward file, as this launders system errors into agent score failures [6].
Citations:
- 1: https://harbor-framework-harbor.mintlify.app/concepts/tasks
- 2: https://mintlify.wiki/harbor-framework/harbor/concepts/verifiers
- 3: https://harborframework.com/docs/datasets/adapters
- 4: https://skillspool.org/en/skills/qiushiyan-harbor-eval-experiments-claude-skills-harbor-adapter-creator-skill-md
- 5: https://mintlify.wiki/harbor-framework/harbor/guides/creating-tasks
- 6: https://github.com/harbor-framework/frontier-bench/blob/2d260bc7fb344dd13b1b920265e372b6dbc6a38d/rubrics/task-implementation.toml
- 7: https://harbor-framework-harbor.mintlify.app/api/job-config
- 8: https://mintlify.wiki/harbor-framework/harbor/cli/tasks
- 9: https://www.harborframework.com/docs/tasks
- 10: https://github.com/harbor-framework/frontier-bench/blob/main/tasks/batched-eval-parity/tests/test.sh
- 11: [codex] Remove internal trial timeout retries harbor-framework/harbor#1628
- 12: https://harbor-framework-harbor.mintlify.app/guides/parallel-execution
- 13: https://mintlify.wiki/harbor-framework/harbor/concepts/trials-and-jobs
Clarify the Harbor verifier contract.
test.sh must write reward 0 and exit 0 for expected wrong output. Reserve non-zero exits for verifier or infrastructure failures, such as missing or malformed evidence. Update lines 230 and 569 to state this distinction.
🧰 Tools
🪛 SkillSpector (2.5.1)
[error] 498: [AS2] MCP Config Access: Skill accesses MCP server configuration files (mcp.json). MCP configs contain server URLs, authentication tokens, and tool definitions — reading them allows the skill to discover and potentially abuse other tool integrations.
Remediation: Remove all code or instructions that read MCP configuration files (mcp.json). MCP server details should be managed by the agent runtime, not read by individual skills.
(Agent Snooping (AS2))
[error] 488: [E2] Env Variable Harvesting: Code accesses environment variables that may contain secrets (API keys, tokens). This is a common pattern for credential theft.
Remediation: Avoid reading sensitive env vars (API keys, tokens) unless strictly required. Use secrets managers or secure config. Never log or transmit credentials.
(Data Exfiltration (E2))
[warning] 352: [E5] Cloud Storage Exfiltration: Data is uploaded to cloud storage (S3 / GCS / Azure Blob). This may be a legitimate backup or exfiltration to an external bucket. Manual review is recommended.
Remediation: Verify the destination bucket is trusted and owned by you. Never upload credentials, secrets, or workspace contents to external or unverified cloud storage.
(Data Exfiltration (E5))
[warning] 365: [E5] Cloud Storage Exfiltration: Data is uploaded to cloud storage (S3 / GCS / Azure Blob). This may be a legitimate backup or exfiltration to an external bucket. Manual review is recommended.
Remediation: Verify the destination bucket is trusted and owned by you. Never upload credentials, secrets, or workspace contents to external or unverified cloud storage.
(Data Exfiltration (E5))
[error] 324: [PE3] Credential Access: Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.
Remediation: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths.
(Privilege Escalation (PE3))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/SKILL.md`
around lines 131 - 132, Update the Harbor verifier contract documentation near
the references to test.sh and the sections around lines 230 and 569: state that
expected incorrect outputs must produce reward 0 and exit 0, while non-zero
exits are reserved for verifier or infrastructure failures such as missing or
malformed evidence.
| - [ ] **`task.toml` `[agent]` section has `env` with `INFERENCE_API_KEY` and `INFERENCE_BASE_URL`** — do NOT include `OPENAI_API_KEY` / `OPENAI_BASE_URL`; inference vars are sufficient and other names create confusion. | ||
| - [ ] **`docker-compose.yaml` `main` service passes `INFERENCE_API_KEY` and `INFERENCE_BASE_URL`** — same rule, inference only. | ||
| - [ ] **`solve.sh` patches the sidecar, not the main container** — if the task environment lives in a sidecar, the oracle must apply changes via the sidecar's `/exec` HTTP endpoint, not directly on the filesystem of the main container (which is minimal and has no task environment). Use base64 encoding when writing patch content through the shell to avoid quoting issues. | ||
| - [ ] **`test.sh` runs task-environment commands via the sidecar's `/exec` endpoint** — the verifier runs in the main container which has no conda, no `/testbed`, no task-specific tools. Delegate all environment-dependent commands to the sidecar via `http://sidecar:8000/exec`. The sidecar's stdout capture is what parser/scorer reads. | ||
| - [ ] **`server.py` exposes a `/exec` custom route** alongside the MCP tools — allows `test.sh` and `solve.sh` to POST commands without implementing the full MCP protocol. Use `@mcp.custom_route("/exec", methods=["POST"])` from FastMCP v2. Body: `{"command": str, "cwd": str, "timeout": int}`. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
file="plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/SKILL.md"
printf '%s\n' '--- target sections ---'
sed -n '530,680p' "$file"
printf '%s\n' '--- Harbor schema and environment references ---'
rg -n -C 3 '\[agent\]|\[environment\]|INFERENCE_API_KEY|INFERENCE_BASE_URL|OPENAI_API_KEY|OPENAI_BASE_URL|task\.toml' \
plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor
printf '%s\n' '--- related repository references ---'
rg -n -C 2 'INFERENCE_API_KEY|INFERENCE_BASE_URL|\[environment\]|\[agent\]' \
plugins/nemo-evaluator --glob '*.md' --glob '*.toml' --glob '*.py' --glob '*.yaml' --glob '*.yml'Repository: NVIDIA-NeMo/nemo-platform
Length of output: 47851
🏁 Script executed:
#!/bin/bash
set -eu
file="plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/SKILL.md"
sed -n '530,680p' "$file"
rg -n -C 3 'INFERENCE_API_KEY|INFERENCE_BASE_URL|OPENAI_API_KEY|OPENAI_BASE_URL|\[agent\]|\[environment\]' \
plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor \
plugins/nemo-evaluatorRepository: NVIDIA-NeMo/nemo-platform
Length of output: 28462
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- all task.toml examples in the repository ---'
git ls-files '*task.toml' | head -80
printf '%s\n' '--- environment and agent env usage in repository examples ---'
rg -n -C 4 '^\[agent\]|^\[environment\]|^env\s*=' \
--glob 'task.toml' --glob '*.md' \
packages plugins | head -300
printf '%s\n' '--- Harbor-related dependency and schema references ---'
rg -n -C 3 'harbor|schema_version|verifier.*env|environment.*env|agent.*env' \
pyproject.toml uv.lock packages plugins \
--glob '*.toml' --glob '*.py' --glob '*.md' | head -400Repository: NVIDIA-NeMo/nemo-platform
Length of output: 43400
🌐 Web query:
Harbor task.toml schema [agent] env [environment] env verifier env
💡 Result:
In the Harbor framework's task.toml schema, environment variables can be defined for different phases using the env field within their respective sections [1][2][3]. Environment Variable Definitions: [environment].env: Defines variables for the base environment [3]. [agent].env: Defines variables used during agent execution (often applied via overrides in multi-step configurations) [4]. [verifier].env: Defines environment variables specifically for the verifier script [1][5][6]. Verifier Environment Configuration: The [verifier] section supports both a dedicated env field and a more complex [verifier.environment] configuration [1][2]. 1. Basic usage: You can specify simple environment variables directly under [verifier] using the env key [1][5]. [verifier] env = { KEY = "value" } 2. Dedicated verifier environment: If you need a separate environment (e.g., specific resources or network policies) for the verifier, you can define [verifier.environment] [1][2]. - If [verifier.environment] is present, the verifier defaults to "separate" mode, where it runs in its own container [1][2]. - If [verifier.environment] is omitted, it defaults to "shared" mode, where the verifier runs in the agent's environment [1][2]. - You can explicitly control this with the verifier.environment_mode field, setting it to "shared" or "separate" [1][2]. - The [verifier.environment] section follows the same schema as the top-level [environment] section [1][2].
Citations:
- 1: https://www.harborframework.com/docs/tasks
- 2: https://github.com/harbor-framework/harbor/blob/ff69e554fac1c751aa608e03de027db9043a2eac/src/harbor/models/task/config.py
- 3: https://github.com/harbor-framework/harbor/blob/main/examples/tasks/hello-skills/task.toml
- 4: https://www.harborframework.com/docs/tasks/multi-step
- 5: https://mintlify.wiki/harbor-framework/harbor/api/task-config
- 6: https://mintlify.wiki/harbor-framework/harbor/concepts/tasks
🏁 Script executed:
#!/bin/bash
set -eu
git ls-files '*task.toml' | head -80
rg -n -C 4 '^\[agent\]|^\[environment\]|^env\s*=' \
--glob 'task.toml' --glob '*.md' packages plugins | head -300
rg -n -C 3 'harbor|schema_version|verifier.*env|environment.*env|agent.*env' \
pyproject.toml uv.lock packages plugins \
--glob '*.toml' --glob '*.py' --glob '*.md' | head -400Repository: NVIDIA-NeMo/nemo-platform
Length of output: 44167
🏁 Script executed:
#!/bin/bash
set -eu
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
base="https://raw.githubusercontent.com/harbor-framework/harbor"
for ref in main v0.20.0; do
url="$base/$ref/src/harbor/models/task/config.py"
echo "--- $url ---"
if curl -fsSL "$url" -o "$tmp/config-$ref.py"; then
rg -n -C 5 'class (Agent|Environment|Verifier)|\benv\b|environment_mode' "$tmp/config-$ref.py" | head -240
else
echo "unavailable"
fi
done
echo '--- Harbor task documentation ---'
curl -fsSL "https://raw.githubusercontent.com/harbor-framework/harbor/main/docs/tasks.md" 2>/dev/null \
| rg -n -C 4 '\[agent\]|\[environment\]|\[verifier\]|env' | head -240 || trueRepository: NVIDIA-NeMo/nemo-platform
Length of output: 9322
🏁 Script executed:
#!/bin/bash
set -eu
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
for ref in main v0.20.0; do
url="https://raw.githubusercontent.com/harbor-framework/harbor/$ref/src/harbor/models/task/config.py"
echo "--- $url ---"
if curl -fsSL "$url" -o "$tmp/config-$ref.py"; then
rg -n -C 5 'class (Agent|Environment|Verifier)|\benv\b|environment_mode' \
"$tmp/config-$ref.py" | head -240
else
echo "unavailable"
fi
doneRepository: NVIDIA-NeMo/nemo-platform
Length of output: 9288
🏁 Script executed:
#!/bin/bash
set -eu
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
url="https://raw.githubusercontent.com/harbor-framework/harbor/v0.20.0/src/harbor/models/task/config.py"
curl -fsSL "$url" -o "$tmp/config.py"
sed -n '220,370p' "$tmp/config.py"
printf '%s\n' '--- all env declarations and model composition ---'
rg -n -C 8 'class PhaseNetworkPolicyConfig|class AgentConfig|class EnvironmentConfig|class VerifierConfig|env:' "$tmp/config.py"
printf '%s\n' '--- repository hello-world task ---'
cat -n packages/nemo_evaluator_sdk/examples/harbor/hello_world_dataset/hello-world/task.tomlRepository: NVIDIA-NeMo/nemo-platform
Length of output: 9681
🏁 Script executed:
#!/bin/bash
set -eu
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
curl -fsSL \
"https://raw.githubusercontent.com/harbor-framework/harbor/v0.20.0/src/harbor/models/task/config.py" \
-o "$tmp/config.py"
sed -n '220,370p' "$tmp/config.py"
rg -n -C 8 \
'class PhaseNetworkPolicyConfig|class AgentConfig|class EnvironmentConfig|class VerifierConfig|env:' \
"$tmp/config.py"
cat -n packages/nemo_evaluator_sdk/examples/harbor/hello_world_dataset/hello-world/task.tomlRepository: NVIDIA-NeMo/nemo-platform
Length of output: 9594
🏁 Script executed:
#!/bin/bash
set -eu
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
curl -fsSL \
"https://raw.githubusercontent.com/harbor-framework/harbor/v0.20.0/src/harbor/models/task/config.py" \
-o "$tmp/config.py"
printf '%s\n' '--- task-level model and extra-field policy ---'
rg -n -C 10 \
'class TaskConfig|model_config|extra=|agent:|environment:|verifier:' \
"$tmp/config.py" | head -300
printf '%s\n' '--- schema examples containing agent.env ---'
for ref in main v0.20.0; do
echo "--- $ref ---"
curl -fsSL "https://api.github.com/repos/harbor-framework/harbor/git/trees/$ref?recursive=1" \
| jq -r '.tree[]?.path' \
| rg 'task\.toml$|tasks/.+\.(md|toml)$' \
| head -80
doneRepository: NVIDIA-NeMo/nemo-platform
Length of output: 4650
🏁 Script executed:
#!/bin/bash
set -eu
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT
curl -fsSL \
"https://raw.githubusercontent.com/harbor-framework/harbor/v0.20.0/src/harbor/models/task/config.py" \
-o "$tmp/config.py"
rg -n -C 10 \
'class TaskConfig|model_config|extra=|agent:|environment:|verifier:' \
"$tmp/config.py" | head -300Repository: NVIDIA-NeMo/nemo-platform
Length of output: 4430
Place inference variables under [environment].env, not [agent].env. Harbor's AgentConfig has no env field. Update the checklist to match the schema example; otherwise the agent runtime will not receive these variables.
🧰 Tools
🪛 LanguageTool
[grammar] ~565-~565: Ensure spelling is correct
Context: ..._route("/exec", methods=["POST"])from FastMCP v2. Body:{"command": str, "cwd": str, "ti...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🪛 SkillSpector (2.5.1)
[error] 498: [AS2] MCP Config Access: Skill accesses MCP server configuration files (mcp.json). MCP configs contain server URLs, authentication tokens, and tool definitions — reading them allows the skill to discover and potentially abuse other tool integrations.
Remediation: Remove all code or instructions that read MCP configuration files (mcp.json). MCP server details should be managed by the agent runtime, not read by individual skills.
(Agent Snooping (AS2))
[error] 488: [E2] Env Variable Harvesting: Code accesses environment variables that may contain secrets (API keys, tokens). This is a common pattern for credential theft.
Remediation: Avoid reading sensitive env vars (API keys, tokens) unless strictly required. Use secrets managers or secure config. Never log or transmit credentials.
(Data Exfiltration (E2))
[warning] 352: [E5] Cloud Storage Exfiltration: Data is uploaded to cloud storage (S3 / GCS / Azure Blob). This may be a legitimate backup or exfiltration to an external bucket. Manual review is recommended.
Remediation: Verify the destination bucket is trusted and owned by you. Never upload credentials, secrets, or workspace contents to external or unverified cloud storage.
(Data Exfiltration (E5))
[warning] 365: [E5] Cloud Storage Exfiltration: Data is uploaded to cloud storage (S3 / GCS / Azure Blob). This may be a legitimate backup or exfiltration to an external bucket. Manual review is recommended.
Remediation: Verify the destination bucket is trusted and owned by you. Never upload credentials, secrets, or workspace contents to external or unverified cloud storage.
(Data Exfiltration (E5))
[error] 324: [PE3] Credential Access: Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.
Remediation: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths.
(Privilege Escalation (PE3))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/SKILL.md`
around lines 561 - 565, Update the task.toml checklist item to require
INFERENCE_API_KEY and INFERENCE_BASE_URL under the [environment].env section
instead of [agent].env. Keep the inference-only variable requirement and align
the wording with the Harbor AgentConfig schema example.
| ## D. Oracle `solve.sh` for MCP-sidecar tasks | ||
|
|
||
| For tasks where correctness is measured against environment state (not just an output file), the oracle must reconstruct the domain environment and replay the ground-truth actions to produce the expected state. | ||
|
|
||
| Structure: | ||
|
|
||
| ```bash | ||
| #!/bin/bash | ||
| set -euo pipefail | ||
|
|
||
| # 1. Write the oracle simulation spec | ||
| cat > /app/simulation.json <<'JSON' | ||
| { | ||
| "actions": [ | ||
| {"name": "<tool_name>", "arguments": {}, "requestor": "assistant"} | ||
| ], | ||
| "communicate_info": ["<expected message to user>"], | ||
| "initialization_data": null, | ||
| "initialization_actions": [] | ||
| } | ||
| JSON | ||
|
|
||
| # 2. Run a Python script that: | ||
| # - loads the domain environment | ||
| # - applies initialization state | ||
| # - replays oracle actions | ||
| # - writes the runtime state JSON that the verifier expects | ||
| python3 - <<'PYEOF' | ||
| import importlib, json, os, sys | ||
| from pathlib import Path | ||
|
|
||
| TAU2_ROOT = Path(os.environ.get("TAU2_BENCH_ROOT", "/opt/tau2-bench")) | ||
| sys.path.insert(0, str(TAU2_ROOT / "src")) | ||
|
|
||
| sim = json.loads(Path("/app/simulation.json").read_text()) | ||
| domain = "<domain>" | ||
| task_id = "<id>" | ||
|
|
||
| # Load domain environment | ||
| mod = importlib.import_module("tau2.domains.<domain>.environment") | ||
| environment = mod.get_environment(solo_mode=False) | ||
|
|
||
| # Replay oracle actions ... | ||
| # Write /logs/agent/<runtime_state>.json | ||
| PYEOF |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Run the oracle through the sidecar.
This template runs python3 in the main container. Line 563 states that MCP-sidecar tasks must execute task-environment commands through the sidecar /exec endpoint. Replace the local execution template with a sidecar request. Otherwise, the example cannot access /opt/tau2-bench in the sidecar-only environment.
🧰 Tools
🪛 SkillSpector (2.5.1)
[error] 498: [AS2] MCP Config Access: Skill accesses MCP server configuration files (mcp.json). MCP configs contain server URLs, authentication tokens, and tool definitions — reading them allows the skill to discover and potentially abuse other tool integrations.
Remediation: Remove all code or instructions that read MCP configuration files (mcp.json). MCP server details should be managed by the agent runtime, not read by individual skills.
(Agent Snooping (AS2))
[error] 488: [E2] Env Variable Harvesting: Code accesses environment variables that may contain secrets (API keys, tokens). This is a common pattern for credential theft.
Remediation: Avoid reading sensitive env vars (API keys, tokens) unless strictly required. Use secrets managers or secure config. Never log or transmit credentials.
(Data Exfiltration (E2))
[warning] 352: [E5] Cloud Storage Exfiltration: Data is uploaded to cloud storage (S3 / GCS / Azure Blob). This may be a legitimate backup or exfiltration to an external bucket. Manual review is recommended.
Remediation: Verify the destination bucket is trusted and owned by you. Never upload credentials, secrets, or workspace contents to external or unverified cloud storage.
(Data Exfiltration (E5))
[warning] 365: [E5] Cloud Storage Exfiltration: Data is uploaded to cloud storage (S3 / GCS / Azure Blob). This may be a legitimate backup or exfiltration to an external bucket. Manual review is recommended.
Remediation: Verify the destination bucket is trusted and owned by you. Never upload credentials, secrets, or workspace contents to external or unverified cloud storage.
(Data Exfiltration (E5))
[error] 324: [PE3] Credential Access: Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.
Remediation: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths.
(Privilege Escalation (PE3))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/SKILL.md`
around lines 783 - 827, Update the oracle template in the “Oracle solve.sh for
MCP-sidecar tasks” section so the Python replay script is executed through the
MCP sidecar’s /exec endpoint rather than locally in the main container. Preserve
the existing simulation setup and runtime-state generation, but route access to
/opt/tau2-bench and all task-environment commands through the sidecar execution
mechanism.
|
| @@ -0,0 +1,55 @@ | |||
| { | |||
There was a problem hiding this comment.
nvskills use a different format - it should be evals/evals.json
https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/skills/nemo-evaluator-plugin/evals/evals.json
We might have to add this new skill to the list here if we see nvskills being triggered for this new skill and fail: https://github.com/NVIDIA/skills/blob/main/components.d/nemo-platform.yml#L4
| - one <metadata>_by_split_pies.svg per metadata dimension | ||
| - metadata_split_counts.json | ||
| - index.html |
There was a problem hiding this comment.
would be nice to explain what svg and index.html represent
| ### 2c. File reference | ||
|
|
||
| ``` | ||
| tasks/<dataset>/<split>/<task_name>/ |
There was a problem hiding this comment.
fyi, evaluator discovery algorithm will not handle this dir structure. It expects tasks/<dataset>/<task_name>. https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/harbor_runtime.py#L1228
We will need to submit each split dir as a dataset_root. Also, there's no concept of a split in evaluator.
More background on this: https://nvidia.slack.com/archives/C069ZQ53HMW/p1785944861373829
2 participants
Summary
Adds a plugin-owned
convert-evals-to-harborskill that guides converting existing evaluation datasets and benchmarks into Harbor task trees that NeMo Evaluator can run viaHarborRunnerTarget.Changes
plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor/SKILL.mdwith Harbor conversion workflow, advanced MCP/sidecar patterns, and NeMo Evaluator runner pointersbalance_splits.pyandplot_metadata_splits.pyfor train/validation/test metadata balance checking and plottingtests.jsonfour-mode routing tests for skill discoveryType of Change
Quality Gates
Verification
Signed-off-by:traileruv run pre-commit run -apasses, or any blocked checks are identified belowTargeted validation:
uv run --frozen ruff check plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor— passeduv run --frozen ruff format --check plugins/nemo-evaluator/src/nemo_evaluator/skills/convert-evals-to-harbor— passeduv run --frozen pre-commit run --files <new skill files>— passed (ruff, ruff format, ty, copyright, merge-conflict, plugin-import hooks)uv run --frozen scripts/skill-test.py --root .— this skill's 10 routing tests pass; remaining 47 failures are pre-existing across other skillsuv run pre-commit run -awas not rerun repo-wide for this PR; file-scoped pre-commit for the new paths passedSummary by CodeRabbit