feat(evaluator): aggregate agent-eval results natively and import Gym's own

[SandyChapman]

Closes AALGO-434.

Why

Agent-eval reported a mean and little else, and a runner's own aggregations were discarded entirely — so a Gym run's headline numbers (pass@k, per-environment metrics) were visible in Gym's output files but nowhere in the eval result.

Native aggregation

• pass@k per task via the unbiased estimator (Chen et al. 2021), for every score-like metric output. An attempt passes on full credit — deliberately not configurable, since it's a reporting-time interpretation and a tunable version would produce numbers that look comparable across runs but aren't.
• Percentiles on every aggregate, reusing the deterministic-metric helper (now public as compute_percentiles) so both paths report distributions the same way.
• median added to AggregateScoreBase, populated natively from p50, so the field means the same thing whether a score was computed here or imported from a backend that reports a median without a full distribution.
• Both standard-deviation conventions, named explicitly. std_dev/variance remain population (÷n); sample_std_dev/sample_variance are new (÷n−1, None when n<2). Gym computes its spread with pandas (ddof=1), so naming both lets the two sets of numbers coexist without either silently changing meaning. No existing value changes.

Importing Gym's own numbers

RunAggregationsProvider is a single method — run_aggregate_scores() — returning typed aggregates that merge into summary.scores. There is no parallel opaque passthrough: Gym already writes its raw rollouts_aggregate_metrics.json inside the run's work dir, so a second copy on the result would duplicate a file already in the bundle and re-introduce the untyped dict[str, Any] bag that #1013 removes.

Names are runner.gym.<metric> — namespaced by runner, not agent, since each run uses a single agent and what a reader needs to know is which backend produced the number. Falls back to runner.gym.<agent>.<metric> only if a run produced several. Gym's reward is skipped: the SDK already scores it natively as gym_reward.reward from the same rollouts.

The part most worth reviewing

Imports read agent_metrics, not key_metrics. This matters: key_metrics is a subset of agent_metrics chosen by the resources-server, and Gym's default get_key_metrics keeps only the mean/* entries — so max/min/median/std never appear there, and sourcing from it would degrade every distribution into a lone runner.gym.mean/<name> scalar. test_imports_read_agent_metrics_not_the_key_metrics_subset pins this.

Gym flattens each distribution into <stat>/<metric> keys. Its describe_dataframe emits mean/max/min/median/std/histogram together, and prepare_for_serialization strips histogram before writing — so the family in the file is exactly those five, and re-assembly requires all five. That guard matters because 36 of Gym's ~97 resources-servers override compute_metrics, emitting keys in their own shapes (arena_elo/score, easy/pass@1/accuracy), and a server is free to define a metric literally named mean. A partial match stays as standalone scalars rather than being renamed into a statistic of a distribution that never existed.

test_nothing_numeric_is_dropped_from_a_custom_environment_payload pins the invariant: importing is a renaming, never a filter. Only reward (redundant by construction) and non-numeric values may disappear.

New AggregateScalarScore

A single pre-computed value with no underlying sample. Distinct from AggregateRangeScore so a reader can tell "this is the whole story" from "this summarizes count samples", instead of seeing a range score with a suspicious count of 1.

count is now int | None, and imported aggregates carry None — Gym reports statistics without the n behind them, and 0 would assert that nothing was evaluated (besides being a division hazard). The dashboard and example render None as an em dash while still showing a genuine 0.

Also

• Dashboard gains Value / Median / Std dev columns and renders scalars by value.
• New examples/gym/inspect_results.py — companion to run_gym_eval.py, showing how to reach each kind of result, with lift-and-paste accessors.

Known scope boundary

Gym's group_level_metrics (its per-task aggregation) is not imported. summary.scores is run-level, so it has no home there yet — the SDK computes per-task groupings internally for pass@k but doesn't surface that granularity. Native per-task rollups plus importing Gym's into them is tracked separately; the raw data remains in Gym's own file inside the bundle.

Verification

• Full run: 1883 passed / 25 failed, all 25 the pre-existing ragas + live-Fabric baseline (verified: zero failures outside it).
• Ruff clean; ty at exactly the main baseline (198); make vendor run with no drift.

Summary by CodeRabbit

• New Features

  • Evaluation summaries and dashboards now include median, variance, standard deviation, percentiles, and improved count handling.
  • Gym results can include runner-provided aggregate metrics.
  • Added pass@k reporting for eligible task-level metrics.
  • Added a command-line example for inspecting saved results and per-task outcomes.

• Bug Fixes

  • Improved scalar metric display and handling of incomplete or malformed Gym aggregate data.
  • Corrected pass@k calculations across supported metric types.

• Documentation

  • Updated Gym example instructions for running evaluations and inspecting saved results.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Changes

Evaluation results and aggregation

Layer / File(s)	Summary
Aggregate score contracts and statistics packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py, packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/metrics/aggregation.py, packages/nemo_evaluator_sdk/tests/test_api.py	Adds scalar aggregate scores, median fields, sample statistics, nullable counts, public percentile computation, and NaN-safe serialization.
Gym aggregate import and runner integration packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py, packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trials.py, packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py, packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/scores.py, packages/nemo_evaluator_sdk/tests/agent_eval/test_gym_aggregate_scores.py, packages/nemo_evaluator_sdk/tests/agent_eval/test_runner_aggregations.py	Reads Gym aggregate sidecars, converts numeric metrics to typed scores, validates namespaces, and merges runner scores into evaluation summaries.
Pass@k aggregation packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/results.py, packages/nemo_evaluator_sdk/tests/agent_eval/test_pass_at_k.py	Adds unbiased task-level pass@k aggregation for eligible score outputs and distinguishes failed trials from unmeasured metrics.
Result presentation and persisted bundle inspection packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/dashboard.py, packages/nemo_evaluator_sdk/examples/gym/inspect_results.py, packages/nemo_evaluator_sdk/examples/gym/README.md, packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py, packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.py, packages/nemo_evaluator_sdk/tests/agent_eval/test_dashboard.py	Updates dashboard aggregate columns and adds a CLI tool for inspecting persisted Gym evaluation results.

Sequence Diagram(s)

sequenceDiagram
  participant GymAgentTaskRunner
  participant GymAggregateMetrics
  participant AgentEvaluator
  participant AgentEvalSummary
  participant Dashboard
  GymAgentTaskRunner->>GymAggregateMetrics: Read aggregate-metrics sidecar
  GymAggregateMetrics-->>GymAgentTaskRunner: Return agent_metrics
  GymAgentTaskRunner->>AgentEvaluator: Return run_aggregate_scores()
  AgentEvaluator->>AgentEvalSummary: Pass extra_scores to from_scores()
  AgentEvalSummary-->>Dashboard: Provide merged aggregate statistics
  Dashboard-->>Dashboard: Render value, median, std dev, count, and NaN

Loading

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#986: Extends the GymAgentTaskRunner introduced by this PR with run-level aggregate score collection.
• NVIDIA-NeMo/nemo-platform#1013: Shares evaluator, persistence, results, and Gym runtime code while implementing a separate result aggregation and metadata flow.

Suggested reviewers: arpitsardhana, ngoncharenko, aleckhoury

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main changes: native agent-eval aggregation and Gym metric import.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch aalgo-434-gym-aggregations/schapman

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 6

🧹 Nitpick comments (3)

packages/nemo_evaluator_sdk/tests/agent_eval/test_pass_at_k.py (2)

73-95: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Cover boolean score outputs.

_scorelike_outputs in packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/results.py, Lines 186-202, accepts continuous and boolean schemas. This test covers continuous and label outputs only. Add a boolean metric with True and False values and assert its pass@k results.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_pass_at_k.py` around lines
73 - 95, The test test_task_pass_at_k_gated_and_uniform_across_metric_types
should include a boolean-output metric with True/False scores, add corresponding
task scores, and assert its pass@1 and pass@2 values alongside the existing
continuous metrics. Ensure the boolean metric is treated as pass@k-eligible
while the label metric remains excluded.

---

73-95: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Use unequal attempt counts to verify per-task weighting.

Both tasks have two attempts. A pooled calculation produces the same pass@1 and pass@2 values as the required per-task calculation. Add unequal trial counts, such as two versus four, and assert the mean of the task-level estimates.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_pass_at_k.py` around lines
73 - 95, Update test_task_pass_at_k_gated_and_uniform_across_metric_types to use
unequal attempt counts between t1 and t2, such as two versus four, by adding the
corresponding scores for the task with four attempts. Recalculate and assert
pass@1 and pass@2 from the unweighted mean of each task’s pass@k estimate,
ensuring the test distinguishes per-task weighting from pooled attempt weighting
while keeping both reward metrics and the label exclusion covered.

packages/nemo_evaluator_sdk/tests/agent_eval/test_gym_aggregate_scores.py (1)

114-114: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Exclude bools by type, not by value.

value is not True only filters the literal True. A False flag in the fixture would pass isinstance(value, (int, float)) and enter numeric_keys, then fail the assertion for the wrong reason. _as_float rejects all bools, so mirror that here.

♻️ Proposed fix

-    numeric_keys = {key for key, value in key_metrics.items() if isinstance(value, (int, float)) and value is not True}
+    numeric_keys = {
+        key
+        for key, value in key_metrics.items()
+        if isinstance(value, (int, float)) and not isinstance(value, bool)
+    }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_gym_aggregate_scores.py` at
line 114, Update the numeric key filtering in the test’s key-metrics aggregation
to exclude booleans by type, not by checking only whether the value is True.
Mirror _as_float’s behavior so both True and False are excluded while numeric
int and float values remain included.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/nemo_evaluator_sdk/examples/gym/inspect_results.py`:
- Around line 15-18: Update the usage text in inspect_results to show the script
being run through uv instead of invoking python directly. Keep the existing
repository-root context and bundle argument, but change the documented command
in the inspect_results example to use the uv run entrypoint so it matches the
coding guidelines.

In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/results.py`:
- Around line 23-25: Update the pass@k filtering in the results logic around
_PASS_AT_K_VALUE_SCHEMAS to use issubclass() against the allowed value-schema
classes instead of exact membership testing, while preserving the existing
eligibility behavior for ContinuousScore and BooleanValue.

In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py`:
- Around line 766-773: Update the aggregate-metrics parsing logic around the
parsed-list handling to validate each entry and its agent_ref mapping before
accessing agent_ref["name"]. Return None, or otherwise follow the existing
unparseable-file behavior, when any entry has a missing or invalid agent
reference, preventing malformed Gym data from propagating exceptions through
run_tasks while preserving valid aggregations.
- Around line 366-373: Update the docstring on run_aggregate_scores to describe
the Gym field actually consumed by _aggregate_scores_from_gym, namely
agent_metrics rather than key_metrics. Keep the existing namespace and
reward-skip explanation intact, and align the summary line with the behavior
documented by _aggregate_scores_from_gym and
test_imports_read_agent_metrics_not_the_key_metrics_subset.

In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/metrics/aggregation.py`:
- Around line 403-404: Update aggregate_metrics in
packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/metrics/aggregation.py at
lines 403-404 to compute median from valid score values and pass it to
AggregateScoreBase, and at lines 430-431 pass median=percentiles.p50 for range
scores so native aggregates expose the correct median.

In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py`:
- Around line 186-193: Update the result model’s serialize_nan configuration to
include both sample_variance and sample_stddev, ensuring NaN values serialize as
the string "NaN". Add unit tests covering float("nan") for each field and verify
the serialized output.

---

Nitpick comments:
In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_gym_aggregate_scores.py`:
- Line 114: Update the numeric key filtering in the test’s key-metrics
aggregation to exclude booleans by type, not by checking only whether the value
is True. Mirror _as_float’s behavior so both True and False are excluded while
numeric int and float values remain included.

In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_pass_at_k.py`:
- Around line 73-95: The test
test_task_pass_at_k_gated_and_uniform_across_metric_types should include a
boolean-output metric with True/False scores, add corresponding task scores, and
assert its pass@1 and pass@2 values alongside the existing continuous metrics.
Ensure the boolean metric is treated as pass@k-eligible while the label metric
remains excluded.
- Around line 73-95: Update
test_task_pass_at_k_gated_and_uniform_across_metric_types to use unequal attempt
counts between t1 and t2, such as two versus four, by adding the corresponding
scores for the task with four attempts. Recalculate and assert pass@1 and pass@2
from the unweighted mean of each task’s pass@k estimate, ensuring the test
distinguishes per-task weighting from pooled attempt weighting while keeping
both reward metrics and the label exclusion covered.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 58e41c8d-be79-4a0b-a562-f0404d7e75fe

📥 Commits

Reviewing files that changed from the base of the PR and between 1fa5b1c and 6a7da9f.

⛔ Files ignored due to path filters (8)

• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/dashboard.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/evaluator.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/persistence.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/results.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/trials.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/metrics/aggregation.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/values/results.py is excluded by !sdk/**

📒 Files selected for processing (14)

• packages/nemo_evaluator_sdk/examples/gym/inspect_results.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/dashboard.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/results.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trials.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/metrics/aggregation.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_dashboard.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_gym_aggregate_scores.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_pass_at_k.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_runner_aggregations.py
• packages/nemo_evaluator_sdk/tests/test_api.py

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	31438/40061	78.5%	63.0%
Integration Tests	18307/38013	48.2%	20.8%

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[ngoncharenko]

Three inline findings from the merge-base review.

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

packages/nemo_evaluator_sdk/tests/agent_eval/test_pass_at_k.py (1)

96-117: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add boolean score eligibility coverage.

This test covers continuous scores and labels, but not boolean scores. A regression that excludes boolean schemas from pass@k remains undetected. Add True and False outputs and assert pass@1 and pass@2.

As per coding guidelines, prefer writing unit tests when verifying solutions instead of executing ad hoc Python snippets.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_pass_at_k.py` around lines
96 - 117, Extend test_task_pass_at_k_gated_and_uniform_across_metric_types with
a boolean score metric and True/False outputs across the existing tasks and
attempts. Include assertions verifying the boolean metric produces the expected
pass@1 and pass@2 values, ensuring boolean schemas are eligible for pass@k
alongside continuous metrics while labels remain excluded.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/nemo_evaluator_sdk/examples/gym/README.md`:
- Around line 55-67: Update the “Read the results” section in the README to
include a tested Python SDK example in a tab set alongside the existing CLI
commands. Show equivalent bundle-reading and result-access usage through the
SDK, while preserving the current CLI workflow and explanatory context.

In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/scores.py`:
- Line 28: Replace the generic TRIAL_STATUS_DETAIL value with an SDK-reserved
discriminator key, and update the trial-failure detection and diagnostic-writing
paths that use it consistently. Add a test covering a metric failure containing
“trial_status” and verify is_trial_failure() remains false so the unusable
measurement is excluded from pass@k.

---

Nitpick comments:
In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_pass_at_k.py`:
- Around line 96-117: Extend
test_task_pass_at_k_gated_and_uniform_across_metric_types with a boolean score
metric and True/False outputs across the existing tasks and attempts. Include
assertions verifying the boolean metric produces the expected pass@1 and pass@2
values, ensuring boolean schemas are eligible for pass@k alongside continuous
metrics while labels remain excluded.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 23bd9dfd-c87c-49db-8c05-b40afd6a7296

📥 Commits

Reviewing files that changed from the base of the PR and between c71ca67 and a9b74f7.

⛔ Files ignored due to path filters (9)

• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/dashboard.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/evaluator.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/persistence.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/results.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/scores.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/trials.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/metrics/aggregation.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/values/results.py is excluded by !sdk/**

📒 Files selected for processing (17)

• packages/nemo_evaluator_sdk/examples/gym/README.md
• packages/nemo_evaluator_sdk/examples/gym/inspect_results.py
• packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/dashboard.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/results.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/scores.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trials.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/metrics/aggregation.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_dashboard.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_gym_aggregate_scores.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_pass_at_k.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_runner_aggregations.py
• packages/nemo_evaluator_sdk/tests/test_api.py

🚧 Files skipped from review as they are similar to previous changes (8)

• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_gym_aggregate_scores.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/metrics/aggregation.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/results.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/dashboard.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trials.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/nemo_evaluator_sdk/examples/gym/inspect_results.py`:
- Around line 76-77: Update the score filtering logic around score.metric_type
and score.status to preserve failed trial attempts. Reuse the existing
is_trial_failure classification used by pass@k, excluding only failures
representing unmeasured metrics while retaining trial failures for task-solved
evaluation.

In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py`:
- Around line 761-765: Update the sidecar parsing error handler in the Gym
aggregate metrics flow to also catch UnicodeDecodeError from Path.read_text,
alongside the existing JSONDecodeError and OSError handling. Preserve the
warning and None return so invalid UTF-8 sidecars are skipped without aborting
run_tasks.
- Around line 840-866: Update the incomplete-family fallback in the metric
aggregation loop to skip the `reward` metric before copying its statistics into
`scalars`. Preserve the existing standalone scalar behavior for other incomplete
families and leave complete-family handling unchanged.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 601114d8-52cc-44b2-8094-04bb6d9cda73

📥 Commits

Reviewing files that changed from the base of the PR and between c71ca67 and c74cbfb.

⛔ Files ignored due to path filters (9)

• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/dashboard.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/evaluator.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/persistence.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/results.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/gym_runtime.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/scores.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/trials.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/metrics/aggregation.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/values/results.py is excluded by !sdk/**

📒 Files selected for processing (17)

• packages/nemo_evaluator_sdk/examples/gym/README.md
• packages/nemo_evaluator_sdk/examples/gym/inspect_results.py
• packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/dashboard.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/results.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/gym_runtime.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/scores.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trials.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/metrics/aggregation.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_dashboard.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_gym_aggregate_scores.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_pass_at_k.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_runner_aggregations.py
• packages/nemo_evaluator_sdk/tests/test_api.py

🚧 Files skipped from review as they are similar to previous changes (13)

• packages/nemo_evaluator_sdk/examples/gym/README.md
• packages/nemo_evaluator_sdk/examples/gym/run_gym_eval.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/trials.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_gym_aggregate_scores.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/dashboard.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_dashboard.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_runner_aggregations.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/metrics/aggregation.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/results.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/results.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_pass_at_k.py