Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions docs/cli/reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6599,7 +6599,7 @@ nemo intake ingest atif create [OPTIONS]
* `--agent`: JSON string
* `--schema-version <CHOICE>`: (required) [possible values: ATIF-v1.0, ATIF-v1.1, ATIF-v1.2, ATIF-v1.3, ATIF-v1.4, ATIF-v1.5, ATIF-v1.6, ATIF-v1.7]
* `--continued-trajectory-ref`
* `--evaluation-context`: Evaluation context accepted by ingest endpoints (the canonical shape).`extra="ignore"` so a producer still sending retired keys (evaluation_sha, evaluation_run_id, metadata) keeps ingesting without error rather than being rejected. (JSON string)
* `--evaluation-context`: Identifies the Evaluation and optional test case associated with ingested telemetry. (JSON string)
* `--extra`: JSON string
* `--final-metrics`: JSON string
* `--notes`
Expand Down Expand Up @@ -6669,7 +6669,7 @@ nemo intake ingest chat-completions create [OPTIONS]
* `--cost-input-usd <FLOAT>`: Estimated input-token cost of this model call in USD.
* `--cost-output-usd <FLOAT>`: Estimated output-token cost of this model call in USD.
* `--cost-usd <FLOAT>`: Total estimated cost of this model call in USD. This matches ATIF step metrics; Intake stores it as semantic cost_total_usd on spans.
* `--evaluation-context`: Evaluation context accepted by ingest endpoints (the canonical shape).`extra="ignore"` so a producer still sending retired keys (evaluation_sha, evaluation_run_id, metadata) keeps ingesting without error rather than being rejected. (JSON string)
* `--evaluation-context`: Identifies the Evaluation and optional test case associated with ingested telemetry. (JSON string)
* `--provider`
* `--session-id`: Groups related chat-completions calls without forcing them into the same trace.
* `--trace-id`: Opt into joining an existing trace built via OTel or ATIF. This is not a grouping mechanism for chat-completions calls; use session_id to group related calls.
Expand Down Expand Up @@ -6925,10 +6925,11 @@ nemo intake spans list [OPTIONS]
JSON-only fields:
started_at: \{gte: str, lte: str}

Filter spans by session_id, trace_id, parent_span_id, project, evaluation_id, test_case_id, source, kind, status, model, tool_name, provider, agent_id, agent_name, and started_at. Every field takes one exact value, except started_at, which takes gte and lte.
Filter spans by session_id, trace_id, parent_span_id, project, evaluation_name, test_case_name, source, kind, status, model, tool_name, provider, agent_id, agent_name, and started_at. Every field takes one exact value, except started_at, which takes gte and lte.
* `--filter.agent-id`
* `--filter.agent-name`
* `--filter.evaluation-id`
* `--filter.evaluation-name`
* `--filter.kind`
* `--filter.model`
* `--filter.parent-span-id`
Expand All @@ -6938,6 +6939,7 @@ Filter spans by session_id, trace_id, parent_span_id, project, evaluation_id, te
* `--filter.source`
* `--filter.status`
* `--filter.test-case-id`
* `--filter.test-case-name`
* `--filter.tool-name`
* `--filter.trace-id`

Expand Down Expand Up @@ -7072,6 +7074,7 @@ Filter spans by the same fields as the span list endpoint, then group matching s
* `--filter.agent-id`
* `--filter.agent-name`
* `--filter.evaluation-id`
* `--filter.evaluation-name`
* `--filter.kind`
* `--filter.model`
* `--filter.parent-span-id`
Expand All @@ -7081,6 +7084,7 @@ Filter spans by the same fields as the span list endpoint, then group matching s
* `--filter.source`
* `--filter.status`
* `--filter.test-case-id`
* `--filter.test-case-name`
* `--filter.tool-name`
* `--filter.trace-id`

Expand Down Expand Up @@ -7139,12 +7143,14 @@ nemo intake traces list [OPTIONS]
JSON-only fields:
started_at: \{gte: str, lte: str}

Filter root-span-backed traces by id, session_id, root status, root span started_at, evaluation_id, and test_case_id.
Filter root-span-backed traces by id, session_id, root status, root span started_at, evaluation_name, and test_case_name.
* `--filter.id`
* `--filter.evaluation-id`
* `--filter.evaluation-name`
* `--filter.session-id`
* `--filter.status`
* `--filter.test-case-id`
* `--filter.test-case-name`

**Help:**

Expand Down
12 changes: 6 additions & 6 deletions docs/evaluator/experiments.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ At read time, each Evaluation is enriched with rollups derived from its sessions

| Rollup | Meaning |
|--------|---------|
| `test_case_count` | Number of distinct test cases (distinct non-empty `test_case_id` values). Sessions with no `test_case_id` do not count toward it or the rollups. |
| `test_case_count` | Number of distinct test cases (distinct non-empty `test_case_name` values). Sessions with no `test_case_name` do not count toward it or the rollups. |
| `cost_usd` | Cost aggregate across the Evaluation's sessions. |
| `latency_ms` | Latency aggregate across the Evaluation's sessions. |
| `tokens` | Average total tokens (input + output) per test case. |
Expand Down Expand Up @@ -233,9 +233,9 @@ session with the Evaluation's identity:

- For Agent Trajectory Interchange Format (ATIF) and chat-completions, add a **top-level**
`evaluation_context` object to the ingest payload
carrying `evaluation_id` (the Evaluation's **name**) and `test_case_id`.
carrying `evaluation_name` and `test_case_name`.
- For OpenTelemetry Protocol (OTLP), set the `nemo.evaluation.name` and
`nemo.test_case.id` root-span attributes.
`nemo.test_case.name` root-span attributes.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

The per-evaluator scores on the leaderboard come from **evaluator results** captured on those
sessions, either automatically from ATIF verifier rewards or explicitly through the evaluator-results
Expand All @@ -246,10 +246,10 @@ The `nemo-experiments-upload` skill walks this through end to end.

<Warning>

**`test_case_id` is required for a populated leaderboard.** A session tagged with only `evaluation_id`
**`test_case_name` is required for a populated leaderboard.** A session tagged with only `evaluation_name`
still ingests and appears in the Evaluation's session list, but it does not count toward
`test_case_count` or any rollup. The row therefore reads as all zeros, with tokens, model, and agent
blank too. Always send `test_case_id` alongside `evaluation_id`.
blank too. Always send `test_case_name` alongside `evaluation_name`.

</Warning>

Expand Down Expand Up @@ -501,7 +501,7 @@ window.

| Symptom | Cause and fix |
|---------|---------------|
| **Rows show zero metrics** | Three causes: no sessions have been ingested for that Evaluation yet; the sessions were ingested without `test_case_id`, so they do not count toward `test_case_count` or the rollups; or ClickHouse is unreachable. Confirm ingestion in Intake, confirm that sessions carry `test_case_id`, and run the explicit metric-sort health check. |
| **Rows show zero metrics** | Three causes: no sessions have been ingested for that Evaluation yet; the sessions were ingested without `test_case_name`, so they do not count toward `test_case_count` or the rollups; or ClickHouse is unreachable. Confirm ingestion in Intake, confirm that sessions carry `test_case_name`, and run the explicit metric-sort health check. |
| **A metric sort or filter returns `503`** | Rollups cannot be computed because ClickHouse is down. Retry after the read path is healthy, or use an entity-column sort. |
| **A list returns `413`** | The Experiment selected more than 1,000 Evaluations for an in-memory sort. Add filters to narrow the set. |
| **An Evaluation is not in the Experiment** | Confirm that you created it with the correct Experiment `id` in `experiment_ids` and that you are querying the correct workspace (`filter[experiment_id]=<experiment-id>`). |
Expand Down
99 changes: 72 additions & 27 deletions openapi/ga/individual/platform.openapi.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading