diff --git a/AGENTS.md b/AGENTS.md index fdc151b56a..55dd24ed98 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,7 +36,7 @@ User-facing skills in `packages/nemo_platform_ext/src/nemo_platform_ext/skills/` - `nemo-status`: read-only health dashboard. - `nemo-teardown`: guided shutdown with confirmation. -Plugin-owned skills under `plugins/*/src/*/skills/` handle their own routing for customization, guardrails, evaluations, optimization, data designer, anonymizer, auditor, and Experimentalist source/harness improvement. +Plugin-owned skills under `plugins/*/src/*/skills/` handle their own routing for customization, guardrails, evaluations, optimization, data designer, anonymizer, auditor, Experimentalist source/harness improvement, and Analyst telemetry analysis. ### Working in a sandboxed environment diff --git a/CLAUDE.md b/CLAUDE.md index 0b4eba76c7..2756949eb6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -34,7 +34,7 @@ User-facing skills in `packages/nemo_platform_ext/src/nemo_platform_ext/skills/` - `nemo-status`: read-only health dashboard. Run this before assuming the platform is up. - `nemo-teardown`: guided shutdown with confirmation. -Plugin-owned skills live under `plugins/*/src/*/skills/` and handle their own routing for customization, guardrails, evaluations, optimization, data designer, anonymizer, auditor, and Experimentalist source/harness improvement. +Plugin-owned skills live under `plugins/*/src/*/skills/` and handle their own routing for customization, guardrails, evaluations, optimization, data designer, anonymizer, auditor, Experimentalist source/harness improvement, and Analyst telemetry analysis. ### Working in a sandboxed coding-agent environment diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-skill-selection/SKILL.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-skill-selection/SKILL.md index 89ce30cc27..0847099974 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-skill-selection/SKILL.md +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-skill-selection/SKILL.md @@ -62,6 +62,7 @@ Match the user's intent to one downstream skill. Pick exactly one. | "status", "what is running", "platform health", "is the platform up", "what's deployed", "show me what's running" | `nemo-status` | Read-only dashboard: platform, agents, providers, models | | "shut down", "stop NeMo", "tear down", "clean up" | `nemo-teardown` | Stop the cluster (keep data, delete platform data, or full cleanup) | | "fine-tune", "customize the model", "train on my data", "SFT", "LoRA" | `nemo-customizer` | Model customization via installed customization contributor plugins (`nemo-customizer-plugin`). Requires plugin skills to be installed (`nemo skills install` / enabled-plugins). | +| "why does my agent keep failing", "analyze my agent's traces", "find recurring failure patterns", "generate insights for my agent" | `nemo-analyst` (plugin-owned, in `plugins/nemo-insights`) | Reads an agent's existing telemetry and files each recurring failure pattern as an Insight citing the traces that evidence it. Requires the Insights plugin; produces the Insight `nemo-experimentalist` acts on. | | "improve the agent's own code", "fix my agent harness", "candidate code change", "optimize from an Insight", "improve on train and validation datasets" | `nemo-experimentalist` (plugin-owned, in `plugins/nemo-experimentalist`) | Source/harness optimization: generate and validate candidate code changes against Harbor-compatible evaluation data. Requires the Experimentalist plugin; use after `agents analyst` has created an Insight, or with explicit datasets. | | "optimize my agent", "make it cheaper", "reduce latency", "smaller model", "switchyard", "routing split", "compare against a newer model" | `agents-optimize` (plugin-owned, in `plugins/nemo-agents`) | Cost / latency / quality optimization for a **deployed** agent. Routing splits, skill tuning, prompt tuning, new-model scans. | | "secure my agent", "harden my agent", "check for PII", "leaked secrets", "guardrail coverage" | `agents-secure` (plugin-owned, in `plugins/nemo-agents`) | Safety and security audit for a **deployed** agent. Guardrails, PII, secrets scan. | @@ -133,6 +134,7 @@ Plugin-owned skills: agents-secure safety and security audit for a deployed agent nemo-evaluator evaluation metrics, LLM-judge, benchmark jobs nemo-customizer fine-tuning of models + nemo-analyst analyze agent telemetry and file recurring problems as Insights nemo-experimentalist source/harness optimization from Insights or evaluation datasets guardrails content-safety middleware via virtual models auditor red-team vulnerability scanning (garak) diff --git a/plugins/nemo-insights/pyproject.toml b/plugins/nemo-insights/pyproject.toml index ad90163faa..0906a1fa53 100644 --- a/plugins/nemo-insights/pyproject.toml +++ b/plugins/nemo-insights/pyproject.toml @@ -35,6 +35,9 @@ insights-analysis = "nemo_insights_plugin.controller:InsightsAnalysisController" [project.entry-points."nemo.sdk"] insights = "nemo_insights_plugin.sdk:insights_sdk_resources" +[project.entry-points."nemo.skills"] +insights = "nemo_insights_plugin.skills:skills_dir" + [build-system] requires = ["hatchling"] build-backend = "hatchling.build" diff --git a/plugins/nemo-insights/src/nemo_insights_plugin/skills.py b/plugins/nemo-insights/src/nemo_insights_plugin/skills.py new file mode 100644 index 0000000000..ca1b5eae0a --- /dev/null +++ b/plugins/nemo-insights/src/nemo_insights_plugin/skills.py @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Skills directory exposure — registered under ``nemo.skills``. + +Returns the path to the ``skills/`` directory inside this package so the +platform can discover and load skill markdown files shipped with the plugin. +""" + +from __future__ import annotations + +from pathlib import Path + + +def skills_dir() -> Path: + """Return the directory containing this plugin's skills.""" + return Path(__file__).parent / "skills" diff --git a/plugins/nemo-insights/src/nemo_insights_plugin/skills/nemo-analyst/SKILL.md b/plugins/nemo-insights/src/nemo_insights_plugin/skills/nemo-analyst/SKILL.md new file mode 100644 index 0000000000..ada3262f72 --- /dev/null +++ b/plugins/nemo-insights/src/nemo_insights_plugin/skills/nemo-analyst/SKILL.md @@ -0,0 +1,163 @@ +--- +name: nemo-analyst +description: >- + Analyze an agent's production traces to find recurring failure patterns and + record each as an Insight. Surveys spans, evaluator scores, and user feedback + across many sessions, clusters similar failures, then files every finding as a + titled Insight carrying the trace IDs that evidence the problem. Answers why + an agent keeps failing, where it gets things wrong, and the recurring + problems hiding in production traces. Produces the Insight that the + Experimentalist later acts on. +triggers: + - nemo-analyst + - analyze my agent's traces + - why my agent keeps failing + - generate insights for my agent + - find recurring failure patterns + - run the analyst + - my agent keeps getting things wrong +not-for: + - nemo-experimentalist (use to act on an Insight and change the agent; this skill produces the Insight it consumes) + - nemo-intake (use to instrument an agent, ingest telemetry, or query raw spans; this skill interprets telemetry that already landed) + - nemo-experiments-upload (use to upload traces and evaluation results into Intake; this skill reads them back out) + - nemo-explore (use to design an agent that does not exist yet; this skill needs a running agent with traces) + - nemo-evaluator (use to author evaluations and metrics; this skill analyzes production behavior) +compatibility: >- + nemo-platform >= 0.1.0; requires the Insights plugin, a reachable platform + with Intake telemetry for the target agent, and a model the platform can call + on the Analyst's behalf. No Docker or datasets needed. +maturity: beta +license: Apache-2.0 +user-invocable: true +allowed-tools: [Bash, Read] +--- + +# NeMo Analyst + +Analyze an agent's behavior from its own telemetry and record what recurs as +Insights. + +## What it produces + +An Insight is a persistent, named description of one recurring problem, and it +is the unit of work the rest of the optimization loop runs on. Each carries: + +- `title` — a sentence naming the failure, such as "Retrieval drops relevant + context near the token limit" +- `description` — the failure mode, the tool or model call it affects, and the + conditions that trigger it +- `trace_refs` — the Intake trace IDs cited as evidence, so a developer can + audit the reasoning and build regression tests + +The Analyst targets at least three representative traces per Insight and appends +evidence to an existing Insight rather than filing a near-duplicate. It judges +behavior rather than status or scores, so it finds failures in sessions that +reported success and passed their evaluations. Two well-evidenced Insights are +worth more than ten vague ones, so a run that files nothing is a valid outcome. + +## Before running + +The Analyst reads telemetry; it cannot create it. Confirm all three: + +- The target agent already has traces in Intake. No traces means no Insights. +- The platform is reachable at `NMP_BASE_URL`. +- The Analyst has a model to run on. It is an LLM agent itself, and how that is + configured is changing, so let pre-flight tell you whether it is satisfied — + it names what is missing and how to set it. Don't reach for the + Experimentalist's configuration; that is a different contract. + +An `AGENT-SPEC.md` is optional but makes the Analyst materially better. It +carries the intent behind the agent — what it is for, its constraints, what +counts as success — none of which is recoverable from code or traces, so +without it the Analyst can only judge an agent against itself. + +## Pre-flight + +```bash +nemo agents analyst doctor +``` + +Only two results block a run: no usable model configured, and an +`optimizer.yaml` that is missing or unparseable — the second only if you intend +to run without `--agent`. Doctor takes no `--agent` flag, so it always checks +for a profile and always reports a red line when there is none; when you pass +`--agent`, that line is noise. Platform reachability and the workspace probe +only ever warn. + +## Run it + +```bash +nemo agents analyst run --agent --workspace +``` + +Add `--agent-spec AGENT-SPEC.md` to tell it what the agent is supposed to do, +and `--verbose` to stream its tool calls and reasoning to stderr. Expect several +minutes; it surveys many sessions before drilling into any of them. + +From an agent directory, an `optimizer.yaml` profile supplies `agent`, +`workspace`, and `agent_spec`, so the flags above become optional: + +```bash +nemo agents analyst run +``` + +The profile is discovered by walking up from the current directory. Only those +three fields are read from it; other keys belong to the Experimentalist and are +ignored. + +## Where Insights are stored + +Insights always go to the platform. `--insights-file-output` additionally +mirrors what the platform stored, platform IDs included, merging into that file +on each run; a mirror that cannot be written warns rather than failing the run. + +```bash +nemo agents analyst run --agent --insights-file-output .nemo-optimizer/insights.yaml +``` + +That path is what the Experimentalist reads by default, so it is the +conventional choice when handing off locally. + +## Verify + +Do not report success on an exit code. The run prints a line per operation — +`- created: [<insight-id>] (<n> trace refs)`, `- updated: <insight-id> +(<n> trace refs)`, or `- no insights created or updated`, which is a successful +run too. Read back by id whatever it says it wrote, and check each carries a +clear title, an actionable description, and non-empty `trace_refs`. Listing by +`?agent=` also returns earlier runs, so it attests the store, not this run: + +```bash +curl --fail-with-body \ + "$NMP_BASE_URL/apis/insights/v2/workspaces/<workspace>/insights/<insight-id>" +``` + +On an authenticated platform pass the token through curl's config, not argv +where any process on the host can read it: + +```bash +printf 'header = "Authorization: Bearer %s"' "$(nemo auth token)" | curl -K - <url> +``` + +Stored Insights also appear in Studio's optimizer view for the workspace. + +## When it finds nothing + +Besides a real "nothing worth filing", three things produce an empty result. +Scoping: the Analyst reads only what `--agent` and `--workspace` together +select, and `agent_name` is carried on agent-level spans, not on their model and +tool children. Volume: too few traces looks the same as a healthy agent. And +telemetry that captures only the shape of a run, spans without the inputs and +outputs, leaves nothing to judge however many spans there are. + +## Hand off + +Once an Insight exists, the Experimentalist acts on it: + +```bash +nemo agents experimentalist run +``` + +For the full data model, the Analyst's tool set, periodic analysis via +`nemo insights analysis enable`, and the rest of the loop, see +[Insight-Driven Optimization](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/docs/agents/insight-driven-optimization.mdx). diff --git a/plugins/nemo-insights/src/nemo_insights_plugin/skills/nemo-analyst/tests.json b/plugins/nemo-insights/src/nemo_insights_plugin/skills/nemo-analyst/tests.json new file mode 100644 index 0000000000..8598e65e9d --- /dev/null +++ b/plugins/nemo-insights/src/nemo_insights_plugin/skills/nemo-analyst/tests.json @@ -0,0 +1,65 @@ +{ + "skill": "nemo-analyst", + "tests": [ + { + "type": "explicit", + "prompt": "Use nemo-analyst to find out what my agent keeps getting wrong.", + "expected_skill": "nemo-analyst" + }, + { + "type": "explicit", + "prompt": "Run the nemo-analyst skill against my agent's telemetry.", + "expected_skill": "nemo-analyst" + }, + { + "type": "explicit", + "prompt": "Invoke nemo-analyst on the support agent's traces and record the insights it finds.", + "expected_skill": "nemo-analyst" + }, + { + "type": "implicit", + "prompt": "Analyze my agent's traces and tell me what it keeps getting wrong.", + "expected_skill": "nemo-analyst" + }, + { + "type": "implicit", + "prompt": "Why is my agent failing? Look at the traces and file insights.", + "expected_skill": "nemo-analyst" + }, + { + "type": "implicit", + "prompt": "Find recurring problems in my agent from production traces.", + "expected_skill": "nemo-analyst" + }, + { + "type": "implicit", + "prompt": "I want insights about my agent's failures from its spans and user feedback.", + "expected_skill": "nemo-analyst" + }, + { + "type": "implicit", + "prompt": "Run the analyst to find failure patterns in recent telemetry.", + "expected_skill": "nemo-analyst" + }, + { + "type": "negative-control", + "prompt": "Optimize my deployed agent's model routing to cut cost and latency.", + "expected_skill_not": "nemo-analyst" + }, + { + "type": "negative-control", + "prompt": "Send my agent's OTLP traces into Intake and check ClickHouse is ready.", + "expected_skill_not": "nemo-analyst" + }, + { + "type": "negative-control", + "prompt": "Design a new customer-support agent and write its specification.", + "expected_skill_not": "nemo-analyst" + }, + { + "type": "negative-control", + "prompt": "Set up a Postgres database with seed data for local development.", + "expected_skill_not": "nemo-analyst" + } + ] +} diff --git a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-skill-selection/SKILL.md b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-skill-selection/SKILL.md index 89ce30cc27..0847099974 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-skill-selection/SKILL.md +++ b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-skill-selection/SKILL.md @@ -62,6 +62,7 @@ Match the user's intent to one downstream skill. Pick exactly one. | "status", "what is running", "platform health", "is the platform up", "what's deployed", "show me what's running" | `nemo-status` | Read-only dashboard: platform, agents, providers, models | | "shut down", "stop NeMo", "tear down", "clean up" | `nemo-teardown` | Stop the cluster (keep data, delete platform data, or full cleanup) | | "fine-tune", "customize the model", "train on my data", "SFT", "LoRA" | `nemo-customizer` | Model customization via installed customization contributor plugins (`nemo-customizer-plugin`). Requires plugin skills to be installed (`nemo skills install` / enabled-plugins). | +| "why does my agent keep failing", "analyze my agent's traces", "find recurring failure patterns", "generate insights for my agent" | `nemo-analyst` (plugin-owned, in `plugins/nemo-insights`) | Reads an agent's existing telemetry and files each recurring failure pattern as an Insight citing the traces that evidence it. Requires the Insights plugin; produces the Insight `nemo-experimentalist` acts on. | | "improve the agent's own code", "fix my agent harness", "candidate code change", "optimize from an Insight", "improve on train and validation datasets" | `nemo-experimentalist` (plugin-owned, in `plugins/nemo-experimentalist`) | Source/harness optimization: generate and validate candidate code changes against Harbor-compatible evaluation data. Requires the Experimentalist plugin; use after `agents analyst` has created an Insight, or with explicit datasets. | | "optimize my agent", "make it cheaper", "reduce latency", "smaller model", "switchyard", "routing split", "compare against a newer model" | `agents-optimize` (plugin-owned, in `plugins/nemo-agents`) | Cost / latency / quality optimization for a **deployed** agent. Routing splits, skill tuning, prompt tuning, new-model scans. | | "secure my agent", "harden my agent", "check for PII", "leaked secrets", "guardrail coverage" | `agents-secure` (plugin-owned, in `plugins/nemo-agents`) | Safety and security audit for a **deployed** agent. Guardrails, PII, secrets scan. | @@ -133,6 +134,7 @@ Plugin-owned skills: agents-secure safety and security audit for a deployed agent nemo-evaluator evaluation metrics, LLM-judge, benchmark jobs nemo-customizer fine-tuning of models + nemo-analyst analyze agent telemetry and file recurring problems as Insights nemo-experimentalist source/harness optimization from Insights or evaluation datasets guardrails content-safety middleware via virtual models auditor red-team vulnerability scanning (garak)