From 42f574a67e9983caffc823008cd1d83d3c7dd231 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Wed, 5 Aug 2026 22:44:55 +0000 Subject: [PATCH 1/9] feat(skills): add nemo-analyst workflow Add the platform-bundled nemo-analyst skill so coding agents have a reliable route to `nemo agents analyst run` and the minimum correct operating knowledge to use it. The body stays lean and defers depth to docs/agents/insight-driven-optimization.mdx rather than restating it. It covers the Insight shape, the credential distinction from the Experimentalist (the Analyst reads only INFERENCE_API_KEY), the doctor pre-flight, the run, where Insights are stored, and a verification step. Persistence is documented as it behaves after #1054: Insights always go to the platform, and --insights-file-output mirrors what was stored. Routing tests follow the explicit/implicit/negative-control convention already used by nemo-explore. Verified that all six analyst-intent prompts resolve to nemo-analyst, that the skill introduces no new failures elsewhere in the catalog, and that it wins no ties by sort order. Signed-off-by: Cursor Agent Co-authored-by: Aditya Pandey Signed-off-by: Aditya Pandey --- .../skills/nemo-analyst/SKILL.md | 159 ++++++++++++++++++ .../skills/nemo-analyst/tests.json | 65 +++++++ 2 files changed, 224 insertions(+) create mode 100644 packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md create mode 100644 packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/tests.json diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md new file mode 100644 index 0000000000..ac4d86a40c --- /dev/null +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md @@ -0,0 +1,159 @@ +--- +name: nemo-analyst +description: >- + Analyze an agent's production traces to find recurring failure patterns and + record each as an Insight. Surveys spans, evaluator scores, and user feedback + across many sessions, clusters similar failures, then files every finding as a + titled Insight carrying the trace IDs that evidence the problem. Answers why + an agent keeps failing, where it gets things wrong, and the recurring + problems hiding in production traces. Produces the Insight that + nemo-experimentalist consumes. +triggers: + - nemo-analyst + - analyze my agent's traces + - why my agent keeps failing + - generate insights for my agent + - find recurring failure patterns + - run the analyst + - my agent keeps getting wrong +not-for: + - nemo-experimentalist (use to act on an Insight and change the agent; this skill produces the Insight it consumes) + - nemo-intake (use to instrument an agent, ingest telemetry, or query raw spans; this skill interprets telemetry that already landed) + - nemo-experiments-upload (use to upload traces and evaluation results into Intake; this skill reads them back out) + - nemo-explore (use to design an agent that does not exist yet; this skill needs a running agent with traces) + - nemo-evaluator (use to author evaluations and metrics; this skill analyzes production behavior) +compatibility: >- + nemo-platform >= 0.1.0; requires the Insights plugin, a reachable platform + with Intake telemetry for the target agent, and INFERENCE_API_KEY for NVIDIA + Inference Gateway access. No Docker or datasets needed. +maturity: beta +license: Apache-2.0 +user-invocable: true +allowed-tools: [Bash, Read] +--- + +# NeMo Analyst + +Find what an agent keeps getting wrong, from its own telemetry, and record it as +an Insight. + +## What it produces + +An Insight is a persistent, named description of one recurring problem, and it +is the unit of work the rest of the optimization loop runs on. Each carries: + +- `title` — a sentence naming the failure, such as "Retrieval drops relevant + context near the token limit" +- `description` — the failure mode, the tool or model call it affects, and the + conditions that trigger it +- `trace_refs` — the Intake trace IDs cited as evidence, so a developer can + audit the reasoning and build regression tests + +The Analyst targets at least three representative traces per Insight and appends +evidence to an existing Insight rather than filing a near-duplicate. Two +well-evidenced Insights are worth more than ten vague ones, so a run that files +nothing is a valid outcome. + +## Before running + +The Analyst reads telemetry; it cannot create it. Confirm all three: + +- The target agent already has traces in Intake. No traces means no Insights. +- `INFERENCE_API_KEY` is set. The Analyst runs on Claude Opus 4.8 through the + NVIDIA Inference Gateway and reads **only** this variable. It does not use the + `NEMO_EXPERIMENTALIST_MODELS_*` tiers the Experimentalist needs, so do not + copy that configuration here. +- The platform is reachable at `NMP_BASE_URL`. + +An `AGENT-SPEC.md` is optional but makes the Analyst materially better: given +the agent's intended behavior it can flag divergence from that contract, not +just outright errors. + +## Pre-flight + +```bash +nemo agents analyst doctor +``` + +Two failures block a run and must be fixed first: an `optimizer.yaml` that is +missing or unparseable when no `--agent` is passed, and an unset +`INFERENCE_API_KEY`. Platform reachability and the workspace probe are advisory +— they warn, and the run still proceeds. + +## Run it + +```bash +nemo agents analyst run \ + --agent \ + --workspace \ + --base-url "$NMP_BASE_URL" +``` + +Add `--agent-spec AGENT-SPEC.md` to enable divergence checking, and `--verbose` +to stream the Analyst's tool calls and reasoning to stderr while it works. +Expect a run to take several minutes; it surveys many sessions before drilling +into any of them. + +From an agent directory, an `optimizer.yaml` profile supplies `agent`, +`workspace`, and `agent_spec`, so the flags above become optional: + +```bash +nemo agents analyst run +``` + +The profile is discovered by walking up from the current directory. Only those +three fields are read from it; other keys belong to the Experimentalist and are +ignored. + +## Where Insights are stored + +Insights always go to the platform. To keep a local copy, pass +`--insights-file-output`, which mirrors what the platform stored, platform IDs +included, and merges into that file on each run: + +```bash +nemo agents analyst run --agent --insights-file-output .nemo-optimizer/insights.yaml +``` + +That path is the one the Experimentalist reads by default, so it is the +conventional choice when handing off locally. A mirror that cannot be written +degrades to a warning rather than failing the run, because the platform is the +source of truth. + +## Verify + +Do not report success without checking that Insights actually landed. There is +no CLI verb for this yet, so read the Insights API directly: + +```bash +curl --fail-with-body \ + -H "Authorization: Bearer $(nemo auth token)" \ + "$NMP_BASE_URL/apis/insights/v2/workspaces//insights?agent=&page=1&page_size=20" +``` + +Drop the `Authorization` header on a local platform with authentication +disabled. A successful run leaves at least one Insight for the agent, each with +a clear title, an actionable description, and non-empty `trace_refs`. Stored +Insights also appear in Studio under the workspace's optimizer view. If you +passed `--insights-file-output`, read the mirror back and confirm it is +non-empty. + +## When it finds nothing + +An empty result is either a real "nothing worth filing" or one of two setup +problems. Check the agent name matches the `agent_name` on the spans exactly, +since the Analyst scopes everything through spans, then confirm telemetry is +actually flowing for that agent and workspace. Too few traces produces the same +empty result as a healthy agent. + +## Hand off + +Once an Insight exists, the Experimentalist acts on it: + +```bash +nemo agents experimentalist run +``` + +For the full data model, the Analyst's tool set, periodic analysis via +`nemo insights analysis enable`, and the rest of the loop, see +[Insight-Driven Optimization](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/docs/agents/insight-driven-optimization.mdx). diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/tests.json b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/tests.json new file mode 100644 index 0000000000..8598e65e9d --- /dev/null +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/tests.json @@ -0,0 +1,65 @@ +{ + "skill": "nemo-analyst", + "tests": [ + { + "type": "explicit", + "prompt": "Use nemo-analyst to find out what my agent keeps getting wrong.", + "expected_skill": "nemo-analyst" + }, + { + "type": "explicit", + "prompt": "Run the nemo-analyst skill against my agent's telemetry.", + "expected_skill": "nemo-analyst" + }, + { + "type": "explicit", + "prompt": "Invoke nemo-analyst on the support agent's traces and record the insights it finds.", + "expected_skill": "nemo-analyst" + }, + { + "type": "implicit", + "prompt": "Analyze my agent's traces and tell me what it keeps getting wrong.", + "expected_skill": "nemo-analyst" + }, + { + "type": "implicit", + "prompt": "Why is my agent failing? Look at the traces and file insights.", + "expected_skill": "nemo-analyst" + }, + { + "type": "implicit", + "prompt": "Find recurring problems in my agent from production traces.", + "expected_skill": "nemo-analyst" + }, + { + "type": "implicit", + "prompt": "I want insights about my agent's failures from its spans and user feedback.", + "expected_skill": "nemo-analyst" + }, + { + "type": "implicit", + "prompt": "Run the analyst to find failure patterns in recent telemetry.", + "expected_skill": "nemo-analyst" + }, + { + "type": "negative-control", + "prompt": "Optimize my deployed agent's model routing to cut cost and latency.", + "expected_skill_not": "nemo-analyst" + }, + { + "type": "negative-control", + "prompt": "Send my agent's OTLP traces into Intake and check ClickHouse is ready.", + "expected_skill_not": "nemo-analyst" + }, + { + "type": "negative-control", + "prompt": "Design a new customer-support agent and write its specification.", + "expected_skill_not": "nemo-analyst" + }, + { + "type": "negative-control", + "prompt": "Set up a Postgres database with seed data for local development.", + "expected_skill_not": "nemo-analyst" + } + ] +} From d2a948fe48452eae7e1e9bfd3f6b3f84cae0151a Mon Sep 17 00:00:00 2001 From: Aditya Pandey Date: Thu, 6 Aug 2026 16:02:48 -0600 Subject: [PATCH 2/9] feat(skills): tighten nemo-analyst from journey findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drove the skill end to end in a container against a real LangGraph agent's telemetry. It carried the workflow, so this sharpens what it left the driver to work out rather than adding to it — net zero lines, paid for by cutting redundancy from the storage and verify sections. - `analyst doctor` takes no `--agent` flag, so it always reports a red profile line. Say that passing `--agent` makes that line noise instead of leaving the reader to reason past a red mark, or to write an `optimizer.yaml` it does not need. - `agent_name` is carried on agent-level spans, not on their model and tool children, so the old "matches the `agent_name` on the spans" pointed at a query that undercounts. - Every span came back `status=success` while the agent was badly broken, so state that the Analyst judges behavior rather than status or scores. - Telemetry that captures only the shape of a run is a third cause of an empty result, alongside scoping and volume. Signed-off-by: Aditya Pandey --- .../skills/nemo-analyst/SKILL.md | 58 +++++++++---------- 1 file changed, 29 insertions(+), 29 deletions(-) diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md index ac4d86a40c..4a927afaa4 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md @@ -50,9 +50,10 @@ is the unit of work the rest of the optimization loop runs on. Each carries: audit the reasoning and build regression tests The Analyst targets at least three representative traces per Insight and appends -evidence to an existing Insight rather than filing a near-duplicate. Two -well-evidenced Insights are worth more than ten vague ones, so a run that files -nothing is a valid outcome. +evidence to an existing Insight rather than filing a near-duplicate. It judges +behavior rather than status or scores, so it finds failures in sessions that +reported success and passed their evaluations. Two well-evidenced Insights are +worth more than ten vague ones, so a run that files nothing is a valid outcome. ## Before running @@ -75,10 +76,12 @@ just outright errors. nemo agents analyst doctor ``` -Two failures block a run and must be fixed first: an `optimizer.yaml` that is -missing or unparseable when no `--agent` is passed, and an unset -`INFERENCE_API_KEY`. Platform reachability and the workspace probe are advisory -— they warn, and the run still proceeds. +Only two results block a run: an unset `INFERENCE_API_KEY`, and an +`optimizer.yaml` that is missing or unparseable — the second only if you intend +to run without `--agent`. Doctor takes no `--agent` flag, so it always checks +for a profile and always reports a red line when there is none; when you pass +`--agent`, that line is noise. Platform reachability and the workspace probe +only ever warn. ## Run it @@ -89,10 +92,9 @@ nemo agents analyst run \ --base-url "$NMP_BASE_URL" ``` -Add `--agent-spec AGENT-SPEC.md` to enable divergence checking, and `--verbose` -to stream the Analyst's tool calls and reasoning to stderr while it works. -Expect a run to take several minutes; it surveys many sessions before drilling -into any of them. +Add `--agent-spec AGENT-SPEC.md` for divergence checking, and `--verbose` to +stream the Analyst's tool calls and reasoning to stderr. Expect several minutes; +it surveys many sessions before drilling into any of them. From an agent directory, an `optimizer.yaml` profile supplies `agent`, `workspace`, and `agent_spec`, so the flags above become optional: @@ -107,18 +109,16 @@ ignored. ## Where Insights are stored -Insights always go to the platform. To keep a local copy, pass -`--insights-file-output`, which mirrors what the platform stored, platform IDs -included, and merges into that file on each run: +Insights always go to the platform. `--insights-file-output` additionally +mirrors what the platform stored, platform IDs included, merging into that file +on each run; a mirror that cannot be written warns rather than failing the run. ```bash nemo agents analyst run --agent --insights-file-output .nemo-optimizer/insights.yaml ``` -That path is the one the Experimentalist reads by default, so it is the -conventional choice when handing off locally. A mirror that cannot be written -degrades to a warning rather than failing the run, because the platform is the -source of truth. +That path is what the Experimentalist reads by default, so it is the +conventional choice when handing off locally. ## Verify @@ -131,20 +131,20 @@ curl --fail-with-body \ "$NMP_BASE_URL/apis/insights/v2/workspaces//insights?agent=&page=1&page_size=20" ``` -Drop the `Authorization` header on a local platform with authentication -disabled. A successful run leaves at least one Insight for the agent, each with -a clear title, an actionable description, and non-empty `trace_refs`. Stored -Insights also appear in Studio under the workspace's optimizer view. If you -passed `--insights-file-output`, read the mirror back and confirm it is -non-empty. +On a local platform with authentication disabled, `nemo auth token` fails and +the header can be dropped. A successful run leaves at least one Insight for the +agent, each with a clear title, an actionable description, and non-empty +`trace_refs`; stored Insights also appear in Studio's optimizer view for the +workspace. ## When it finds nothing -An empty result is either a real "nothing worth filing" or one of two setup -problems. Check the agent name matches the `agent_name` on the spans exactly, -since the Analyst scopes everything through spans, then confirm telemetry is -actually flowing for that agent and workspace. Too few traces produces the same -empty result as a healthy agent. +Besides a real "nothing worth filing", three things produce an empty result. +Scoping: the Analyst reads only what `--agent` and `--workspace` together +select, and `agent_name` is carried on agent-level spans, not on their model and +tool children. Volume: too few traces looks the same as a healthy agent. And +telemetry that captures only the shape of a run, spans without the inputs and +outputs, leaves nothing to judge however many spans there are. ## Hand off From 119fa882c3727959d5213ccfd6dae5ac02d7f0f0 Mon Sep 17 00:00:00 2001 From: Aditya Pandey Date: Fri, 7 Aug 2026 11:44:52 -0600 Subject: [PATCH 3/9] feat(skills): decouple nemo-analyst from a specific model and credential MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review feedback on #1116. #1159 moves the optimization agents onto the models chosen in `nemo setup` and drops `INFERENCE_API_KEY` as an agent credential, so naming that variable, the gateway, and Claude Opus 4.8 would go stale the day it lands. State the requirement instead — the Analyst needs a model to run on — and let `analyst doctor` be the authority on whether it is satisfied, which is true before and after that change. Also from review: - Frame the skill as analyzing behavior and producing Insights rather than "finding what an agent keeps getting wrong". - Describe `--agent-spec` by what it gives the Analyst: the intent behind the agent, which is not recoverable from code or traces. Per the AGENT-SPEC design doc, that is its whole job. (The proposed ETHOS.md rename has not landed, so the filename stays.) - Drop `--base-url` from the run example; it defaults to `NMP_BASE_URL`. The wider `NMP_*` rename is ASE-815 — 414 files reference those variables, so it is not a skill-PR change. Signed-off-by: Aditya Pandey --- .../skills/nemo-analyst/SKILL.md | 36 +++++++++---------- 1 file changed, 17 insertions(+), 19 deletions(-) diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md index 4a927afaa4..42da6cb628 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md @@ -24,8 +24,8 @@ not-for: - nemo-evaluator (use to author evaluations and metrics; this skill analyzes production behavior) compatibility: >- nemo-platform >= 0.1.0; requires the Insights plugin, a reachable platform - with Intake telemetry for the target agent, and INFERENCE_API_KEY for NVIDIA - Inference Gateway access. No Docker or datasets needed. + with Intake telemetry for the target agent, and a model the platform can call + on the Analyst's behalf. No Docker or datasets needed. maturity: beta license: Apache-2.0 user-invocable: true @@ -34,8 +34,8 @@ allowed-tools: [Bash, Read] # NeMo Analyst -Find what an agent keeps getting wrong, from its own telemetry, and record it as -an Insight. +Analyze an agent's behavior from its own telemetry and record what recurs as +Insights. ## What it produces @@ -60,15 +60,16 @@ worth more than ten vague ones, so a run that files nothing is a valid outcome. The Analyst reads telemetry; it cannot create it. Confirm all three: - The target agent already has traces in Intake. No traces means no Insights. -- `INFERENCE_API_KEY` is set. The Analyst runs on Claude Opus 4.8 through the - NVIDIA Inference Gateway and reads **only** this variable. It does not use the - `NEMO_EXPERIMENTALIST_MODELS_*` tiers the Experimentalist needs, so do not - copy that configuration here. - The platform is reachable at `NMP_BASE_URL`. +- The Analyst has a model to run on. It is an LLM agent itself, and it uses the + models the platform is configured with rather than any credential of its own. + Don't hand it the Experimentalist's configuration. Pre-flight is the authority + on whether this is satisfied, and says how to fix it if not. -An `AGENT-SPEC.md` is optional but makes the Analyst materially better: given -the agent's intended behavior it can flag divergence from that contract, not -just outright errors. +An `AGENT-SPEC.md` is optional but makes the Analyst materially better. It +carries the intent behind the agent — what it is for, its constraints, what +counts as success — none of which is recoverable from code or traces, so +without it the Analyst can only judge an agent against itself. ## Pre-flight @@ -76,7 +77,7 @@ just outright errors. nemo agents analyst doctor ``` -Only two results block a run: an unset `INFERENCE_API_KEY`, and an +Only two results block a run: no usable model configured, and an `optimizer.yaml` that is missing or unparseable — the second only if you intend to run without `--agent`. Doctor takes no `--agent` flag, so it always checks for a profile and always reports a red line when there is none; when you pass @@ -86,15 +87,12 @@ only ever warn. ## Run it ```bash -nemo agents analyst run \ - --agent \ - --workspace \ - --base-url "$NMP_BASE_URL" +nemo agents analyst run --agent --workspace ``` -Add `--agent-spec AGENT-SPEC.md` for divergence checking, and `--verbose` to -stream the Analyst's tool calls and reasoning to stderr. Expect several minutes; -it surveys many sessions before drilling into any of them. +Add `--agent-spec AGENT-SPEC.md` to tell it what the agent is supposed to do, +and `--verbose` to stream its tool calls and reasoning to stderr. Expect several +minutes; it surveys many sessions before drilling into any of them. From an agent directory, an `optimizer.yaml` profile supplies `agent`, `workspace`, and `agent_spec`, so the flags above become optional: From 192eb126ba1d5b7ba85e0bce97c6fdb390aae794 Mon Sep 17 00:00:00 2001 From: Aditya Pandey Date: Fri, 7 Aug 2026 12:00:25 -0600 Subject: [PATCH 4/9] fix(skills): don't claim nemo-analyst has no credential of its own MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit wrote the credential story as #1159 will leave it, but that PR has not landed: the analyst still reads INFERENCE_API_KEY directly today, so "rather than any credential of its own" was false in the world this ships into. Say only that the model configuration is in flux and that pre-flight names what is missing — accurate before and after. Signed-off-by: Aditya Pandey --- .../src/nemo_platform_ext/skills/nemo-analyst/SKILL.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md index 42da6cb628..318f9b713f 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md @@ -61,10 +61,10 @@ The Analyst reads telemetry; it cannot create it. Confirm all three: - The target agent already has traces in Intake. No traces means no Insights. - The platform is reachable at `NMP_BASE_URL`. -- The Analyst has a model to run on. It is an LLM agent itself, and it uses the - models the platform is configured with rather than any credential of its own. - Don't hand it the Experimentalist's configuration. Pre-flight is the authority - on whether this is satisfied, and says how to fix it if not. +- The Analyst has a model to run on. It is an LLM agent itself, and how that is + configured is changing, so let pre-flight tell you whether it is satisfied — + it names what is missing and how to set it. Don't reach for the + Experimentalist's configuration; that is a different contract. An `AGENT-SPEC.md` is optional but makes the Analyst materially better. It carries the intent behind the agent — what it is for, its constraints, what From 7d73c271f8b3b4437abeea3a39ccc52f336e85ce Mon Sep 17 00:00:00 2001 From: Aditya Pandey Date: Fri, 7 Aug 2026 13:33:20 -0600 Subject: [PATCH 5/9] fix(skills): vendor nemo-analyst into the SDK MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `lint-sdk-vendored` and `lint-cli` both fail on this branch because every skill is mirrored into `sdk/python/nemo-platform` and this one never was — it was the only skill missing from the vendored tree. Generated with `make vendor`; as with every other skill, `tests.json` is not vendored. Signed-off-by: Aditya Pandey --- .../skills/nemo-analyst/SKILL.md | 157 ++++++++++++++++++ 1 file changed, 157 insertions(+) create mode 100644 sdk/python/nemo-platform/src/nemo_platform/skills/nemo-analyst/SKILL.md diff --git a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-analyst/SKILL.md b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-analyst/SKILL.md new file mode 100644 index 0000000000..318f9b713f --- /dev/null +++ b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-analyst/SKILL.md @@ -0,0 +1,157 @@ +--- +name: nemo-analyst +description: >- + Analyze an agent's production traces to find recurring failure patterns and + record each as an Insight. Surveys spans, evaluator scores, and user feedback + across many sessions, clusters similar failures, then files every finding as a + titled Insight carrying the trace IDs that evidence the problem. Answers why + an agent keeps failing, where it gets things wrong, and the recurring + problems hiding in production traces. Produces the Insight that + nemo-experimentalist consumes. +triggers: + - nemo-analyst + - analyze my agent's traces + - why my agent keeps failing + - generate insights for my agent + - find recurring failure patterns + - run the analyst + - my agent keeps getting wrong +not-for: + - nemo-experimentalist (use to act on an Insight and change the agent; this skill produces the Insight it consumes) + - nemo-intake (use to instrument an agent, ingest telemetry, or query raw spans; this skill interprets telemetry that already landed) + - nemo-experiments-upload (use to upload traces and evaluation results into Intake; this skill reads them back out) + - nemo-explore (use to design an agent that does not exist yet; this skill needs a running agent with traces) + - nemo-evaluator (use to author evaluations and metrics; this skill analyzes production behavior) +compatibility: >- + nemo-platform >= 0.1.0; requires the Insights plugin, a reachable platform + with Intake telemetry for the target agent, and a model the platform can call + on the Analyst's behalf. No Docker or datasets needed. +maturity: beta +license: Apache-2.0 +user-invocable: true +allowed-tools: [Bash, Read] +--- + +# NeMo Analyst + +Analyze an agent's behavior from its own telemetry and record what recurs as +Insights. + +## What it produces + +An Insight is a persistent, named description of one recurring problem, and it +is the unit of work the rest of the optimization loop runs on. Each carries: + +- `title` — a sentence naming the failure, such as "Retrieval drops relevant + context near the token limit" +- `description` — the failure mode, the tool or model call it affects, and the + conditions that trigger it +- `trace_refs` — the Intake trace IDs cited as evidence, so a developer can + audit the reasoning and build regression tests + +The Analyst targets at least three representative traces per Insight and appends +evidence to an existing Insight rather than filing a near-duplicate. It judges +behavior rather than status or scores, so it finds failures in sessions that +reported success and passed their evaluations. Two well-evidenced Insights are +worth more than ten vague ones, so a run that files nothing is a valid outcome. + +## Before running + +The Analyst reads telemetry; it cannot create it. Confirm all three: + +- The target agent already has traces in Intake. No traces means no Insights. +- The platform is reachable at `NMP_BASE_URL`. +- The Analyst has a model to run on. It is an LLM agent itself, and how that is + configured is changing, so let pre-flight tell you whether it is satisfied — + it names what is missing and how to set it. Don't reach for the + Experimentalist's configuration; that is a different contract. + +An `AGENT-SPEC.md` is optional but makes the Analyst materially better. It +carries the intent behind the agent — what it is for, its constraints, what +counts as success — none of which is recoverable from code or traces, so +without it the Analyst can only judge an agent against itself. + +## Pre-flight + +```bash +nemo agents analyst doctor +``` + +Only two results block a run: no usable model configured, and an +`optimizer.yaml` that is missing or unparseable — the second only if you intend +to run without `--agent`. Doctor takes no `--agent` flag, so it always checks +for a profile and always reports a red line when there is none; when you pass +`--agent`, that line is noise. Platform reachability and the workspace probe +only ever warn. + +## Run it + +```bash +nemo agents analyst run --agent --workspace +``` + +Add `--agent-spec AGENT-SPEC.md` to tell it what the agent is supposed to do, +and `--verbose` to stream its tool calls and reasoning to stderr. Expect several +minutes; it surveys many sessions before drilling into any of them. + +From an agent directory, an `optimizer.yaml` profile supplies `agent`, +`workspace`, and `agent_spec`, so the flags above become optional: + +```bash +nemo agents analyst run +``` + +The profile is discovered by walking up from the current directory. Only those +three fields are read from it; other keys belong to the Experimentalist and are +ignored. + +## Where Insights are stored + +Insights always go to the platform. `--insights-file-output` additionally +mirrors what the platform stored, platform IDs included, merging into that file +on each run; a mirror that cannot be written warns rather than failing the run. + +```bash +nemo agents analyst run --agent --insights-file-output .nemo-optimizer/insights.yaml +``` + +That path is what the Experimentalist reads by default, so it is the +conventional choice when handing off locally. + +## Verify + +Do not report success without checking that Insights actually landed. There is +no CLI verb for this yet, so read the Insights API directly: + +```bash +curl --fail-with-body \ + -H "Authorization: Bearer $(nemo auth token)" \ + "$NMP_BASE_URL/apis/insights/v2/workspaces//insights?agent=&page=1&page_size=20" +``` + +On a local platform with authentication disabled, `nemo auth token` fails and +the header can be dropped. A successful run leaves at least one Insight for the +agent, each with a clear title, an actionable description, and non-empty +`trace_refs`; stored Insights also appear in Studio's optimizer view for the +workspace. + +## When it finds nothing + +Besides a real "nothing worth filing", three things produce an empty result. +Scoping: the Analyst reads only what `--agent` and `--workspace` together +select, and `agent_name` is carried on agent-level spans, not on their model and +tool children. Volume: too few traces looks the same as a healthy agent. And +telemetry that captures only the shape of a run, spans without the inputs and +outputs, leaves nothing to judge however many spans there are. + +## Hand off + +Once an Insight exists, the Experimentalist acts on it: + +```bash +nemo agents experimentalist run +``` + +For the full data model, the Analyst's tool set, periodic analysis via +`nemo insights analysis enable`, and the rest of the loop, see +[Insight-Driven Optimization](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/docs/agents/insight-driven-optimization.mdx). From e9e96eff5c4d3bf58a3b2bb5fcc11efddc1e3582 Mon Sep 17 00:00:00 2001 From: Aditya Pandey Date: Fri, 7 Aug 2026 13:51:05 -0600 Subject: [PATCH 6/9] refactor(skills): ship nemo-analyst with the Insights plugin MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow the pattern #1088 settled on. That PR started with the Experimentalist skill in the platform package and deliberately moved it into the plugin; this one was written against the earlier shape. The skill documents `nemo agents analyst`, which only exists when nemo-insights is installed, so the platform package should not ship it. Moving it also removes it from the vendored SDK tree — plugin-owned skills are discovered through the `nemo.skills` entry point instead, which is what made lint-sdk-vendored care about it in the first place. Mirrors the Experimentalist layout: skills.py exposing skills_dir, a skills README, and an entry-point test. `nemo skills list` now reports the skill with source nemo-insights-plugin. Also adds the nemo-analyst row to nemo-skill-selection, as #1088 did for the Experimentalist. The catalog already told agents to use the Experimentalist "after `agents analyst` has created an Insight" while having no row that routed there. The description said "Produces the Insight that nemo-experimentalist consumes". Now that both skills are in the catalog together, that literal token made this skill win the Experimentalist's own explicit routing test — scoring treats a hyphenated name as one token. Reworded to "the Experimentalist", which keeps the loop context without the collision. Catalog-wide: 47 failures both before and after, 12 more passes, and nemo-analyst involved in no failure. Signed-off-by: Aditya Pandey --- .../skills/nemo-skill-selection/SKILL.md | 2 + plugins/nemo-insights/pyproject.toml | 3 + .../src/nemo_insights_plugin/skills.py | 17 ++ .../src/nemo_insights_plugin/skills/README.md | 10 ++ .../skills/nemo-analyst/SKILL.md | 4 +- .../skills/nemo-analyst/tests.json | 0 .../tests/test_skills_entry_point.py | 30 ++++ .../skills/nemo-analyst/SKILL.md | 157 ------------------ .../skills/nemo-skill-selection/SKILL.md | 2 + 9 files changed, 66 insertions(+), 159 deletions(-) create mode 100644 plugins/nemo-insights/src/nemo_insights_plugin/skills.py create mode 100644 plugins/nemo-insights/src/nemo_insights_plugin/skills/README.md rename {packages/nemo_platform_ext/src/nemo_platform_ext => plugins/nemo-insights/src/nemo_insights_plugin}/skills/nemo-analyst/SKILL.md (99%) rename {packages/nemo_platform_ext/src/nemo_platform_ext => plugins/nemo-insights/src/nemo_insights_plugin}/skills/nemo-analyst/tests.json (100%) create mode 100644 plugins/nemo-insights/tests/test_skills_entry_point.py delete mode 100644 sdk/python/nemo-platform/src/nemo_platform/skills/nemo-analyst/SKILL.md diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-skill-selection/SKILL.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-skill-selection/SKILL.md index 89ce30cc27..0847099974 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-skill-selection/SKILL.md +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-skill-selection/SKILL.md @@ -62,6 +62,7 @@ Match the user's intent to one downstream skill. Pick exactly one. | "status", "what is running", "platform health", "is the platform up", "what's deployed", "show me what's running" | `nemo-status` | Read-only dashboard: platform, agents, providers, models | | "shut down", "stop NeMo", "tear down", "clean up" | `nemo-teardown` | Stop the cluster (keep data, delete platform data, or full cleanup) | | "fine-tune", "customize the model", "train on my data", "SFT", "LoRA" | `nemo-customizer` | Model customization via installed customization contributor plugins (`nemo-customizer-plugin`). Requires plugin skills to be installed (`nemo skills install` / enabled-plugins). | +| "why does my agent keep failing", "analyze my agent's traces", "find recurring failure patterns", "generate insights for my agent" | `nemo-analyst` (plugin-owned, in `plugins/nemo-insights`) | Reads an agent's existing telemetry and files each recurring failure pattern as an Insight citing the traces that evidence it. Requires the Insights plugin; produces the Insight `nemo-experimentalist` acts on. | | "improve the agent's own code", "fix my agent harness", "candidate code change", "optimize from an Insight", "improve on train and validation datasets" | `nemo-experimentalist` (plugin-owned, in `plugins/nemo-experimentalist`) | Source/harness optimization: generate and validate candidate code changes against Harbor-compatible evaluation data. Requires the Experimentalist plugin; use after `agents analyst` has created an Insight, or with explicit datasets. | | "optimize my agent", "make it cheaper", "reduce latency", "smaller model", "switchyard", "routing split", "compare against a newer model" | `agents-optimize` (plugin-owned, in `plugins/nemo-agents`) | Cost / latency / quality optimization for a **deployed** agent. Routing splits, skill tuning, prompt tuning, new-model scans. | | "secure my agent", "harden my agent", "check for PII", "leaked secrets", "guardrail coverage" | `agents-secure` (plugin-owned, in `plugins/nemo-agents`) | Safety and security audit for a **deployed** agent. Guardrails, PII, secrets scan. | @@ -133,6 +134,7 @@ Plugin-owned skills: agents-secure safety and security audit for a deployed agent nemo-evaluator evaluation metrics, LLM-judge, benchmark jobs nemo-customizer fine-tuning of models + nemo-analyst analyze agent telemetry and file recurring problems as Insights nemo-experimentalist source/harness optimization from Insights or evaluation datasets guardrails content-safety middleware via virtual models auditor red-team vulnerability scanning (garak) diff --git a/plugins/nemo-insights/pyproject.toml b/plugins/nemo-insights/pyproject.toml index ad90163faa..0906a1fa53 100644 --- a/plugins/nemo-insights/pyproject.toml +++ b/plugins/nemo-insights/pyproject.toml @@ -35,6 +35,9 @@ insights-analysis = "nemo_insights_plugin.controller:InsightsAnalysisController" [project.entry-points."nemo.sdk"] insights = "nemo_insights_plugin.sdk:insights_sdk_resources" +[project.entry-points."nemo.skills"] +insights = "nemo_insights_plugin.skills:skills_dir" + [build-system] requires = ["hatchling"] build-backend = "hatchling.build" diff --git a/plugins/nemo-insights/src/nemo_insights_plugin/skills.py b/plugins/nemo-insights/src/nemo_insights_plugin/skills.py new file mode 100644 index 0000000000..ca1b5eae0a --- /dev/null +++ b/plugins/nemo-insights/src/nemo_insights_plugin/skills.py @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Skills directory exposure — registered under ``nemo.skills``. + +Returns the path to the ``skills/`` directory inside this package so the +platform can discover and load skill markdown files shipped with the plugin. +""" + +from __future__ import annotations + +from pathlib import Path + + +def skills_dir() -> Path: + """Return the directory containing this plugin's skills.""" + return Path(__file__).parent / "skills" diff --git a/plugins/nemo-insights/src/nemo_insights_plugin/skills/README.md b/plugins/nemo-insights/src/nemo_insights_plugin/skills/README.md new file mode 100644 index 0000000000..226a6b739b --- /dev/null +++ b/plugins/nemo-insights/src/nemo_insights_plugin/skills/README.md @@ -0,0 +1,10 @@ + + + +# NeMo Insights plugin skills + +Skills this plugin ships to coding agents, discovered via the `nemo.skills` +entry point and installed with `nemo skills install`. Currently: + +- `nemo-analyst/` — guided workflow for analyzing an agent's telemetry and + recording recurring problems as Insights. diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md b/plugins/nemo-insights/src/nemo_insights_plugin/skills/nemo-analyst/SKILL.md similarity index 99% rename from packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md rename to plugins/nemo-insights/src/nemo_insights_plugin/skills/nemo-analyst/SKILL.md index 318f9b713f..ca26906faa 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/SKILL.md +++ b/plugins/nemo-insights/src/nemo_insights_plugin/skills/nemo-analyst/SKILL.md @@ -6,8 +6,8 @@ description: >- across many sessions, clusters similar failures, then files every finding as a titled Insight carrying the trace IDs that evidence the problem. Answers why an agent keeps failing, where it gets things wrong, and the recurring - problems hiding in production traces. Produces the Insight that - nemo-experimentalist consumes. + problems hiding in production traces. Produces the Insight that the + Experimentalist later acts on. triggers: - nemo-analyst - analyze my agent's traces diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/tests.json b/plugins/nemo-insights/src/nemo_insights_plugin/skills/nemo-analyst/tests.json similarity index 100% rename from packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-analyst/tests.json rename to plugins/nemo-insights/src/nemo_insights_plugin/skills/nemo-analyst/tests.json diff --git a/plugins/nemo-insights/tests/test_skills_entry_point.py b/plugins/nemo-insights/tests/test_skills_entry_point.py new file mode 100644 index 0000000000..0abe387428 --- /dev/null +++ b/plugins/nemo-insights/tests/test_skills_entry_point.py @@ -0,0 +1,30 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Tests for the nemo.skills entry-point exposure.""" + +from __future__ import annotations + +from pathlib import Path + + +def test_skills_dir_contains_analyst() -> None: + from nemo_insights_plugin.skills import skills_dir + + analyst = skills_dir() / "nemo-analyst" + assert analyst.is_dir(), ( + f"Expected 'nemo-analyst' subdir under {skills_dir()!r} — got {list(skills_dir().iterdir())}" + ) + + +def test_entry_point_loads_skills_dir() -> None: + """The nemo.skills entry-point must resolve to our skills_dir function.""" + from importlib.metadata import entry_points + + eps = [ep for ep in entry_points(group="nemo.skills") if ep.name == "insights"] + assert len(eps) == 1, f"Expected exactly one 'insights' entry-point, got {eps}" + loaded = eps[0].load() + assert callable(loaded), f"Entry-point did not resolve to a callable: {loaded!r}" + result = loaded() + assert isinstance(result, Path), f"skills_dir() returned {result!r} (not Path)" + assert result.is_dir(), f"skills_dir() returned {result!r} which is not a directory" diff --git a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-analyst/SKILL.md b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-analyst/SKILL.md deleted file mode 100644 index 318f9b713f..0000000000 --- a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-analyst/SKILL.md +++ /dev/null @@ -1,157 +0,0 @@ ---- -name: nemo-analyst -description: >- - Analyze an agent's production traces to find recurring failure patterns and - record each as an Insight. Surveys spans, evaluator scores, and user feedback - across many sessions, clusters similar failures, then files every finding as a - titled Insight carrying the trace IDs that evidence the problem. Answers why - an agent keeps failing, where it gets things wrong, and the recurring - problems hiding in production traces. Produces the Insight that - nemo-experimentalist consumes. -triggers: - - nemo-analyst - - analyze my agent's traces - - why my agent keeps failing - - generate insights for my agent - - find recurring failure patterns - - run the analyst - - my agent keeps getting wrong -not-for: - - nemo-experimentalist (use to act on an Insight and change the agent; this skill produces the Insight it consumes) - - nemo-intake (use to instrument an agent, ingest telemetry, or query raw spans; this skill interprets telemetry that already landed) - - nemo-experiments-upload (use to upload traces and evaluation results into Intake; this skill reads them back out) - - nemo-explore (use to design an agent that does not exist yet; this skill needs a running agent with traces) - - nemo-evaluator (use to author evaluations and metrics; this skill analyzes production behavior) -compatibility: >- - nemo-platform >= 0.1.0; requires the Insights plugin, a reachable platform - with Intake telemetry for the target agent, and a model the platform can call - on the Analyst's behalf. No Docker or datasets needed. -maturity: beta -license: Apache-2.0 -user-invocable: true -allowed-tools: [Bash, Read] ---- - -# NeMo Analyst - -Analyze an agent's behavior from its own telemetry and record what recurs as -Insights. - -## What it produces - -An Insight is a persistent, named description of one recurring problem, and it -is the unit of work the rest of the optimization loop runs on. Each carries: - -- `title` — a sentence naming the failure, such as "Retrieval drops relevant - context near the token limit" -- `description` — the failure mode, the tool or model call it affects, and the - conditions that trigger it -- `trace_refs` — the Intake trace IDs cited as evidence, so a developer can - audit the reasoning and build regression tests - -The Analyst targets at least three representative traces per Insight and appends -evidence to an existing Insight rather than filing a near-duplicate. It judges -behavior rather than status or scores, so it finds failures in sessions that -reported success and passed their evaluations. Two well-evidenced Insights are -worth more than ten vague ones, so a run that files nothing is a valid outcome. - -## Before running - -The Analyst reads telemetry; it cannot create it. Confirm all three: - -- The target agent already has traces in Intake. No traces means no Insights. -- The platform is reachable at `NMP_BASE_URL`. -- The Analyst has a model to run on. It is an LLM agent itself, and how that is - configured is changing, so let pre-flight tell you whether it is satisfied — - it names what is missing and how to set it. Don't reach for the - Experimentalist's configuration; that is a different contract. - -An `AGENT-SPEC.md` is optional but makes the Analyst materially better. It -carries the intent behind the agent — what it is for, its constraints, what -counts as success — none of which is recoverable from code or traces, so -without it the Analyst can only judge an agent against itself. - -## Pre-flight - -```bash -nemo agents analyst doctor -``` - -Only two results block a run: no usable model configured, and an -`optimizer.yaml` that is missing or unparseable — the second only if you intend -to run without `--agent`. Doctor takes no `--agent` flag, so it always checks -for a profile and always reports a red line when there is none; when you pass -`--agent`, that line is noise. Platform reachability and the workspace probe -only ever warn. - -## Run it - -```bash -nemo agents analyst run --agent --workspace -``` - -Add `--agent-spec AGENT-SPEC.md` to tell it what the agent is supposed to do, -and `--verbose` to stream its tool calls and reasoning to stderr. Expect several -minutes; it surveys many sessions before drilling into any of them. - -From an agent directory, an `optimizer.yaml` profile supplies `agent`, -`workspace`, and `agent_spec`, so the flags above become optional: - -```bash -nemo agents analyst run -``` - -The profile is discovered by walking up from the current directory. Only those -three fields are read from it; other keys belong to the Experimentalist and are -ignored. - -## Where Insights are stored - -Insights always go to the platform. `--insights-file-output` additionally -mirrors what the platform stored, platform IDs included, merging into that file -on each run; a mirror that cannot be written warns rather than failing the run. - -```bash -nemo agents analyst run --agent --insights-file-output .nemo-optimizer/insights.yaml -``` - -That path is what the Experimentalist reads by default, so it is the -conventional choice when handing off locally. - -## Verify - -Do not report success without checking that Insights actually landed. There is -no CLI verb for this yet, so read the Insights API directly: - -```bash -curl --fail-with-body \ - -H "Authorization: Bearer $(nemo auth token)" \ - "$NMP_BASE_URL/apis/insights/v2/workspaces//insights?agent=&page=1&page_size=20" -``` - -On a local platform with authentication disabled, `nemo auth token` fails and -the header can be dropped. A successful run leaves at least one Insight for the -agent, each with a clear title, an actionable description, and non-empty -`trace_refs`; stored Insights also appear in Studio's optimizer view for the -workspace. - -## When it finds nothing - -Besides a real "nothing worth filing", three things produce an empty result. -Scoping: the Analyst reads only what `--agent` and `--workspace` together -select, and `agent_name` is carried on agent-level spans, not on their model and -tool children. Volume: too few traces looks the same as a healthy agent. And -telemetry that captures only the shape of a run, spans without the inputs and -outputs, leaves nothing to judge however many spans there are. - -## Hand off - -Once an Insight exists, the Experimentalist acts on it: - -```bash -nemo agents experimentalist run -``` - -For the full data model, the Analyst's tool set, periodic analysis via -`nemo insights analysis enable`, and the rest of the loop, see -[Insight-Driven Optimization](https://github.com/NVIDIA-NeMo/nemo-platform/blob/main/docs/agents/insight-driven-optimization.mdx). diff --git a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-skill-selection/SKILL.md b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-skill-selection/SKILL.md index 89ce30cc27..0847099974 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-skill-selection/SKILL.md +++ b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-skill-selection/SKILL.md @@ -62,6 +62,7 @@ Match the user's intent to one downstream skill. Pick exactly one. | "status", "what is running", "platform health", "is the platform up", "what's deployed", "show me what's running" | `nemo-status` | Read-only dashboard: platform, agents, providers, models | | "shut down", "stop NeMo", "tear down", "clean up" | `nemo-teardown` | Stop the cluster (keep data, delete platform data, or full cleanup) | | "fine-tune", "customize the model", "train on my data", "SFT", "LoRA" | `nemo-customizer` | Model customization via installed customization contributor plugins (`nemo-customizer-plugin`). Requires plugin skills to be installed (`nemo skills install` / enabled-plugins). | +| "why does my agent keep failing", "analyze my agent's traces", "find recurring failure patterns", "generate insights for my agent" | `nemo-analyst` (plugin-owned, in `plugins/nemo-insights`) | Reads an agent's existing telemetry and files each recurring failure pattern as an Insight citing the traces that evidence it. Requires the Insights plugin; produces the Insight `nemo-experimentalist` acts on. | | "improve the agent's own code", "fix my agent harness", "candidate code change", "optimize from an Insight", "improve on train and validation datasets" | `nemo-experimentalist` (plugin-owned, in `plugins/nemo-experimentalist`) | Source/harness optimization: generate and validate candidate code changes against Harbor-compatible evaluation data. Requires the Experimentalist plugin; use after `agents analyst` has created an Insight, or with explicit datasets. | | "optimize my agent", "make it cheaper", "reduce latency", "smaller model", "switchyard", "routing split", "compare against a newer model" | `agents-optimize` (plugin-owned, in `plugins/nemo-agents`) | Cost / latency / quality optimization for a **deployed** agent. Routing splits, skill tuning, prompt tuning, new-model scans. | | "secure my agent", "harden my agent", "check for PII", "leaked secrets", "guardrail coverage" | `agents-secure` (plugin-owned, in `plugins/nemo-agents`) | Safety and security audit for a **deployed** agent. Guardrails, PII, secrets scan. | @@ -133,6 +134,7 @@ Plugin-owned skills: agents-secure safety and security audit for a deployed agent nemo-evaluator evaluation metrics, LLM-judge, benchmark jobs nemo-customizer fine-tuning of models + nemo-analyst analyze agent telemetry and file recurring problems as Insights nemo-experimentalist source/harness optimization from Insights or evaluation datasets guardrails content-safety middleware via virtual models auditor red-team vulnerability scanning (garak) From 6740cbf53e2a2975eac385c1cfea2e3720056afd Mon Sep 17 00:00:00 2001 From: Aditya Pandey Date: Fri, 7 Aug 2026 13:57:29 -0600 Subject: [PATCH 7/9] docs(skills): note Analyst telemetry analysis as plugin-owned Completes the #1088 parallel: that PR appended the Experimentalist to the plugin-owned skills sentence in both files once its skill moved into the plugin. The Analyst now lives there too. Signed-off-by: Aditya Pandey --- AGENTS.md | 2 +- CLAUDE.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index fdc151b56a..55dd24ed98 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,7 +36,7 @@ User-facing skills in `packages/nemo_platform_ext/src/nemo_platform_ext/skills/` - `nemo-status`: read-only health dashboard. - `nemo-teardown`: guided shutdown with confirmation. -Plugin-owned skills under `plugins/*/src/*/skills/` handle their own routing for customization, guardrails, evaluations, optimization, data designer, anonymizer, auditor, and Experimentalist source/harness improvement. +Plugin-owned skills under `plugins/*/src/*/skills/` handle their own routing for customization, guardrails, evaluations, optimization, data designer, anonymizer, auditor, Experimentalist source/harness improvement, and Analyst telemetry analysis. ### Working in a sandboxed environment diff --git a/CLAUDE.md b/CLAUDE.md index 0b4eba76c7..2756949eb6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -34,7 +34,7 @@ User-facing skills in `packages/nemo_platform_ext/src/nemo_platform_ext/skills/` - `nemo-status`: read-only health dashboard. Run this before assuming the platform is up. - `nemo-teardown`: guided shutdown with confirmation. -Plugin-owned skills live under `plugins/*/src/*/skills/` and handle their own routing for customization, guardrails, evaluations, optimization, data designer, anonymizer, auditor, and Experimentalist source/harness improvement. +Plugin-owned skills live under `plugins/*/src/*/skills/` and handle their own routing for customization, guardrails, evaluations, optimization, data designer, anonymizer, auditor, Experimentalist source/harness improvement, and Analyst telemetry analysis. ### Working in a sandboxed coding-agent environment From 8cea8c5687722d7489eca2e3193f3fb82833a09c Mon Sep 17 00:00:00 2001 From: Aditya Pandey Date: Fri, 7 Aug 2026 14:39:06 -0600 Subject: [PATCH 8/9] fix(skills): verify the analyst run, not the stored Insight set MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit review on #1116. The Verify section contradicted the skill's own premise: it said a successful run leaves at least one Insight, while "What it produces" says filing nothing is a valid outcome. It also verified the wrong thing — listing by `?agent=` returns every Insight for the agent, including earlier runs', so it attests the store rather than this run. Verify against what the run reports instead. `analyst_backend` emits a line per operation (`- created: [<id>]`, `- updated: <id>`, or `- no insights created or updated`), so read those ids back individually via `GET .../insights/<insight-id>`, and treat "no insights created or updated" as the success it is. Two smaller fixes from the same review: - Keep the bearer token out of `curl`'s argv, where any process on the host can read it, by passing it through curl's config on stdin. Verified the header still arrives. - The trigger `my agent keeps getting wrong` was missing its object; now `my agent keeps getting things wrong`. Routing is unchanged against a clean main baseline — 47 failures both sides, and nemo-analyst still fails none and steals none. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com> --- .../skills/nemo-analyst/SKILL.md | 26 ++++++++++++------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/plugins/nemo-insights/src/nemo_insights_plugin/skills/nemo-analyst/SKILL.md b/plugins/nemo-insights/src/nemo_insights_plugin/skills/nemo-analyst/SKILL.md index ca26906faa..ada3262f72 100644 --- a/plugins/nemo-insights/src/nemo_insights_plugin/skills/nemo-analyst/SKILL.md +++ b/plugins/nemo-insights/src/nemo_insights_plugin/skills/nemo-analyst/SKILL.md @@ -15,7 +15,7 @@ triggers: - generate insights for my agent - find recurring failure patterns - run the analyst - - my agent keeps getting wrong + - my agent keeps getting things wrong not-for: - nemo-experimentalist (use to act on an Insight and change the agent; this skill produces the Insight it consumes) - nemo-intake (use to instrument an agent, ingest telemetry, or query raw spans; this skill interprets telemetry that already landed) @@ -120,20 +120,26 @@ conventional choice when handing off locally. ## Verify -Do not report success without checking that Insights actually landed. There is -no CLI verb for this yet, so read the Insights API directly: +Do not report success on an exit code. The run prints a line per operation — +`- created: <title> [<insight-id>] (<n> trace refs)`, `- updated: <insight-id> +(<n> trace refs)`, or `- no insights created or updated`, which is a successful +run too. Read back by id whatever it says it wrote, and check each carries a +clear title, an actionable description, and non-empty `trace_refs`. Listing by +`?agent=` also returns earlier runs, so it attests the store, not this run: ```bash curl --fail-with-body \ - -H "Authorization: Bearer $(nemo auth token)" \ - "$NMP_BASE_URL/apis/insights/v2/workspaces/<workspace>/insights?agent=<agent-name>&page=1&page_size=20" + "$NMP_BASE_URL/apis/insights/v2/workspaces/<workspace>/insights/<insight-id>" ``` -On a local platform with authentication disabled, `nemo auth token` fails and -the header can be dropped. A successful run leaves at least one Insight for the -agent, each with a clear title, an actionable description, and non-empty -`trace_refs`; stored Insights also appear in Studio's optimizer view for the -workspace. +On an authenticated platform pass the token through curl's config, not argv +where any process on the host can read it: + +```bash +printf 'header = "Authorization: Bearer %s"' "$(nemo auth token)" | curl -K - <url> +``` + +Stored Insights also appear in Studio's optimizer view for the workspace. ## When it finds nothing From f7af7ff056152c9eda3a082ca35758b5d2e28616 Mon Sep 17 00:00:00 2001 From: Aditya Pandey <aditypandey@nvidia.com> Date: Fri, 7 Aug 2026 15:02:25 -0600 Subject: [PATCH 9/9] refactor(skills): drop the Insights skills README and entry-point test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review feedback on #1116. Neither file was pulling its weight. skills/README.md is never read — the registry only looks at subdirectories containing a SKILL.md — and with a single skill in the directory it restated that skill's own frontmatter. The Experimentalist's equivalent earns its place by disambiguating a user-facing skill from an internal one; this one had nothing to disambiguate. test_skills_entry_point.py guarded the `nemo.skills` declaration in pyproject.toml. Without that entry the skill simply never appears in `nemo skills list`, so the failure announces itself the first time anyone uses the feature and does not need a regression test of its own. Verified by hand instead: the entry point resolves and `nemo skills list` reports nemo-analyst with source nemo-insights-plugin. Signed-off-by: Aditya Pandey <aditypandey@nvidia.com> --- .../src/nemo_insights_plugin/skills/README.md | 10 ------- .../tests/test_skills_entry_point.py | 30 ------------------- 2 files changed, 40 deletions(-) delete mode 100644 plugins/nemo-insights/src/nemo_insights_plugin/skills/README.md delete mode 100644 plugins/nemo-insights/tests/test_skills_entry_point.py diff --git a/plugins/nemo-insights/src/nemo_insights_plugin/skills/README.md b/plugins/nemo-insights/src/nemo_insights_plugin/skills/README.md deleted file mode 100644 index 226a6b739b..0000000000 --- a/plugins/nemo-insights/src/nemo_insights_plugin/skills/README.md +++ /dev/null @@ -1,10 +0,0 @@ -<!-- SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> -<!-- SPDX-License-Identifier: Apache-2.0 --> - -# NeMo Insights plugin skills - -Skills this plugin ships to coding agents, discovered via the `nemo.skills` -entry point and installed with `nemo skills install`. Currently: - -- `nemo-analyst/` — guided workflow for analyzing an agent's telemetry and - recording recurring problems as Insights. diff --git a/plugins/nemo-insights/tests/test_skills_entry_point.py b/plugins/nemo-insights/tests/test_skills_entry_point.py deleted file mode 100644 index 0abe387428..0000000000 --- a/plugins/nemo-insights/tests/test_skills_entry_point.py +++ /dev/null @@ -1,30 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -"""Tests for the nemo.skills entry-point exposure.""" - -from __future__ import annotations - -from pathlib import Path - - -def test_skills_dir_contains_analyst() -> None: - from nemo_insights_plugin.skills import skills_dir - - analyst = skills_dir() / "nemo-analyst" - assert analyst.is_dir(), ( - f"Expected 'nemo-analyst' subdir under {skills_dir()!r} — got {list(skills_dir().iterdir())}" - ) - - -def test_entry_point_loads_skills_dir() -> None: - """The nemo.skills entry-point must resolve to our skills_dir function.""" - from importlib.metadata import entry_points - - eps = [ep for ep in entry_points(group="nemo.skills") if ep.name == "insights"] - assert len(eps) == 1, f"Expected exactly one 'insights' entry-point, got {eps}" - loaded = eps[0].load() - assert callable(loaded), f"Entry-point did not resolve to a callable: {loaded!r}" - result = loaded() - assert isinstance(result, Path), f"skills_dir() returned {result!r} (not Path)" - assert result.is_dir(), f"skills_dir() returned {result!r} which is not a directory"