-
Notifications
You must be signed in to change notification settings - Fork 20
feat(nemo-agents): add specialist subagents + rename to Email Security Triage #1222
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
deb031f
feat(nemo-agents): add specialist subagents to the email-phishing exa…
walston 90ebc24
docs(nemo-agents): address CodeRabbit review on the specialist example
walston 45170a5
refactor(nemo-agents): rename example email-phishing-agent -> email-s…
walston 875ec7a
docs(nemo-agents): name both delegated roles in triage goal
walston File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
65 changes: 0 additions & 65 deletions
65
plugins/nemo-agents/examples/nemo-agent-config/email-phishing-agent/README.md
This file was deleted.
Oops, something went wrong.
103 changes: 0 additions & 103 deletions
103
plugins/nemo-agents/examples/nemo-agent-config/email-phishing-agent/agent.yaml
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
94 changes: 94 additions & 0 deletions
94
plugins/nemo-agents/examples/nemo-agent-config/email-security-triage/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| # Tutorial: Deploy and try the Email Security Triage agent | ||
|
|
||
| Deploy a Fabric (`nemo-agents-spec-v1`) agent end to end and watch it classify a | ||
| phishing email. The agent is a DeepAgents orchestrator that calls a deterministic | ||
| `extract_iocs` tool, fans out to specialist sub-agents (brand impersonation, | ||
| attack category, SMTP header auth), and delegates the final verdict to a phishing | ||
| sub-agent. | ||
|
|
||
| **What you'll do:** deploy the example, send it an email, read the verdict, watch | ||
| a specialist fire, find the steps in the trace, and score it against labeled data. | ||
|
|
||
| **Time:** ~5 minutes. | ||
|
|
||
| **Prerequisites:** | ||
|
|
||
| - NeMo Platform running locally (see [SETUP.md](../../../../../SETUP.md)); `export NMP_BASE_URL=http://localhost:8080`. | ||
| - `export NVIDIA_API_KEY=<your key>`. | ||
| - Dependencies synced from the repo root: `uv sync --all-packages` (installs the `email-security-triage-iocs` tool this agent calls). | ||
|
|
||
| ## Step 1: Deploy the agent | ||
|
|
||
| ```bash | ||
| nemo agents create --name email-security-triage \ | ||
| --agent-config plugins/nemo-agents/examples/nemo-agent-config/email-security-triage/agent.yaml | ||
| nemo agents deploy --agent email-security-triage \ | ||
| --name email-security-triage-deployment --mode subprocess | ||
| ``` | ||
|
|
||
| The deploy command waits until the deployment reports `running` on a loopback port. | ||
|
|
||
| ## Step 2: Classify an email | ||
|
|
||
| ```bash | ||
| nemo agents invoke --agent-deployment email-security-triage-deployment \ | ||
| --input $'From: it-support@paypa1-secure.example\nSubject: Verify your account\n\nYour account is locked. Confirm your password at http://paypa1-secure.example/login' | ||
| ``` | ||
|
|
||
| The agent returns a YAML verdict with `is_likely_phishing: true`, listing the | ||
| lookalike sender domain (`paypa1-secure.example`) among its indicators. | ||
| `phishing-analyzer` owns the verdict and emits every field, including | ||
| `attack_type` (e.g. `credential`) and `impersonated_brand` (e.g. `paypal`) — it | ||
| fills those from the `attack-attributor` and `url-brand-analyst` findings, which | ||
| are advisory: it can override them, or supply a value itself when a specialist is | ||
| silent. | ||
|
|
||
| ## Step 3: Watch the header specialist fire | ||
|
|
||
| `header-auth-analyst` reads SMTP authentication results, so it only runs when the | ||
| email actually carries them. Send one that does: | ||
|
|
||
| ```bash | ||
| nemo agents invoke --agent-deployment email-security-triage-deployment \ | ||
| --input $'Received: from mail.evil.example (203.0.113.9)\nFrom: security@paypal.com\nReturn-Path: bounce@evil.example\nAuthentication-Results: mx.example.com; spf=fail smtp.mailfrom=evil.example; dkim=fail header.d=paypal.com; dmarc=fail header.from=paypal.com\nSubject: Unusual sign-in\n\nReview the sign-in at http://paypal-secure-review.example/verify' | ||
| ``` | ||
|
|
||
| The authentication results fail across the board — critically `dmarc=fail` on | ||
| `header.from=paypal.com`, the check tied to the visible `From:` domain, so the | ||
| message isn't authorized to claim `paypal.com`. `header-auth-analyst` names the | ||
| failed mechanism, and it surfaces among the indicators. | ||
|
|
||
| The labeled dataset in Step 5 carries no SMTP headers, so this specialist stays | ||
| idle there. That is deliberate: synthesizing auth results per row would put the | ||
| `phishing`/`benign` label into the input and inflate the score. | ||
|
|
||
| ## Step 4: Find the steps in the trace | ||
|
|
||
| ```bash | ||
| nemo agents logs --agent email-security-triage | ||
| ``` | ||
|
|
||
| The deployment's `artifacts/.../events.atof.jsonl` records the `extract_iocs` tool | ||
| call and a task for each specialist the orchestrator consulted — evidence the tool | ||
| ran and the specialists were invoked, not that the model guessed. Which specialists | ||
| appear depends on the input: `url-brand-analyst` and `attack-attributor` run on the | ||
| Step 2 email, while `header-auth-analyst` appears only for header-bearing input | ||
| like Step 3's. With NeMo Studio Intake enabled (`VITE_FF_INTAKE_ENABLED=true`), the | ||
| same run appears under **Traces**, one span per step. | ||
|
|
||
| ## Step 5: Evaluate against labeled emails | ||
|
|
||
| ```bash | ||
| nemo agents evaluate run \ | ||
| --eval-config plugins/nemo-agents/examples/nemo-agent-config/email-security-triage/email-security-triage-eval.yml \ | ||
| --agent email-security-triage | ||
| ``` | ||
|
|
||
| The judge scores each verdict against the `label` column in | ||
| `data/smaller_test.csv` and prints an accuracy score. | ||
|
|
||
| ## Next Steps | ||
|
|
||
| - **Make it your own:** [CUSTOMIZE.md](CUSTOMIZE.md) — swap the tool, prompts, specialists, model, and data for your own agent. | ||
| - **Container deploys (docker/k8s):** [docs/agents/deploy-agents.mdx](../../../../../docs/agents/deploy-agents.mdx). | ||
| - **Compare with/without a tool:** the sibling [calculator-agent](../calculator-agent) example. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.