Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,26 +1,28 @@
# How to adapt this example for your own agent

**Goal:** turn the email-phishing example into your own agent — a DeepAgents
orchestrator that delegates to a sub-agent and calls your tool.
**Goal:** turn the Email Security Triage example into your own agent — a DeepAgents
orchestrator that delegates to a verdict sub-agent, consults specialist sub-agents,
and calls your tool.

**Prerequisites:** you can deploy and invoke the example ([README.md](README.md)).
Comment thread
coderabbitai[bot] marked this conversation as resolved.

The shape you're reusing:

```text
orchestrator (deepagents) ── delegates ──▶ <your> sub-agent
orchestrator (deepagents) ── delegates ──▶ <your> verdict sub-agent
├──────────── consults ────────────▶ <your> specialist sub-agents
└───────────── calls ─────────────▶ <your tool> (stdio MCP)
```

## Parts & what to change

| Path | What it is | Swap for your own |
|---|---|---|
| `agent.yaml` | The `nemo-agents-spec-v1` config: harness, sub-agent, model, MCP server, telemetry | Rewrite the orchestrator `instructions.system` and the sub-agent `system_prompt` + `description`; set `models.default` (+ `temperature`); rename `name` / `telemetry.project` |
| `agent.yaml` | The `nemo-agents-spec-v1` config: harness, sub-agents, model, MCP server, telemetry | Rewrite the orchestrator `instructions.system` and each sub-agent `system_prompt` + `description`; set `models.default` (+ `temperature`); rename `name` / `telemetry.project` |
| `mcps/iocs.py` | `extract_iocs` (pure regex) + a FastMCP stdio server | Replace the function body with your tool's logic; keep the `@mcp.tool()` wrapper + `main()`. Rename the module and tool |
| `pyproject.toml` | Packages `mcps/`; exposes console `email-phishing-iocs` | Set `name` and `[project.scripts] <console> = "mcps.<module>:main"` |
| `pyproject.toml` | Packages `mcps/`; exposes console `email-security-triage-iocs` | Set `name` and `[project.scripts] <console> = "mcps.<module>:main"` |
| `data/smaller_test.csv` + `build_dataset.py` | Labeled eval rows; the builder assembles a sender-inclusive `email` column | Drop in your rows; edit the assembly to the fields your agent reads |
| `email-phishing-eval.yml` | Eval config (`question_key: email`, `answer_key: label`, `id_key: subject`) | Point the keys at your columns; tune the judge weights/prompt |
| `email-security-triage-eval.yml` | Eval config (`question_key: email`, `answer_key: label`, `id_key: subject`) | Point the keys at your columns; tune the judge weights/prompt |
| `tests/test_extract_iocs.py` | Unit tests for the tool | Rewrite for your tool's contract |

## Keep in sync
Expand All @@ -32,6 +34,32 @@ Two couplings break silently if you rename one side only:

Keep the `mcps/` directory name (a shared namespace across examples); rename the *module* inside it and the console, not the directory. `id_key` (default `subject`) must be unique across your rows — `build_dataset.py` fails generation on duplicates.

## Adding or removing a specialist

Specialists are entries in `harnesses.deepagents.settings.deepagents.subagents`.
Each needs a `name`, a `description` (this is what the orchestrator routes on —
say when to use it, not just what it is), and a `system_prompt`. Three rules the
existing ones follow:

- **Give it a crisp output contract.** Each specialist's first line is a single
lowercase token (`credential`, `paypal`, `spf`, `none`) with reasoning after,
so the verdict sub-agent can lift the value without parsing prose.
- **Paste the material in.** Sub-agents are stateless — they see only the task
text. The orchestrator prompt tells it to inline the full email on every
delegation; keep that if you add specialists.
- **Repeat the untrusted-data guardrail** in every specialist prompt. Email
content is attacker-controlled; each prompt says to treat it as evidence, never
as instructions.

Then wire it into the orchestrator's numbered steps and, if it produces a field
you want in the output, add that key to the verdict schema in both the
orchestrator prompt and the verdict sub-agent's prompt.

A specialist that has no data to read will invent one. `header-auth-analyst` is
gated on the email actually containing `Authentication-Results:`/`Received:`
headers for exactly that reason — and the scored dataset deliberately has none,
since synthesizing them would leak the label into the input.

## Steps

1. **Copy** this directory to `nemo-agent-config/<your-agent>/` — a working starting point.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Tutorial: Deploy and try the Email Security Triage agent

Deploy a Fabric (`nemo-agents-spec-v1`) agent end to end and watch it classify a
phishing email. The agent is a DeepAgents orchestrator that calls a deterministic
`extract_iocs` tool, fans out to specialist sub-agents (brand impersonation,
attack category, SMTP header auth), and delegates the final verdict to a phishing
sub-agent.

**What you'll do:** deploy the example, send it an email, read the verdict, watch
a specialist fire, find the steps in the trace, and score it against labeled data.

**Time:** ~5 minutes.

**Prerequisites:**

- NeMo Platform running locally (see [SETUP.md](../../../../../SETUP.md)); `export NMP_BASE_URL=http://localhost:8080`.
- `export NVIDIA_API_KEY=<your key>`.
- Dependencies synced from the repo root: `uv sync --all-packages` (installs the `email-security-triage-iocs` tool this agent calls).

## Step 1: Deploy the agent

```bash
nemo agents create --name email-security-triage \
--agent-config plugins/nemo-agents/examples/nemo-agent-config/email-security-triage/agent.yaml
nemo agents deploy --agent email-security-triage \
--name email-security-triage-deployment --mode subprocess
```

The deploy command waits until the deployment reports `running` on a loopback port.

## Step 2: Classify an email

```bash
nemo agents invoke --agent-deployment email-security-triage-deployment \
--input $'From: it-support@paypa1-secure.example\nSubject: Verify your account\n\nYour account is locked. Confirm your password at http://paypa1-secure.example/login'
```

The agent returns a YAML verdict with `is_likely_phishing: true`, listing the
lookalike sender domain (`paypa1-secure.example`) among its indicators.
`phishing-analyzer` owns the verdict and emits every field, including
`attack_type` (e.g. `credential`) and `impersonated_brand` (e.g. `paypal`) — it
fills those from the `attack-attributor` and `url-brand-analyst` findings, which
are advisory: it can override them, or supply a value itself when a specialist is
silent.

## Step 3: Watch the header specialist fire

`header-auth-analyst` reads SMTP authentication results, so it only runs when the
email actually carries them. Send one that does:

```bash
nemo agents invoke --agent-deployment email-security-triage-deployment \
--input $'Received: from mail.evil.example (203.0.113.9)\nFrom: security@paypal.com\nReturn-Path: bounce@evil.example\nAuthentication-Results: mx.example.com; spf=fail smtp.mailfrom=evil.example; dkim=fail header.d=paypal.com; dmarc=fail header.from=paypal.com\nSubject: Unusual sign-in\n\nReview the sign-in at http://paypal-secure-review.example/verify'
```

The authentication results fail across the board — critically `dmarc=fail` on
`header.from=paypal.com`, the check tied to the visible `From:` domain, so the
message isn't authorized to claim `paypal.com`. `header-auth-analyst` names the
failed mechanism, and it surfaces among the indicators.

The labeled dataset in Step 5 carries no SMTP headers, so this specialist stays
idle there. That is deliberate: synthesizing auth results per row would put the
`phishing`/`benign` label into the input and inflate the score.

## Step 4: Find the steps in the trace

```bash
nemo agents logs --agent email-security-triage
```

The deployment's `artifacts/.../events.atof.jsonl` records the `extract_iocs` tool
call and a task for each specialist the orchestrator consulted — evidence the tool
ran and the specialists were invoked, not that the model guessed. Which specialists
appear depends on the input: `url-brand-analyst` and `attack-attributor` run on the
Step 2 email, while `header-auth-analyst` appears only for header-bearing input
like Step 3's. With NeMo Studio Intake enabled (`VITE_FF_INTAKE_ENABLED=true`), the
same run appears under **Traces**, one span per step.

## Step 5: Evaluate against labeled emails

```bash
nemo agents evaluate run \
--eval-config plugins/nemo-agents/examples/nemo-agent-config/email-security-triage/email-security-triage-eval.yml \
--agent email-security-triage
```

The judge scores each verdict against the `label` column in
`data/smaller_test.csv` and prints an accuracy score.

## Next Steps

- **Make it your own:** [CUSTOMIZE.md](CUSTOMIZE.md) — swap the tool, prompts, specialists, model, and data for your own agent.
- **Container deploys (docker/k8s):** [docs/agents/deploy-agents.mdx](../../../../../docs/agents/deploy-agents.mdx).
- **Compare with/without a tool:** the sibling [calculator-agent](../calculator-agent) example.
Loading
Loading