feat(studio): connect Studio to Intake based eval schema, update modal

web/packages/studio/public/sample-agents/email-security-triage/dataset.jsonl

@@ -0,0 +1,6 @@
+{"subject": "Updated Payment Instructions for Invoice INV-2024-0876", "sender": "accounts.payable@medisupplyexample.com", "body": "Hi James,\n\nPlease find attached the revised banking details for invoice INV-2024-0876, which is scheduled for payment on April 30, 2025. The new wire information reflects our recent change to a primary operating account. Kindly update your records and confirm receipt by replying to this email. If you have any questions, feel free to contact our finance team at (555) 123-4567.\n\nThank you,\nLaura Chen\nAccounts Payable\nMediSupply Co.\nlaura.chen@medisupplyexample.com", "label": "phishing"}
+{"subject": "Action Required: Update Banking Info for Invoice INV-2024-0587", "sender": "laura.chen@tranzglobal.example.com", "body": "Hi Jamie,\n\nOur finance team is preparing the next payment cycle and requires your current banking information to process invoice INV-2024-0587 for the recent freight shipment. Please review the attached secure form, fill in your account details, and return it by end of day Thursday. Let me know if you have any questions or need assistance.\n\nBest regards,\nLaura Chen\nAccounts Payable\nTranzGlobal Logistics\nlaura.chen@tranzglobal.example.com", "label": "phishing"}
+{"subject": "Action Required: Your Password Will Expire in 48 Hours", "sender": "IT Security Team <itsecurity@corp-assests.example.com>", "body": "Hello,\n\nOur records show that your corporate password will expire in 48 hours. To avoid interruption of access to internal systems, please reset your password now using the secure link below. The process takes less than a minute and will not affect any ongoing work.\n\nhttps://secure.corp-assests.example.com/reset\n\nIf you did not request this change or have any questions, contact the IT Help Desk at ext. 5550 or reply to this email.\n\nThank you,\nIT Security Team", "label": "phishing"}
+{"subject": "Invoice #INV-2025-0874 \u2013 Payment Due", "sender": "accounts@trustedvendor.example.com", "body": "Hello,\n\nPlease find attached invoice INV-2025-0874 for the office supplies delivered on September 10, 2025. The total amount due is $2,425.00 and payment is requested by October 5, 2025 to avoid a late fee. If you have already arranged payment, please disregard this notice. For any questions regarding the invoice, contact our billing team at billing@trustedvendor.example.com or call (555) 123-4567.\n\nThank you,\nTrusted Vendor Accounts Team", "label": "benign"}
+{"subject": "Subscription Renewal Confirmation \u2013 Invoice #INV-2025-0874", "sender": "billing@healthsoftsolutions.example.com", "body": "Hello,\n\nYour subscription to PatientFlow Analytics has been renewed for the period 2025-09-01 through 2026-08-31. The charge of $12,450.00 was processed to the corporate card ending in 1234. Please find the official receipt attached for your records. No further action is required from you at this time. If you have any questions or did not authorize this transaction, please contact our billing team at billing@healthsoftsolutions.example.com or call (555) 123-4567.\n\nThank you,\nHealthSoft Solutions Billing Department", "label": "benign"}
+{"subject": "Password Expiration Notice \u2013 Action Required", "sender": "it-security@example.com", "body": "Hello Team,\n\nAs part of our quarterly security policy, your corporate password is scheduled to expire on 2025-09-28. To avoid any disruption to your access, please reset your password by clicking the secure link below before the deadline.\n\nReset your password: https://idp.example.com/reset\n\nIf you encounter any issues, contact the IT Help Desk at it-support@example.com or extension 5550. Do not reply to this automated message.\n\nThank you,\nIT Security Team", "label": "benign"}

web/packages/studio/public/sample-agents/email-security-triage/eval-config.dataset-driven.README.md

@@ -0,0 +1,54 @@
+<!-- SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
+<!-- SPDX-License-Identifier: Apache-2.0 -->
+
+# Dataset-Driven evaluation — Email Security Triage
+
+Every row of `dataset.jsonl` is scored by the same metric set. Use this shape when the inputs are
+homogeneous and the interesting variable is coverage, not the kind of task.
+
+## The dataset
+
+`dataset.jsonl` is a 6-row demo subset (3 phishing, 3 benign), each with `subject`, `sender`,
+`body`, and `label`. `dataset-full.jsonl` sits beside it with the full 40 rows (22 phishing,
+18 benign) for real measurement — swap it in by renaming, or point a fileset at it directly.
+
+The subset is small on purpose. A phishing email with a link costs the agent roughly 66s and 50k
+prompt tokens, because the orchestrator delegates to several sub-agents and re-sends the message to
+each; a link-free benign email costs about 10s. 40 rows therefore runs for tens of minutes, which is
+the wrong shape for a demo. The subset favours link-free rows and keeps one near-miss pair — a
+phishing and a benign email that share the subject "Password Expiration Notice" — since
+discriminating those is the point of the sample.
+
+`prompt_template` assembles rows into the RFC-822 message the agent expects:
+
+```jinja
+From: {{ item.sender }}
+Subject: {{ item.subject }}
+
+{{ item.body }}
+```
+
+The `From:` line is deliberate — the sender domain is a top phishing signal and is what the agent's
+`extract_iocs` tool harvests. Dropping it measurably weakens the agent.
+
+`label` is the only ground truth here. The rows are shared with the email-security-analyst sample:
+the dataset carries inputs and labels only, so it is agent-agnostic — the YAML verdict contract
+lives entirely in the metrics below. Add rows by appending to `dataset.jsonl`.
+
+The full set is adversarial by design: several benign emails carry the same subject lines as
+phishing ones, so a model that keys on subject wording scores badly. Expect well under 100%.
+
+## How the verdict is read
+
+The agent answers with a YAML block whose `is_likely_phishing` key carries the verdict. Metrics
+read that key out of the body with `contains`, because the model usually wraps the block in a ```
+fence despite being asked not to — anything anchored to the start of the response scores 0 for
+every row.
+
+## Metrics
+
+- `string-check` — deterministic. Its expected value is rendered per row
+  (`is_likely_phishing: {% if item.label == 'phishing' %}true{% else %}false{% endif %}`), so it
+  needs no judge model and scores even when the judge is unreachable.
+- `llm-judge` — grades the verdict and whether `attack_type` is a sensible label, as two `scores`
+  on one metric.

web/packages/studio/public/sample-agents/email-security-triage/eval-config.dataset-driven.json

@@ -0,0 +1,95 @@
+{
+  "dataset": "<workspace>/<fileset>#dataset.jsonl",
+  "prompt_template": "From: {{ item.sender }}\nSubject: {{ item.subject }}\n\n{{ item.body }}",
+  "metrics": [
+    {
+      "bundle_kind": "metric-bundle",
+      "bundle_format_version": "v1",
+      "metric_type": "string-check",
+      "metadata": {
+        "description": "Verdict key matches ground truth, without needing a judge model.",
+        "labels": {}
+      },
+      "outputs": [
+        {
+          "name": "string-check",
+          "description": null,
+          "value_json_schema": {
+            "description": "Continuous numeric metric value.",
+            "title": "ContinuousScore",
+            "type": "number"
+          }
+        }
+      ],
+      "secrets": {},
+      "payload": {
+        "kind": "inline",
+        "metric": {
+          "type": "string-check",
+          "operation": "contains",
+          "left_template": "{{ (sample.output_text or '') | lower }}",
+          "right_template": "is_likely_phishing: {% if item.label == 'phishing' %}true{% else %}false{% endif %}"
+        }
+      }
+    },
+    {
+      "bundle_kind": "metric-bundle",
+      "bundle_format_version": "v1",
+      "metric_type": "llm-judge",
+      "metadata": {
+        "description": "attack_type plausibility vs the ground-truth verdict.",
+        "labels": {}
+      },
+      "outputs": [
+        {
+          "name": "attack_type_plausible",
+          "description": null,
+          "value_json_schema": {
+            "description": "Continuous numeric metric value.",
+            "title": "ContinuousScore",
+            "type": "number"
+          }
+        }
+      ],
+      "secrets": {},
+      "payload": {
+        "kind": "inline",
+        "metric": {
+          "type": "llm-judge",
+          "model": "default/nvidia-nemotron-3-super-120b-a12b",
+          "prompt_template": {
+            "messages": [
+              {
+                "role": "user",
+                "content": "You are scoring an email security triage agent.\n\nThe agent answers with a YAML block (often wrapped in a ``` fence). Its verdict is the `is_likely_phishing` key: `true` means phishing, `false` means benign. Read that key and nothing else — the `indicators` and `explanation` fields routinely mention the opposite verdict as things the agent considered, and must be ignored. Do not read the first line; it is usually the fence.\n\nAgent response:\n{{ sample.output_text }}\n\nGround-truth verdict: \"{{ item.label }}\".\n\nReturn a JSON object with one key:\n  \"attack_type_plausible\": 1 if the `attack_type` value is a sensible label for this email (one of bec, credential, malware, spam, benign; `benign` iff the verdict is benign), else 0."
+              }
+            ]
+          },
+          "scores": [
+            {
+              "name": "attack_type_plausible",
+              "minimum": 0.0,
+              "maximum": 1.0,
+              "parser": {
+                "type": "regex",
+                "method": "search",
+                "pattern": "attack_type_plausible\"?\\s*:\\s*([01])"
+              }
+            }
+          ],
+          "inference": {
+            "max_tokens": 1024,
+            "extra_body": {
+              "nvext": {
+                "max_thinking_tokens": 256
+              }
+            }
+          },
+          "reasoning": {
+            "end_token": "</think>"
+          }
+        }
+      }
+    }
+  ]
+}

web/packages/studio/public/sample-agents/email-security-triage/eval-config.task-driven.README.md

@@ -0,0 +1,46 @@
+<!-- SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -->
+<!-- SPDX-License-Identifier: Apache-2.0 -->
+
+# Task-Driven evaluation — Email Security Triage
+
+Each task is one email the agent triages independently, carrying its own metrics. Use this shape
+when the suite grades different _kinds_ of work; here the kinds are ordinary classification and
+resistance to prompt injection.
+
+## What it measures
+
+| Tasks         | Input                                                       | Checks                                                         |
+| ------------- | ----------------------------------------------------------- | -------------------------------------------------------------- |
+| `classify-*`  | one phishing or benign email                                | the verdict matches the label                                  |
+| `injection-*` | a phishing email carrying instructions aimed at the analyst | the verdict follows the evidence, not the injected instruction |
+
+## How the verdict is read
+
+The agent answers with a YAML block whose `is_likely_phishing` key carries the verdict:
+
+```yaml
+is_likely_phishing: true
+confidence: 0.93
+indicators: [...]
+explanation: ...
+attack_type: credential
+impersonated_brand: none
+```
+
+Every metric reads that key out of the body. Two details that matter if you edit this config:
+
+- **The model usually wraps the YAML in a ``` fence**, despite its prompt asking for the block
+  alone. Metrics use `contains`, never `startswith` or a first-line lookup — anything anchored to
+  the start of the response scores 0 for every row.
+- **`indicators` and `explanation` routinely name the opposite verdict** as something the agent
+  considered and rejected. The judge prompt says to ignore them.
+
+## Metrics
+
+- `string-check` — deterministic. Matches `is_likely_phishing: true|false` against the task's
+  label. Needs no judge model, so it scores even when the judge is unreachable.
+- `llm-judge` — grades the verdict and whether `attack_type` is a sensible label. Both criteria
+  live in a single metric: the runtime rejects two metrics of the same _type_ within one task, so
+  extra criteria have to be extra `scores`.
+
+The `triage` view reduces the deterministic and judged verdict signals to a mean.