Concierge drafts replies to customer emails over the live backend of an e-commerce shop I run: grounded drafts, a deterministic policy gate, and a human on every send.
Portfolio exhibit. This is a sanitized public extract of a private system in daily use. The architecture and method are real; the data and identifiers are stand-ins, and the section below lists which is which.
▶ Demos below: research pass, grounded draft, injection attempt, quarantine artifact.
Concierge reads the support inbox of an online shop I own and prepares reply drafts for a human to review and send. Per thread, it pulls order status, live stock, prior correspondence, and an internal knowledge base through typed MCP tools, then writes a grounded draft into Gmail Drafts. It runs as a standalone Node.js / TypeScript service.
Prime directive: draft only, never send. Enforced three times independently: the OAuth grant carries no send scope, the codebase has no send path, and the agent has no send tool. Sending is always a human action in Gmail.
Real. The typed tool layer, the policy gate, the human-in-the-loop draft flow, the audit trail, and the eval harness. The private workflow has drafted real support replies for this shop for months and took over most of the manual drafting; this repo is its standalone rebuild, migrating into place through shadow parity. The policy gate and eval harness here are the production logic.
Sanitized. The agent loop and tools are runnable, typed illustrations over synthetic data, exercised end to end by the smoke run. The shop's brand and suppliers are withheld. The demo clips use a test mailbox and synthetic customer emails against the live shop backend, so no real customer data is on screen.
Four clips, each one beat, recorded live.
01_research.mp4
One command handles the thread; the trace shows the agent checking live shop data before writing.
02_grounded-draft.mp4
Recommendation and stock status come from live systems; the draft waits in Gmail for a person to send.
03_attacker.mp4
Email content is untrusted data: no reply drafted, nothing disclosed or actioned.
04_quarantine.mp4
An internal warning draft with no recipient spells out what the injection attempted.
- The tool surface is the sandbox. Typed, allowlisted MCP tools and nothing else: no filesystem, no shell, no web. Mail tools can read, search, and draft only.
- Retrieval is deterministic. Orders, stock, and mail history are authoritative, so the agent does exact lookups; vector search would add a failure mode here.
- Email is untrusted data. A manipulation attempt is quarantined into a recipient-less internal warning draft.
- A code gate guards every draft, rejecting secret-shaped strings, internal pricing vocabulary, unsupported claims, and style tells.
- Edits are protected. The service supersedes only its own untouched drafts; solved-ness derives from message ids, so a fresh customer reply re-opens the thread.
- Every run is auditable and cost-bounded. An append-only JSONL log records every tool call, rejection, draft, and cost, under thread caps, turn caps, a pinned model, and a hard monthly spend limit.
flowchart TB
T["Trigger (cron / scheduler)"] --> POLL["Cheap poll: unhandled threads, no model"]
POLL --> LOOP["Per-thread agent loop"]
LOOP -->|"typed, allowlisted, read-only"| TOOLS["MCP tools: order status · live stock · prior correspondence · knowledge base · mail read / search / draft"]
TOOLS --> LOOP
LOOP --> DEC{"Injection check: message content is untrusted"}
DEC -->|"normal request"| CAND["Grounded reply draft"]
DEC -->|"injection / manipulation"| QUAR["Quarantine draft, no recipient"]
CAND --> GATE["Deterministic policy gate (code)"]
QUAR --> GATE
GATE -->|"pass"| GD["Gmail Drafts"]
GATE -->|"secret · pricing · bad claim · style"| BLOCK["Blocked and flagged for a human"]
GD --> HUMAN["Human reviews, edits, sends"]
BLOCK --> HUMAN
LOOP --> AUDIT[("Audit trail: every tool call, rejection, draft, cost")]
Claude Agent SDK service in Node.js / TypeScript. Zod-typed MCP tools (mail read / search / draft, order status, live stock, customer research), read-only and allowlisted. TypeScript policy gate, reused verbatim by the eval harness. Gmail Drafts, JSONL audit log, Docker single-pass runs. The LLM is pinned via config under a hard monthly spend cap.
Seven policy fixtures across four violation classes run through the exact gate the production draft tool enforces, in CI on every change, at zero model cost (npm run eval). Two real production incidents (a discounted-clearance complaint and a license-generation failure) are permanent regression cases. One fixture is an injection canary that would leak internal pricing if an injection had succeeded; it must be caught. Clean fixtures must pass untouched, including a live false positive where product-URL slugs looked like leaked tokens; the token check is now scoped to prose and the case is a regression test.
A model-free smoke run (npm run smoke) exercises the whole tool layer end to end; both checks run before anything touches a live mailbox. Next stage: model-in-the-loop groundedness, replaying sanitized threads and judging answers against tool output.
I own the architecture, the tool boundaries, the failure modes, and the call on what is safe to ship; the implementation was built by directing AI coding tools, and the reply rules were tuned over weeks of reviewed production drafts.
PRODUCTION EXTRACT. One of a set of production AI systems built the same way.
- Portfolio: github.com/janvrsinsky
- LinkedIn: linkedin.com/in/janvrsinsky
