ops-agent

A customer support agent built on Claude that can look up orders, track shipments, and issue refunds — with the guardrails, memory, and testing you'd actually want before letting an AI touch a refund button.

I built this to learn how production agentic systems really work, end to end: not just "call an LLM," but the whole picture — tool use, a multi-agent coordinator, deterministic safety limits, semantic memory, and an automated evaluation suite that scores the agent's answers.

What it does

Ask it something like "Where's my order #12345, and refund me if it's late?" and it will:

1. Look up the order and its live shipping status (chaining two tools — the second needs the tracking number the first returns).
2. Decide whether a refund is actually warranted.
3. Refuse to issue a refund over $100 without human approval — enforced in code, not just asked of the model.
4. Pull in anything it remembers about the customer (e.g. "VIP — always ship express") and use it.
5. Return one clean, coherent reply.

Why it's more than a demo

A lot of "AI agent" projects stop at a single LLM call wrapped in a loop. The parts I care about here are the ones that make an agent trustworthy:

• Deterministic guardrails. The model can decide to refund $5000 all it wants — a hard check in the tool path blocks it regardless. The safety doesn't depend on the model "behaving."
• Eval-driven development. There's an LLM-as-judge harness that scores every answer against a golden dataset (accuracy, helpfulness, safety). When I refactored to a multi-agent design, the evals caught a regression I'd never have spotted by eye — and caught a second one when I fixed the first.
• Semantic memory. Customer facts are matched by meaning, not keywords, using local embeddings — so "deliver quickly" finds the "express shipping" note even with no shared words.
• Least-privilege workers. The status worker physically doesn't have the refund tool. It can't misuse what it doesn't have.

Architecture

            user request
                 │
         ┌───────▼────────┐
         │  Coordinator    │   decides which workers to use, then
         │                 │   synthesizes their findings into ONE reply
         └───────┬────────┘
        ┌─────────┴──────────┐
        ▼                    ▼
 ┌──────────────┐    ┌──────────────────┐
 │ Status Worker│    │ Refund Worker    │
 │ (read tools) │    │ (refund + guard) │
 └──────┬───────┘    └────────┬─────────┘
        │                     │
        └─────────┬───────────┘
                  ▼
   ┌──────────────────────────────┐
   │ shared agent loop            │
   │ think → act → observe        │
   │ + tool registry              │
   │ + deterministic guardrail    │
   │ + semantic memory injection  │
   └──────────────────────────────┘

Each worker is just the core agent loop handed a scoped set of tools. The coordinator decomposes the request, delegates focused sub-tasks, and writes a single coherent reply instead of stapling the workers' answers together.

Project layout

ops_agent/
├── client.py        # one shared Anthropic client
├── tools.py         # the tools (find_order, get_shipping_status, issue_refund) + registry
├── guardrails.py    # the deterministic refund limit — the hard lock
├── memory.py        # semantic recall over customer notes (embeddings)
├── loop.py          # the agent loop: think → act → observe
├── workers.py       # scoped worker agents (status, refund)
└── coordinator.py   # decides who handles what, then synthesizes one reply
tests/
├── golden_cases.py  # the eval dataset (the "answer key")
├── run_evals.py     # the eval harness: run agent → judge → scorecard
└── test_guardrail.py# deterministic unit test for the refund limit
main.py              # entry point

Running it

# 1. Set up a virtual environment and install
python -m venv .venv
.venv\Scripts\Activate.ps1        # Windows PowerShell
pip install -e .

# 2. Add your Anthropic API key
#    create a .env file with:  ANTHROPIC_API_KEY=sk-ant-...

# 3. Run the agent
python main.py

Testing & evaluation

Two kinds of tests, because an agent needs both:

# Deterministic test — for the predictable machinery (does the guardrail block a $5000 refund?)
python tests/test_guardrail.py

# Eval suite — for the chatty, probabilistic part (is the answer accurate / helpful / safe?)
python tests/run_evals.py

The eval suite runs the full agent against each golden case, has a judge model score the answers on a rubric, and prints a pass rate. A case passes only if it clears the bar on accuracy, helpfulness, and safety.

A few design decisions worth calling out

• Safety lives in code, not the prompt. Telling the model "don't refund over $100" is a request it can be argued out of. The real limit is a deterministic check in the tool path that the model can't talk past.
• Tests for the machine, evals for the meaning. You can't assert answer == "expected" on text that's worded differently every time. So predictable logic (the guardrail) gets strict unit tests; the model's actual answers get scored by a judge against a rubric.
• Multi-agent wasn't free. Splitting one agent into a coordinator + workers bought better scoping, but introduced coordination problems (the workers contradicted each other at first). The eval suite is what made that tradeoff visible and fixable.

Built with

Python · Anthropic Claude API · sentence-transformers (local embeddings) · a lot of reading error messages and fixing them.

---

This started as a structured learning project to go deep on agentic AI engineering. Every piece was built from scratch and is something I can explain — the point was understanding, not just getting it running.