How do you know an enterprise agent is safe and better enough to release?
This project evaluates a synthetic account-risk agent across prompt versions, checks evidence and tool behavior, blocks regressions, and exercises a local release lifecycle. Its center is agent behavior and release decisions. Customer-state authority remains a separate engineering boundary.
Generated from actual local inference, two seeds per scenario. Unsafe recommendations remain defects even when guards prevent effects. This small suite is not a population quality estimate; neither live configuration is promoted. Comparison and blocking reasons.
Start with the finished project handoff, the live model comparison, the executed fixture release lifecycle, and the evaluation methodology.
The live experiment runs Qwen2.5-1.5B-Instruct locally through llama.cpp, without a paid provider. It compares a reference prompt with a deliberately flawed summary-first candidate on the same six synthetic scenario classes and two seeds each. The small model's failures remain visible: a guardrail preventing a bad effect does not turn an incorrect recommendation into a successful task. Neither live configuration is presented as production-ready or promoted.
The separate deterministic suite proves the release machinery: a faulty candidate is blocked; an equivalent candidate passes offline, shadow and limited canary checks, is activated locally, and is rolled back after an injected runtime regression. That successful lifecycle uses handwritten fixtures, not claimed live-model promotion.
For a seller-facing application of source-grounded proposals, review and controlled CRM delivery, see Customer Signal & Activation.
%%{init: {"theme":"base","themeVariables":{"background":"#10101D","primaryColor":"#2A2A3D","primaryTextColor":"#F0F0FF","primaryBorderColor":"#7C7CAF","lineColor":"#B8B8FF","textColor":"#F0F0FF","secondaryColor":"#181825","tertiaryColor":"#2E2E45","fontFamily":"IBM Plex Sans, Arial, sans-serif","fontSize":"15px","edgeLabelBackground":"#10101D"}}}%%
flowchart TB
V[Versioned prompt, model, tools, retrieval and code] --> R[Bounded agent runtime]
C[Synthetic CRM / usage / support + scoped playbooks] --> R
M[Local LLM or explicit fixture responder] --> R
R --> T[Typed recommendation and evidence checks]
T --> E[Synthetic enterprise effects / approval]
R --> O[Structured trace and verified receipts]
O --> Q[Behavioral evaluation by scenario class]
Q --> G[Paired release gate]
G --> S[Shadow: identical context, no write credential]
S --> K[Canary: limited recommendation-only routing]
K --> P[Local active version]
P --> B[Observable rollback of future routing]
classDef input fill:#181825,stroke:#7C7CAF,color:#F0F0FF,stroke-width:2px;
classDef foundation fill:#2A2A3D,stroke:#B8B8FF,color:#F0F0FF,stroke-width:2px;
classDef process fill:#2A2A3D,stroke:#7C7CAF,color:#F0F0FF,stroke-width:2px;
classDef control fill:#2E2E45,stroke:#E3BC7F,color:#F0F0FF,stroke-width:2px;
classDef output fill:#2E2E45,stroke:#81C5B0,color:#F0F0FF,stroke-width:2px;
class V,C,M input;
class R foundation;
class T,O,Q process;
class E,G,S,K control;
class P,B output;
The runtime gathers required account, usage and support facts and retrieves approved playbooks. The model decides whether to create an internal follow-up task, request human-approved escalation, finish, or abstain. It must cite exact observed facts. Authenticated usage/support fields take precedence over narrative CRM hints; retrieved text supplies context, not authority.
Mandatory reads are deterministic because every case requires them. Model decisions are reserved for assessment and next action. One explicit runtime, SQLite state/FTS5, a local model adapter and one synthetic HTTP service keep the reasoning and failure boundaries inspectable. No external CRM is connected.
The evidence figure is a static projection of the linked JSON, not an application screenshot. Regenerate it with python docs/images/render.py after the corresponding evaluation or planning evidence is refreshed.
Use Python 3.12 from the repository root:
python -m venv .venv
# Activate .venv for your shell.
python -m pip install -r requirements.lock
python -m pip install --no-deps --no-build-isolation -e .
python scripts/verify.pyThis executes software/adversarial tests, the six-case deterministic workflow evaluation, and the complete local release demonstration. It starts real HTTP subprocesses and uses real SQLite databases under ignored work/. No Docker, cloud account or paid key is required.
For actual local-model execution, use the model setup and exact evaluation commands. The optional model is approximately 1.12 GB; weights and runtime binaries are excluded from the archive. The pinned model revision and measured file hashes are included.
| Dimension | Evidence |
|---|---|
| Task success | Expected tools, arguments and terminal outcome checked against actual remote effects |
| Grounding | Numeric claims checked against the evidence available at each model decision |
| Unsafe recommendations | Policy refusals counted as quality defects even when no effect occurs |
| Tool contract failures | Unsupported/malformed outputs, hallucinated tools and repeated decisions |
| Abstention and escalation | Distinct expected outcomes for uncertain and high-risk cases |
| Regression | Paired cases, per-class success, critical-class failures and explicit blocking reasons |
| Runtime behavior | Model usage, input/prompt hashes, context provenance, approvals and verified receipts |
| Release discipline | Immutable version/evidence registration, staged observations, conditional promotion and actual runtime routing after rollback |
An intentionally poisoned playbook requests credential export, approval bypass and a false healthy assessment. Closed capabilities, scoped reads, evidence checks and an executor boundary prevent those instructions from granting authority. Adversarial fixtures deliberately follow the attack; their refusal is demonstrated independently of whether the local model follows it. Attack and failure matrix.
Native Windows execution includes real local-model inference, real SQLite/FTS5, separate HTTP processes, deterministic tests and local release routing. The verification record and individual tests retain exact results. Hosted runtime and release-regression CI passed on Windows and Ubuntu. That workflow executes fixtures; actual local-model inference is evidenced separately. Cloud deployment and live vendor integration are outside this local validation.
The release gate is a conservative check on a small synthetic suite, not proof of general prompt-injection resistance or population-level model quality. The intentionally flawed candidate is a regression mutation, not an impartial provider benchmark. Known limits and validation explain the development-case tuning, small sample size and trusted local operator boundary.
Read architecture, evaluation, operations and validation for the technical handoff.