Improve l8r agent behavior and eval coverage

[philhetzel]

Summary

This PR improves the l8r customer-service agent using production-observability findings and stricter eval coverage.

Changes included:

• tightened agent scope and refusal behavior for off-topic, unsupported, and prompt-injection requests
• added targeted fast paths for:
  • gibberish / malformed input
  • credit-limit increase requests
  • payment-hardship guidance with account-specific installment details
• normalized ambiguous numeric order references like order 2 -> order_002
• switched eval coverage to the stricter L8r Agent Eval Dataset
• expanded local eval scoring to better check behavior match and grounding
• synced the Braintrust topic-adherence prompt/scorer definition used in the flywheel run

Why

Production traces showed two concrete problems:

• unsupported / off-topic requests could still drift into account answers
• ambiguous order references and hardship flows were brittle in simulation

The flywheel also found stale measurement:

• the topic-adherence scorer could false-positive when the assistant answered the wrong latest user request but still talked about l8r

Files

• src/lib/chatbot/system-prompt.ts
• src/lib/chatbot/chat.ts
• src/lib/chatbot/index.ts
• src/app/api/chat/route.ts
• src/lib/services/order-service.ts
• evals/conversation.eval.ts
• evals/sim.eval.ts
• braintrust/l8r-customer-service.ts

Verification

Local:

• npx tsc --noEmit
• npm run lint
  • passes with one existing next/image warning in src/components/ui/avatar.tsx

Braintrust:

• updated topic-adherence scorer pushed
• updated prompt params pushed

Sim eval smoke runs:

• l8r-sim-eval-2026-06-30
  • GoalAchievement: 85%
  • QualityCheck: 75%
  • ResolutionCheck: 100%
• l8r-sim-eval-2026-06-30-d4d2f515
  • GoalAchievement: 85%
  • QualityCheck: 85%
  • ResolutionCheck: 100%

Notes

This branch intentionally excludes unrelated local worktree changes and contains only the flywheel-related edits.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
l8r	Ready	Preview, Comment	Jun 30, 2026 7:47pm