A model quality workbench for regression suites, statistical comparison, output diffs, cost and latency analysis, and human judgments.
- Suite Comparison: Contrasts baseline and candidate runs across accuracy, latency, deviation, cost, tokens, and regressions.
- Computed Metrics: Derives summaries from individual cases through simple-statistics instead of decorative constants.
- Regression Gate: Makes candidate failures and aggregate quality changes visible before promotion.
- Deterministic Runs: Simulates progress and result refresh without API keys, external API calls, or model spend.
- Case Triage: Searches examples and filters regressions, improvements, and human-review work.
- Output Diff: Keeps the expected answer beside a word-level baseline and candidate comparison.
- Human Judgment: Stores pass, fail, and needs-review decisions with reviewer notes.
- Durable Mock State: Validates the suite, persists review changes in IndexedDB, and restores the fixture.
- Responsive Evaluation UI: Uses a fixed-column desktop table, dedicated mobile case cards, and a full-screen mobile inspector without page overflow.
public/
|-- favicon.svg
`-- sample-evaluation-suite.json
src/
|-- app.tsx
|-- index.css
|-- main.tsx
|-- components/
| |-- case-inspector.tsx
| |-- case-table.tsx
| `-- score-summary.tsx
|-- domain/
| `-- evaluations.ts
`-- services/
`-- evaluation-store.ts
- Preact 10, TypeScript 6, Vite 8, and Tailwind CSS 4
- simple-statistics, diff, Zod, Preact Signals, Lucide icons, and date-fns
- IndexedDB persistence through
idb-keyval
public/sample-evaluation-suite.json contains ten synthetic policy-grounded support cases with prompts, expected answers, baseline and candidate outputs, correctness, latency, token usage, cost, and reviewer data.
No prompt is sent to a model and no API credentials are required. Every run uses deterministic browser-only demo logic without external API calls or model spend.
npm install
npm run devOpen http://localhost:5173.
- Statistics are computed from individual case results; cost and token totals are not decorative constants.
- Word-level output differences use the
diffpackage and distinguish baseline-only from candidate-only text. - Evaluation execution is deterministic mock logic, making the repository safe to run without API keys or model spend.
- Replace
src/services/evaluation-store.tswith a real evaluation backend while retaining the suite, result, and judgment contracts. - Keep model credentials, provider calls, prompt templates, and privileged datasets on a trusted backend.
- Version datasets, prompts, judges, model settings, and scoring rules so runs remain reproducible.
- Add confidence intervals and significance testing before treating small metric differences as meaningful.
- Redact sensitive traces, define retention, and separate automated scores from accountable human approval.
- The desktop table becomes purpose-built case cards on mobile, while the inspector opens as a readable full-screen workflow.
npm run format:check
npm run build
npm run preview
npm audit --omit=dev{
"$schema": "https://openapi.vercel.sh/vercel.json",
"framework": "vite",
"buildCommand": "npm run build",
"outputDirectory": "dist",
"rewrites": [
{
"source": "/(.*)",
"destination": "/index.html"
}
]
}MIT License. See LICENSE.