Skip to content

Repository files navigation

∆ Preact AI Evaluation Console

A model quality workbench for regression suites, statistical comparison, output diffs, cost and latency analysis, and human judgments.

Preact TypeScript Tailwind CSS Statistics Vite

Deploy with Vercel


✨ Highlights

  • Suite Comparison: Contrasts baseline and candidate runs across accuracy, latency, deviation, cost, tokens, and regressions.
  • Computed Metrics: Derives summaries from individual cases through simple-statistics instead of decorative constants.
  • Regression Gate: Makes candidate failures and aggregate quality changes visible before promotion.
  • Deterministic Runs: Simulates progress and result refresh without API keys, external API calls, or model spend.
  • Case Triage: Searches examples and filters regressions, improvements, and human-review work.
  • Output Diff: Keeps the expected answer beside a word-level baseline and candidate comparison.
  • Human Judgment: Stores pass, fail, and needs-review decisions with reviewer notes.
  • Durable Mock State: Validates the suite, persists review changes in IndexedDB, and restores the fixture.
  • Responsive Evaluation UI: Uses a fixed-column desktop table, dedicated mobile case cards, and a full-screen mobile inspector without page overflow.

🧱 Project Structure

public/
|-- favicon.svg
`-- sample-evaluation-suite.json
src/
|-- app.tsx
|-- index.css
|-- main.tsx
|-- components/
|   |-- case-inspector.tsx
|   |-- case-table.tsx
|   `-- score-summary.tsx
|-- domain/
|   `-- evaluations.ts
`-- services/
    `-- evaluation-store.ts

🛠️ Tech Stack

  • Preact 10, TypeScript 6, Vite 8, and Tailwind CSS 4
  • simple-statistics, diff, Zod, Preact Signals, Lucide icons, and date-fns
  • IndexedDB persistence through idb-keyval

📦 Included Demo Data

public/sample-evaluation-suite.json contains ten synthetic policy-grounded support cases with prompts, expected answers, baseline and candidate outputs, correctness, latency, token usage, cost, and reviewer data.

No prompt is sent to a model and no API credentials are required. Every run uses deterministic browser-only demo logic without external API calls or model spend.

🚀 Local Development

npm install
npm run dev

Open http://localhost:5173.

🔌 Extension Notes & Nuances

  • Statistics are computed from individual case results; cost and token totals are not decorative constants.
  • Word-level output differences use the diff package and distinguish baseline-only from candidate-only text.
  • Evaluation execution is deterministic mock logic, making the repository safe to run without API keys or model spend.
  • Replace src/services/evaluation-store.ts with a real evaluation backend while retaining the suite, result, and judgment contracts.
  • Keep model credentials, provider calls, prompt templates, and privileged datasets on a trusted backend.
  • Version datasets, prompts, judges, model settings, and scoring rules so runs remain reproducible.
  • Add confidence intervals and significance testing before treating small metric differences as meaningful.
  • Redact sensitive traces, define retention, and separate automated scores from accountable human approval.
  • The desktop table becomes purpose-built case cards on mobile, while the inspector opens as a readable full-screen workflow.

🧪 Build & Checks

npm run format:check
npm run build
npm run preview
npm audit --omit=dev

🌐 Vercel Deployment

{
  "$schema": "https://openapi.vercel.sh/vercel.json",
  "framework": "vite",
  "buildCommand": "npm run build",
  "outputDirectory": "dist",
  "rewrites": [
    {
      "source": "/(.*)",
      "destination": "/index.html"
    }
  ]
}

📄 License

MIT License. See LICENSE.

Releases

Packages

Contributors

Languages