Skip to content

About

Reproducible, provider-agnostic LLM benchmark for domain-specific AI performance evaluation

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

ForgeBench

CI Python 3.11+ Python 3.12+ Version License: MIT Stripe: sandbox Status: MVP Code style: ruff Tests: pytest Contributor Covenant OpenRouter Ollama Mistral OpenAI Anthropic Google Built with Claude Code

Measure domain adaptation, not just model intelligence.

ForgeBench is a reproducible, provider-agnostic benchmark harness for answering one question: did customizing a model β€” a better prompt, few-shot examples, structured output, a fine-tune β€” actually improve performance on a specific domain task, or did it just make the pipeline more complicated?

The first benchmark domain is financial and transaction operations: synthetic banking, accounting, claims, and e-commerce scenarios (transaction exception classification and reconciliation triage). No private data and no GPU required. Benchmark runs need a provider key or a local Ollama daemon; custom tasks can validate the pipeline offline with --provider mock.

Who it's for: teams shipping AI in finance, healthcare, legal, insurance, payments, procurement and support operations β€” anyone who needs "we think the new prompt is better" replaced with a pinned, auditable, cost-measured comparison on a fixed test set. See PITCH.md for the full business case: value proposition, problems solved, commercial model.

Status: MVP (v0.2.0). Actively developed; expect breaking changes between versions. Monetization is credit-based ($0.10/credit, runs counted) with the Stripe loop wired in this repository β€” nothing billed until the Stripe account is configured (Pricing).


πŸ“Έ ForgeBench in Action

Dashboard β€” Provider & Model Selection

ForgeBench dashboard β€” API key view with Mistral selected and its default model

Run Details & Trace Waterfall

Run detail view with per-case traces

Regression Comparison β€” Baseline vs. Experiment

Compare two runs side-by-side with metric deltas and case-level diff

Results Table with Evidence

Results table showing predictions and ground truth

Datasets Explorer

Datasets view with splits and manifests

Task Library

14 built-in evaluation tasks

Light Theme & Mobile Responsive

Light Theme Mobile Dashboard Mobile Ollama
Light theme Mobile dashboard Mobile Ollama

Setup Wizard

Setup wizard β€” provider configuration


✨ Key Features

Feature Description
Fixed, versioned synthetic test sets Deterministic ground truth generated by rules, never by a model. Every split has a SHA-256 recorded in the manifest.
Pluggable provider interface Mistral, OpenAI, Anthropic, Google, OpenRouter and Ollama adapters share one contract, plus a deterministic rule engine used only by the offline unit tests. OpenRouter β€” one key, every hosted model β€” is the default in the dashboard and CLI.
Strict output validation A plausible explanation is not a correct answer. Unknown categories, malformed JSON, and invalid evidence fields are rejected.
Risk-aware metrics Critical false-match rate and escalation recall are tracked separately β€” accuracy alone hides the expensive errors.
Full run manifests Dataset hash, config, model id, versions, timestamps, git SHA, and environment are captured for every run β€” including the prompt that actually ran (name@version, SHA-256, provider default vs. registry vs. inline). Task runs also pin the SHA-256 of task.yaml and the case file, plus the case limit. If a run uses a different provider than its config asked for, or a judge could not be built, the manifest and report say so.
Regression view Pin a run per dataset as its baseline, then open any later run and choose Vs baseline: metric deltas plus the exact cases that improved or regressed β€” the number that moves and the cases that moved it.
Per-case trace waterfall Every run writes trace.jsonl: wall-clock spans per case (provider call with token counts, parse & score, judge). The run modal's View trace tab renders it, so "which step cost the time" is an answer, not a guess.
Versioned prompt registry Prompts live in prompts/<name>/vN.md with immutable versions and labels (production). Pin one from a config with prompt: {name: …, version: …} or from the dashboard, and the run records the exact bytes it used.
Machine-readable + human-readable results predictions.jsonl, metrics.json, manifest.json, trace.jsonl, and report.md per run.
Bring-your-own-key (BYOK) API keys saved to a git-ignored .env (chmod 600), verified against the provider API, never committed, never in run artifacts.
Local dashboard FastAPI-served, Hallmark-designed UI with a dark-first Cobalt theme, side-rail navigation, and a ⌘K jump palette β€” environment setup with per-provider key entry, key management, dataset browsing, run launching, and results browsing where every run opens a detail modal (report, predictions, downloadable artifacts) plus cross-run comparison.
Deterministic dataset generator Same seed, byte-identical files. Regenerate finance_v1 or add new datasets with scripts/generate_dataset.py. CI regenerates all generated data with make check-data and fails on any difference.
User-defined tasks Define any evaluation task in YAML β€” input fields, output schema, weighted scoring rules β€” and run it with forgebench task-run. No code required.

πŸ’Ό Why ForgeBench for business

Business benefit How it lands
Lower model cost Cost per 1,000 cases is a headline metric in every run and comparison, computed from actual token usage with dated pricing β€” pick the cheapest model that clears the risk metrics instead of debating vendor decks.
Fewer production incidents Regression gating on PR: if a prompt or model change degrades a pinned baseline past tolerance, CI fails the change β€” not your customers. The financially dangerous error (a false "matched") is a first-class metric.
Faster model decisions Swap providers with --provider on the same config; decide between models in an afternoon, not a week of ad-hoc evaluation.
No ML infrastructure No GPU, no database, no paid platform. Python 3.11, flat files, bring-your-own-key β€” or fully local inference via Ollama with zero egress.
Compliance-ready evidence Deterministic synthetic ground truth (no customer PII in the test path), SHA-256 dataset manifests, prompt hashes, and sanitized self-contained HTML exports safe to attach to an audit file.
Team-readable output CLI for automation, local dashboard for humans, Markdown reports and JSON/CSV exports for whoever needs the numbers.

The tool itself is proprietary (see LICENSE): the Community tier is free for internal evaluation β€” you pay only the model provider you configure, exactly for the tokens used, and the zero-cost path (offline mock provider, local Ollama) keeps pipeline validation spend-free.

🏷️ Pricing: credit packs (Stripe)

Credits are the unit of purchase: $0.10 per credit. Metering: a benchmark run costs 30 credits ($3.00) flat β€” one price whatever the case limit β€” and a custom-task run costs 1 credit ($0.10) per case (each case is one model call). Runs are counted, not seats. Credits are charged after the data loads and the provider is built, so a run that cannot start costs nothing. A dataset pack can be bought from the same wallet. The Stripe fulfillment loop is wired end-to-end in this repository: checkout β†’ webhook (services/stripe_webhook) β†’ single-use voucher code β†’ forgebench redeem FB-… β†’ local credit ledger (.forgebench/credits.json, git-ignored).

Package Credits Price
Starter 50 $5
Growth 100 $10
Scale 250 $25
Max 500 $50
Domain dataset pack 250 $25
Enterprise Unlimited unlimited $5,000 one-time
forgebench credits            # balance, mode, grants
forgebench redeem FB-SCALE-250-AB12CD34   # one-time redemption β†’ local balance

Status: the catalog, webhook listener, voucher store and redeem flow are implemented and tested in this repository; checkout goes live as soon as the Stripe account is configured (payment links + STRIPE_WEBHOOK_SECRET/STRIPE_PRICE_MAP β€” see services/stripe_webhook/README.md). Until then nothing is billed. Enterprise also covers SSO/on-prem priorities and shapes the shared-backend roadmap; the fuller business case is in PITCH.md.

πŸš€ Quickstart

Requires Python 3.11+.

git clone https://github.com/seyhunak/forgebench
cd forgebench
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

End-to-end pipeline

# 1. Validate the synthetic dataset
forgebench validate-dataset finance_v1

# 2. Save a key (OpenRouter by default), or use a local Ollama daemon
forgebench set-key --provider openrouter

# 3. Run the baseline and a prompt variant
forgebench run --config configs/baseline.yaml --run-id baseline
forgebench run --config configs/experiments/prompt_v1.yaml --run-id prompt-v1

# 4. Compare the two runs
forgebench compare runs/baseline runs/prompt-v1

Each run writes predictions.jsonl, metrics.json, manifest.json, trace.jsonl, and report.md under runs/<run-id>/.

Without a key, use a local model instead: forgebench run --config configs/baseline.yaml --provider ollama --model <your-pulled-model> β€” or omit --model entirely: Ollama picks up OLLAMA_MODEL from your .env, falling back to beam:cloud.

Running against real models (bring your own key)

# Save a key interactively (hidden prompt, verified against the provider API).
# OpenRouter is the default provider β€” pass --provider for another.
forgebench set-key --provider openrouter
forgebench set-key --provider mistral
forgebench set-key --provider openai
forgebench set-key --provider anthropic
forgebench set-key --provider google
forgebench set-key --provider ollama      # local/cloud Ollama β€” key optional

# Save a default model id with the key
forgebench set-key --model ministral-8b-2512 --provider mistral
forgebench set-key --model gpt-4o --provider openai
forgebench set-key --model claude-3-5-sonnet --provider anthropic
forgebench set-key --model gemini-2.5-pro --provider google
forgebench set-key --model anthropic/claude-3.5-sonnet --provider openrouter

# Check masked status
forgebench set-key --show

# Browse the catalog (OpenRouter alone has hundreds of models)
forgebench models --provider openrouter --search claude
forgebench models --provider openrouter --live   # straight from openrouter.ai

# Run with a real provider
forgebench run --config configs/baseline.yaml --provider openrouter --run-id baseline
forgebench run --config configs/baseline.yaml --provider openai --model gpt-4o --run-id openai-baseline
forgebench run --config configs/baseline.yaml --provider openrouter --model google/gemini-2.5-pro --run-id openrouter-run

An exported MISTRAL_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY, or OPENROUTER_API_KEY in your shell takes precedence over .env. Config references like ${MISTRAL_MODEL} resolve from either.

Local dashboard (optional UI)

pip install -e ".[ui]"
forgebench dashboard   # http://127.0.0.1:8502

The dashboard is a single-page app served by FastAPI. Navigate with the left rail, or press ⌘K / Ctrl K to jump straight to any view:

  • 01 Setup β€” the first stop: a BYOK checklist of providers β€” OpenRouter, Google, Anthropic, OpenAI, Ollama (local, key optional), Mistral β€” one row each for its *_API_KEY (and default *_MODEL), with masked set/not-set status and source. Fill in the keys you use and save β€” everything is written to the git-ignored .env (chmod 600), never committed. Any other environment variable can be added below the provider list.
  • 02 API key β€” provider selector (Mock is gone from this list β€” it needs no key, and runs that name it are upgraded to the selected provider β€” and Ollama is listed with an install/running check: pick it and the view reports whether the daemon is up and fills the model dropdown with your locally pulled models). OpenRouter leads (the default: one key, every hosted model). Picking a provider fills the model dropdown with its full catalog from models.dev and preselects a sensible model for it (e.g. openai/gpt-4o-mini for OpenRouter), so a first run works without hunting for an id. Save, verify, or remove a key, with live masked status and cost/context details. (The screenshot shows Mistral and its default open-mistral-nemo β€” that is the key saved in the demo environment.)
  • 03 Datasets β€” browse datasets and splits, inspect case tables, download JSON.
  • 04 Config β€” edit experiment YAML, validated before saving.
  • 05 Run β€” pick a config, optional case limit, live progress bar, run id.
  • 06 Results β€” run cards that open a detail modal: metrics, the full Markdown report, a predictions table, and Markdown/JSON downloads. Plus N-way comparison with colour-coded deltas and a History view (every run over time, grouped by task/dataset, with per-group deltas).
  • 07 Tasks β€” validate and run user-defined evaluation tasks; provider defaults to OpenRouter with a model prefilled. Each task run writes the same artifacts as a benchmark run (predictions, metrics, manifest, report, trace).

Sharing the dashboard safely

By default the dashboard is a local, single-user tool. To expose it (tail tunnel, container, LAN), set a token and every /api/* request must carry it:

FORGEBENCH_TOKEN=$(openssl rand -hex 24) forgebench dashboard
# the UI prompts for the token once (401) and stores it in localStorage;
# API clients send:  x-forgebench-token: <token>

It ships a dark-first Cobalt theme with a light mode (toggle, top-right), and calls the exact same runner and reporting code as the CLI β€” so artifacts are identical. (The maintenance cost of keeping a second, parallel dashboard is why src/forgebench/dashboard.py functions as the shared data layer β€” list_runs, read_predictions, read_trace β€” consumed by both the FastAPI app and the CLI; the Streamlit entry point inside it is legacy and no longer the primary UI.)

Results β€” run cards Compare runs β€” deltas
Results view with run cards Compare runs view with metric deltas
Run detail β€” modal report Run detail β€” predictions
Run detail modal with the Markdown report and download buttons Predictions table inside the run detail modal
Datasets Custom tasks
Dataset browser Custom tasks view
Setup β€” provider keys Setup β€” narrow layout
Setup view listing every provider API key with masked status Narrow layout of the Setup view
API key β€” Ollama install check Ollama β€” narrow layout
API key view with Ollama selected, daemon status and local models Narrow layout of the API key view with Ollama selected
Light theme Narrow layout
Light theme Narrow layout β€” API key view with Mistral and its default model

πŸ“‹ CLI Reference

Command What it does
forgebench version Print the installed version
forgebench set-key [--provider mistral|openai|anthropic|google|openrouter] [--model MODEL] [--show|--clear|--verify/--no-verify] Save an API key to git-ignored .env (chmod 600), optionally verify against the provider API. Default provider: openrouter
forgebench models [--provider <p>] [--live] [--search <text>] [--limit N] List a provider's models (OpenRouter's full catalog; --live fetches from openrouter.ai)
forgebench list-datasets List available datasets
forgebench validate-dataset <name> Validate a dataset (schema, ids, splits, families, class distribution)
forgebench task-list [--tasks-root <path>] List user-defined evaluation tasks under tasks/
forgebench task-validate <name> [--tasks-root <path>] Validate a task manifest and its case files
forgebench task-run <name> [--split <s>] [--provider <p>] [--model <id>] [--models "a,b,c"] [--limit N] [--judge-provider <p>] [--judge-model <id>] Run a custom task evaluation end-to-end. With --models, every model runs and a comparison.md matrix lands next to the first run (baseline). Default provider: openrouter
forgebench run --config <yaml> [--provider <name>] [--model <id>] [--limit N] [--run-id <id>] [--datasets-root <path>] [--output-root <path>] Execute one benchmark end-to-end
forgebench prompt-list [--prompts-root <path>] List versioned prompts in the registry (prompts/)
forgebench prompt-show <name> [--version N] [--prompts-root <path>] Print one prompt version (default: latest)
forgebench prompt-save <name> [--file <path>] [--note <text>] [--label <text>] [--prompts-root <path>] Write the next immutable version of a prompt
forgebench prompt-label <name> <label> <version> [--prompts-root <path>] Point a label (e.g. production) at a version; 0 removes the label
forgebench history [--runs-root <path>] [--limit N] Chronological trend of every run, grouped by task/dataset, with deltas
forgebench compare <run-a> <run-b> [...] [--out <file>] N-way comparison with deltas vs. the first run (baseline). Works for benchmark runs and task runs
forgebench report <run-dir> [--out <file>] (Re)generate the Markdown report for a run
forgebench export <run-dir> [--format json|csv|html] [--out <file>] Export predictions + metrics (JSON/CSV) or a standalone sanitized HTML report
forgebench safety-scan [--split <s>] [--provider <p>] [--model <id>] [--limit N] Run the safety red-team task (injection, jailbreak, PII, bias, toxicity)
forgebench dashboard Launch the local web UI
FORGEBENCH_TOKEN=... forgebench dashboard Same, but every /api/* request requires the x-forgebench-token header (the UI prompts for it)
FORGEBENCH_WEBHOOK_URL=... forgebench run --config ... POST a compact run summary to the URL after every completed run

πŸ—οΈ Providers

All providers implement the same predict(case) -> ProviderResponse contract:

Provider Adapter Models API key
Mock providers/mock.py rule-mock-v1 None β€” offline deterministic rule engine, used by unit tests only. A benchmark run that names it is upgraded to the selected provider, with a CLI warning and a provider_substitution entry in the manifest. task-run --provider mock stays offline, for validating the pipeline only.
Mistral providers/mistral.py ministral-8b, ministral-3b, ministral-14b, mistral-small-4, mistral-large, mistral-medium MISTRAL_API_KEY
OpenAI providers/openai.py GPT-5, GPT-4.1, GPT-4o, o1, o3-mini families OPENAI_API_KEY
Anthropic providers/anthropic.py Claude 4 Opus, Claude 4 Sonnet, Claude 3.5 Sonnet, Claude 3 Opus, Claude 3 Haiku ANTHROPIC_API_KEY
Google providers/google.py Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 2.0 Flash, Gemini 1.5 Pro, Gemini 1.5 Flash GOOGLE_API_KEY
OpenRouter providers/openrouter.py Every model OpenRouter hosts (hundreds), addressed as vendor/model β€” e.g. anthropic/claude-3.5-sonnet, openai/gpt-4o, meta-llama/llama-3.1-70b-instruct, google/gemini-2.5-pro OPENROUTER_API_KEY
Ollama providers/ollama.py (OllamaProvider) β€” local daemon (also cloud-served :cloud models) Whatever you have pulled. Default: OLLAMA_MODEL env, else beam:cloud OLLAMA_API_KEY (optional for local Ollama) / OLLAMA_HOST

The OpenRouter adapter speaks the REST API directly with the standard library, so it works on a bare install (no extra SDK). Its catalog is available from forgebench models --provider openrouter (the pinned snapshot) or --live (straight from openrouter.ai/api/v1/models).

OpenRouter is the default provider. The dashboard preselects it, and forgebench set-key / forgebench task-run use it unless --provider says otherwise. Config-driven forgebench run takes the provider from the config file (the bundled configs use OpenRouter), and an empty selection falls back to the first provider with a saved key β€” a run never silently degrades to the offline rule engine.

Model lists are synchronized from models.dev and include pricing (input/output/cache read $/M tokens), context windows, and capabilities.

Example: switching providers without changing config

# Same config, different provider
forgebench run --config configs/baseline.yaml --provider ollama --model beam:cloud --run-id ollama-run
forgebench run --config configs/baseline.yaml --provider mistral --run-id mistral-run
forgebench run --config configs/baseline.yaml --provider openai --model gpt-4o --run-id openai-run
forgebench run --config configs/baseline.yaml --provider anthropic --model claude-3-5-sonnet --run-id anthropic-run
forgebench run --config configs/baseline.yaml --provider google --model gemini-2.5-pro --run-id google-run
forgebench run --config configs/baseline.yaml --provider openrouter --model anthropic/claude-3.5-sonnet --run-id openrouter-run

πŸ“Š Datasets

Dataset Task Cases Splits Version
finance_v1 Bank/ledger exception classification 200 100 train / 40 dev / 60 test 1.0.0
healthcare_v1 Medical claims reconciliation 125 80 train / 30 dev / 15 test 1.0.0
ecommerce_v1 Marketplace order vs ERP reconciliation 125 80 train / 30 dev / 15 test 1.0.0
legal_v1 Contract clause reconciliation and exception triage 105 42 train / 21 dev / 42 test 1.0.0
insurance_v1 Insurance claims adjudication and exception triage 105 42 train / 21 dev / 42 test 1.0.0
support_v1 Customer support ticket triage and exception reconciliation 105 42 train / 21 dev / 42 test 1.0.0
payments_v1 Payment settlement reconciliation and fraud/exception triage 105 42 train / 21 dev / 42 test 1.0.0
procurement_v1 Purchase-order vs. supplier-invoice reconciliation 105 42 train / 21 dev / 42 test 1.0.0
subscriptions_v1 SaaS subscription billing reconciliation 105 42 train / 21 dev / 42 test 1.0.0

All nine datasets reuse the same input field names as finance_v1, so they run through the existing pipeline with no code changes. Each has a matching config under configs/ (for example configs/legal_baseline.yaml, configs/payments_baseline.yaml).

All datasets share six scenario families:

Family Meaning
matched Amounts, dates, and references agree
amount_mismatch Amounts differ (takes precedence when other fields also differ)
date_mismatch Dates differ, amounts agree
reference_mismatch References differ, amounts agree
duplicate One side shows more occurrences than the other
ambiguous Conflicting weak signals; human review required

Dataset files live under datasets/<name>/:

  • dataset.json β€” manifest (name, version, task, license, families, splits, SHA-256 hashes)
  • dataset_card.md β€” human-readable documentation
  • train.jsonl, dev.jsonl, test.jsonl β€” JSONL split files with ground-truth labels

View any dataset in the dashboard's Datasets view, or via the API:

curl http://localhost:8502/api/datasets
curl http://localhost:8502/api/datasets/healthcare_v1
curl http://localhost:8502/api/datasets/finance_v1/splits/test

Regenerate datasets deterministically:

# finance_v1 (and its variants)
python scripts/generate_dataset.py --seed 42 --train 100 --dev 40 --test 60

# legal_v1, insurance_v1, support_v1, payments_v1, procurement_v1, subscriptions_v1 (fixed seed 2026)
python scripts/generate_multi_dataset.py

# saas_billing_v1, telecom_v1, travel_v1
python scripts/generate_more_datasets.py

# Regenerate everything with a generator, and fail if the committed data differs
make check-data

healthcare_v1 and ecommerce_v1 have no generator in this repository, so a test pins their split hashes instead.

πŸ“ˆ Metrics

Metric Definition Why it matters
Accuracy Correct / schema-valid classifications Overall performance
Macro-F1 Mean F1 across classes with ground truth Common classes can't dominate
Critical false-match rate Truly non-matching cases wrongly called matched The financially dangerous error β€” a false "matched" means money is assumed reconciled when it isn't
Escalation recall Cases requiring review that were flagged for review Missed exceptions that need human attention
Schema validity Structured outputs passing the Prediction schema Integration reliability β€” models that can't emit valid JSON are useless in pipelines
Latency (p50/p95) Median and tail response time Operational performance
Estimated cost per 1,000 cases Provider pricing Γ— tokens used Budget comparison across providers/models

Denominator conventions and evaluation rules are specified in docs/benchmark_protocol.md.

Run history (trend over time)

A comparison answers "did candidate B beat baseline A". History answers the question you ask a month later: is this task or dataset getting better as I keep working on it?

forgebench history              # every run, grouped by task/dataset, with deltas
curl localhost:8501/api/history # the same data as JSON (the dashboard's History view renders it)

Each row carries the headline score (accuracy for benchmarks, valid rate for task runs) and Ξ” vs previous β€” the delta against the previous run of the same group. Click a row in the dashboard to open that run's report.

Report exports

forgebench report  runs/<id> --out report.md   # Markdown (per-case rows include the test data)
forgebench export  runs/<id> --format html --out report.html   # standalone sanitized HTML page
forgebench export  runs/<id> --format csv  --out results.csv
forgebench export  runs/<id> --format json > results.json

Webhooks

Set FORGEBENCH_WEBHOOK_URL (a Slack incoming-webhook URL, CI listener, or any collector) and every completed run β€” CLI or dashboard β€” POSTs one compact JSON summary (run_id, model, dataset/task, cases, accuracy). Delivery failures warn on stderr and never fail the run.

Regression view (run vs. baseline)

Pin one run per dataset as the baseline, then open any later run and choose Vs baseline in the run modal:

# via the API (the dashboard's pin button posts exactly this)
curl -X POST localhost:8501/api/baselines \
  -H 'Content-Type: application/json' \
  -d '{"dataset": "finance_v1", "run_id": "runs/baseline-run-id"}'

curl localhost:8501/api/runs/new-run-id/regression

The comparison answers both questions a metric delta can't: which scalar metrics moved, and which cases flipped (improved / regressed / still correct / still wrong, plus cases added or dropped). Pins live in runs/baselines.json β€” local state, never committed. The CLI equivalent is forgebench compare runs/baseline runs/new for a Markdown report.

βš™οΈ Configuration

Experiments are YAML files under configs/. Example β€” configs/baseline.yaml:

experiment:
  name: baseline
  dataset: finance_v1
  dataset_version: "1.0.0"
  split: test

provider:
  name: openrouter
  model: openai/gpt-4o-mini
  options:
    temperature: 0.0

inference:
  temperature: 0
  max_retries: 2
  timeout_seconds: 30
  max_tokens: 1024

prompt:
  name: reconciler      # optional: pull the system prompt from prompts/reconciler/
  version: null          # null/omitted = latest version
  system: null           # inline text; ignored when prompt.name is set
  few_shot_count: 0

evaluation:
  metrics:
    - accuracy
    - macro_f1
    - critical_false_match_rate
    - escalation_recall
    - schema_validity
    - latency
    - estimated_cost

Config values support ${ENV_VAR} interpolation (e.g. "${MISTRAL_MODEL}"). Missing variables fail loudly β€” never silently.

The provider is optional in practice: an empty/missing selection falls back to the first provider with a saved key, and the model falls back to the *_MODEL value saved with forgebench set-key.

Prompt registry

Prompts live as immutable files under prompts/<name>/vN.md with named labels:

forgebench prompt-list
forgebench prompt-save reconciler --file my_system_prompt.md --note "tighter rules" --label production
forgebench prompt-show reconciler --version 2
forgebench prompt-label reconciler production 3    # 0 removes the label

A config that sets prompt: {name: reconciler} runs whatever the registry resolves (highest version unless version: pins one), and the run's manifest records the prompt's SHA-256 plus the digest of the text the provider actually sent β€” so two runs can be proven to have used the same instructions.

Configs can be edited in the dashboard's Config view, validated with Pydantic before saving.

πŸ“ Repository Layout

forgebench/
β”œβ”€β”€ README.md                    # This file
β”œβ”€β”€ LICENSE                      # Proprietary β€” ForgeBench Commercial License
β”œβ”€β”€ CODE_OF_CONDUCT.md           # Community covenant
β”œβ”€β”€ CONTRIBUTING.md              # How to contribute
β”œβ”€β”€ SECURITY.md                  # Security policy
β”œβ”€β”€ Makefile                     # install, test, lint, data, demo, dashboard
β”œβ”€β”€ pyproject.toml               # Project metadata, dependencies, scripts
β”œβ”€β”€ .env.example                 # Environment variable template (git-ignored .env)
β”œβ”€β”€ .gitignore                   # Python, secrets, run artifacts
β”œβ”€β”€ configs/                     # Experiment YAML configs
β”‚   β”œβ”€β”€ baseline.yaml            # OpenRouter reference baseline
β”‚   β”œβ”€β”€ ecommerce_baseline.yaml  # OpenRouter baselines for the ecommerce, healthcare,
β”‚   β”œβ”€β”€ healthcare_baseline.yaml #   insurance, legal, payments, procurement,
β”‚   β”œβ”€β”€ insurance_baseline.yaml  #   saas_billing, subscriptions, support, telecom
β”‚   β”œβ”€β”€ legal_baseline.yaml      #   and travel datasets
β”‚   β”œβ”€β”€ payments_baseline.yaml
β”‚   β”œβ”€β”€ procurement_baseline.yaml
β”‚   β”œβ”€β”€ saas_billing_baseline.yaml
β”‚   β”œβ”€β”€ subscriptions_baseline.yaml
β”‚   β”œβ”€β”€ support_baseline.yaml
β”‚   β”œβ”€β”€ telecom_baseline.yaml
β”‚   β”œβ”€β”€ travel_baseline.yaml
β”‚   └── experiments/
β”‚       β”œβ”€β”€ prompt_v1.yaml       # Prompt variant
β”‚       └── fewshot_v1.yaml      # Few-shot variant
β”œβ”€β”€ prompts/                     # Versioned prompt registry (immutable vN.md + labels.json)
β”‚   └── reconciler/              # Provider default system prompt, pinned as v1
β”œβ”€β”€ datasets/                    # Versioned synthetic datasets
β”‚   β”œβ”€β”€ finance_v1/              # Bank/ledger exception classification
β”‚   β”œβ”€β”€ healthcare_v1/           # Medical claims reconciliation
β”‚   β”œβ”€β”€ ecommerce_v1/            # Marketplace order vs ERP
β”‚   β”œβ”€β”€ legal_v1/                # Contract clause reconciliation
β”‚   β”œβ”€β”€ insurance_v1/            # Insurance claims adjudication
β”‚   β”œβ”€β”€ support_v1/              # Support ticket triage
β”‚   β”œβ”€β”€ payments_v1/             # Payment settlement / fraud triage
β”‚   β”œβ”€β”€ procurement_v1/          # Purchase order vs supplier invoice
β”‚   └── subscriptions_v1/        # SaaS subscription billing
β”œβ”€β”€ tasks/                       # User-defined evaluation tasks (task.yaml + JSONL)
β”‚   β”œβ”€β”€ sentiment_analysis/      # Classification with confidence and key phrases
β”‚   β”œβ”€β”€ summarization_qa/        # Summary accuracy and hallucination checks
β”‚   β”œβ”€β”€ code_review/             # Bug/security/style review with severity
β”‚   β”œβ”€β”€ intent_classification/   # Support intent, confidence and entities
β”‚   β”œβ”€β”€ named_entity_extraction/ # Typed entity extraction from free text
β”‚   └── contract_clause_detection/ # Clause type, risk level, counsel flag
β”œβ”€β”€ docs/                        # Architecture, protocol, integration notes, dashboard screenshots
β”œβ”€β”€ services/                    # Business services (not part of the harness)
β”‚   └── stripe_webhook/          # Stripe checkout β†’ voucher β†’ credit fulfillment (devctl-managed)
β”œβ”€β”€ scripts/                     # Deterministic dataset generators
β”‚   β”œβ”€β”€ generate_dataset.py      # Regenerate finance_v1
β”‚   └── generate_multi_dataset.py # Generate the six 105-case datasets
β”œβ”€β”€ src/forgebench/              # Source code
β”‚   β”œβ”€β”€ __init__.py              # Package, version
β”‚   β”œβ”€β”€ cli.py                   # Typer CLI
β”‚   β”œβ”€β”€ config.py                # Pydantic experiment config
β”‚   β”œβ”€β”€ runner.py                # Load β†’ validate β†’ infer β†’ score β†’ record
β”‚   β”œβ”€β”€ task_runner.py           # Custom task evaluation loop
β”‚   β”œβ”€β”€ tasks.py                 # Task registry (locate, load, hash, validate)
β”‚   β”œβ”€β”€ registry.py              # Dataset registry (locate, load, hash, validate)
β”‚   β”œβ”€β”€ schemas.py               # BenchmarkCase, Prediction, Category, TaskDefinition enums
β”‚   β”œβ”€β”€ credentials.py           # BYOK storage (.env, chmod 600)
β”‚   β”œβ”€β”€ judge.py                 # G-Eval style LLM-as-judge scoring
β”‚   β”œβ”€β”€ evidence.py              # Evidence field validation
β”‚   β”œβ”€β”€ parsing.py               # Structured output parsing
β”‚   β”œβ”€β”€ providers/               # Provider adapters
β”‚   β”‚   β”œβ”€β”€ __init__.py
β”‚   β”‚   β”œβ”€β”€ base.py              # ModelProvider protocol
β”‚   β”‚   β”œβ”€β”€ mock.py              # Deterministic rule engine
β”‚   β”‚   β”œβ”€β”€ mistral.py           # Mistral API adapter
β”‚   β”‚   β”œβ”€β”€ openai.py            # OpenAI API adapter
β”‚   β”‚   β”œβ”€β”€ anthropic.py         # Anthropic API adapter
β”‚   β”‚   β”œβ”€β”€ google.py            # Google API adapter
β”‚   β”‚   β”œβ”€β”€ openrouter.py        # OpenRouter adapter (OpenAI-compatible, stdlib HTTP)
β”‚   β”‚   └── verify.py            # Shared provider API-key verification
β”‚   β”œβ”€β”€ metrics/                 # Metric computation (classification, performance, pricing, reliability, rag)
β”‚   β”œβ”€β”€ reporting/               # Markdown + JSON report renderers
β”‚   β”œβ”€β”€ api.py                   # FastAPI backend for the dashboard
β”‚   β”œβ”€β”€ dashboard.py             # Legacy Streamlit entry point (kept for compat)
β”‚   └── static/                  # Dashboard frontend (index.html, style.css, app.js, shell.js, models.json)
β”œβ”€β”€ tests/                       # Unit + end-to-end tests (no API key needed)
└── runs/                        # Local run artifacts (demo/evidence runs may be committed)

πŸ§ͺ Testing

pip install -e ".[dev]"
pytest

The suite runs offline: tests that exercise the run pipeline inject a deterministic fake at the provider seam, so nothing spends API credits or flakes on the network. One smoke test hits a real provider and skips unless OPENROUTER_API_KEY is set.

🀝 Contributing

See CONTRIBUTING.md. Good first contributions:

  • Additional provider adapters (Cohere, Cohere Command, xAI Grok, etc.)
  • Synthetic datasets for new domains (legal, insurance, supply chain)
  • New metrics
  • Independently reproduced runs with published report.md outputs
  • Dashboard improvements (light/dark theme refinements, export formats)

All contributions follow the Contributor Covenant.

πŸ”’ Security

API keys are never stored in the repository. See SECURITY.md for the full policy, including how keys are stored (git-ignored .env, chmod 600), verified against provider APIs, and excluded from run artifacts.


Evaluation features

ForgeBench implements LLM evaluation capabilities inspired by common open-source evaluation frameworks:

LLM-as-judge (G-Eval style)

Subjective scoring criteria evaluated by any configured provider (Mistral, OpenAI, Anthropic, Google, OpenRouter, Ollama). Define criteria and chain-of-thought evaluation steps per scoring rule:

scoring_rules:
  - field: summary_quality
    method: llm_judge
    weight: 2.0
    threshold: 0.7
    criteria: "Determine whether the summary captures all key facts without hallucination."
    evaluation_steps:
      - "Compare the summary against the source document."
      - "Penalize any claim not supported by the source."
      - "Score highly when meaning and key details match."
judge:                       # which provider/model judges the llm_judge rules
  provider: openai
  model: gpt-4o-mini

Benchmark runs can also add a judge on top of the rule-based metrics via the evaluation.judge section β€” every schema-valid prediction's reason gets scored against your criterion, and metrics["judge"] (avg score, pass rate) appears in the report and comparisons:

evaluation:
  metrics: [accuracy, macro_f1, escalation_recall, schema_validity]
  judge:                      # optional LLM-as-judge (G-Eval style)
    provider: openrouter
    model: openai/gpt-4o-mini
    # options: {}             # provider options (e.g. the mock judge takes none)
    # criteria: >-            # override the default reason-quality criterion

The judge is advisory: a broken or unparseable judge never fails a run β€” failures are counted in metrics["judge"]["n_failures"] and the run continues on the rule-based metrics alone.

RAG metrics (RAGAS style)

Deterministic lexical metrics for retrieval-augmented generation, no LLM calls needed:

  • faithfulness β€” share of the answer grounded in retrieved context
  • answer_relevancy β€” vocab overlap between answer and question
  • contextual_relevancy β€” share of retrieved chunks relevant to the question
scoring_rules:
  - field: answer
    method: faithfulness
    threshold: 0.8
  - field: answer
    method: answer_relevancy
    threshold: 0.5

Set retrieval_context on each test case (in the JSONL expected block or case input) to enable context-aware scoring.

Custom task builder

Define arbitrary evaluation tasks in YAML without code β€” input fields, output schema (string/number/boolean/enum/list[string]), scoring rules with weights and thresholds, plus G-Eval criteria:

forgebench task-list
forgebench task-validate sentiment_analysis
forgebench task-run sentiment_analysis --split test --provider openrouter --judge-provider openai --judge-model gpt-4o-mini

Fourteen tasks ship with the repo under tasks/:

Task What it evaluates Scoring methods
sentiment_analysis Feedback sentiment with confidence and key phrases exact_match, contains, f1
summarization_qa Summary accuracy, completeness, hallucination list boolean, exact_match, f1
code_review Bug/security/style review with severity and fix flag exact_match, boolean, contains
intent_classification Support intent with confidence and extracted entities exact_match, contains, f1
named_entity_extraction Typed entities from free text with count and primary type f1, exact_match
contract_clause_detection Clause type, risk level and counsel flag from contract text exact_match, boolean
email_classification Email category (spam/ham/phishing) with confidence and reason exact_match, contains, contains
topic_classification Article topic with confidence exact_match, contains
question_type Question type and whether external context is needed exact_match, boolean
safety_redteam Red-team probes: prompt injection, jailbreak, PII, bias, toxicity boolean, exact_match
expense_reimbursement Expense claim review: approve/reject/flag, reimbursable amount exact_match, contains
invoice_matching Invoice vs. purchase order: match status and review flag exact_match, boolean, contains
purchase_order_approval Purchase order decision and approver routing exact_match, contains
ticket_triage Support ticket priority, owning team and response SLA exact_match, contains

Safety / red-team scan

A red-team workflow is available via forgebench safety-scan (prompt injection, jailbreaking, PII leakage, bias, toxicity); --provider mock runs it fully offline for pipeline validation.

Regression-on-PR CI

templates/regression.yml is a copy-in GitHub Actions template: an offline gate (lint, tests, dataset reproducibility) plus a task regression gate that compares a pinned baseline run against a fresh candidate run and fails the PR beyond a tolerance.

Changelog

See CHANGELOG.md for what shipped in each version.


πŸ“š Documentation Index

All linked markdown files in the docs/ folder:

Document Description
architecture.md System architecture, data flow, and component design
benchmark_protocol.md Evaluation methodology, metric definitions, and denominator conventions
forge_integration.md Integration guide for Forge platform
OPPORTUNITY-FRAMEWORK.md Strategic opportunity analysis and framework
PITCH.md Business case, value proposition, and commercial model
results.md Benchmark results and analysis
WORKFLOW.md Development workflow, CI/CD, and release process
CHANGELOG.md Version history and release notes
CODE_OF_CONDUCT.md Community code of conduct (Contributor Covenant v2.1)
CONTRIBUTING.md Contribution guidelines and development setup
SECURITY.md Security policy and vulnerability reporting
05_execute_rag.md RAG execution guide and examples

πŸ“œ License

ForgeBench is proprietary software at MVP stage β€” see LICENSE. A final commercial license has not been published: today there are no licenses on sale and no commercial use terms; the software is free for internal evaluation, while redistribution, a competing managed service and team-scale deployment need an agreement with the copyright holder. The pricing section above describes planned tiers, not an active offer. Model weights and third-party datasets/APIs carry their own terms β€” check each model card and license independently. Synthetic data in datasets/ was generated by this repository and follows the same terms.


ForgeBench β€” measure domain adaptation, not just model intelligence.

About

Reproducible, provider-agnostic LLM benchmark for domain-specific AI performance evaluation

Resources

Code of conduct

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages