Measure domain adaptation, not just model intelligence.
ForgeBench is a reproducible, provider-agnostic benchmark harness for answering one question: did customizing a model β a better prompt, few-shot examples, structured output, a fine-tune β actually improve performance on a specific domain task, or did it just make the pipeline more complicated?
The first benchmark domain is financial and transaction operations: synthetic banking, accounting, claims, and e-commerce scenarios (transaction exception classification and reconciliation triage). No private data and no GPU required. Benchmark runs need a provider key or a local Ollama daemon; custom tasks can validate the pipeline offline with --provider mock.
Who it's for: teams shipping AI in finance, healthcare, legal, insurance, payments, procurement and support operations β anyone who needs "we think the new prompt is better" replaced with a pinned, auditable, cost-measured comparison on a fixed test set. See PITCH.md for the full business case: value proposition, problems solved, commercial model.
Status: MVP (v0.2.0). Actively developed; expect breaking changes between versions. Monetization is credit-based ($0.10/credit, runs counted) with the Stripe loop wired in this repository β nothing billed until the Stripe account is configured (Pricing).
| Light Theme | Mobile Dashboard | Mobile Ollama |
|---|---|---|
![]() |
![]() |
![]() |
| Feature | Description |
|---|---|
| Fixed, versioned synthetic test sets | Deterministic ground truth generated by rules, never by a model. Every split has a SHA-256 recorded in the manifest. |
| Pluggable provider interface | Mistral, OpenAI, Anthropic, Google, OpenRouter and Ollama adapters share one contract, plus a deterministic rule engine used only by the offline unit tests. OpenRouter β one key, every hosted model β is the default in the dashboard and CLI. |
| Strict output validation | A plausible explanation is not a correct answer. Unknown categories, malformed JSON, and invalid evidence fields are rejected. |
| Risk-aware metrics | Critical false-match rate and escalation recall are tracked separately β accuracy alone hides the expensive errors. |
| Full run manifests | Dataset hash, config, model id, versions, timestamps, git SHA, and environment are captured for every run β including the prompt that actually ran (name@version, SHA-256, provider default vs. registry vs. inline). Task runs also pin the SHA-256 of task.yaml and the case file, plus the case limit. If a run uses a different provider than its config asked for, or a judge could not be built, the manifest and report say so. |
| Regression view | Pin a run per dataset as its baseline, then open any later run and choose Vs baseline: metric deltas plus the exact cases that improved or regressed β the number that moves and the cases that moved it. |
| Per-case trace waterfall | Every run writes trace.jsonl: wall-clock spans per case (provider call with token counts, parse & score, judge). The run modal's View trace tab renders it, so "which step cost the time" is an answer, not a guess. |
| Versioned prompt registry | Prompts live in prompts/<name>/vN.md with immutable versions and labels (production). Pin one from a config with prompt: {name: β¦, version: β¦} or from the dashboard, and the run records the exact bytes it used. |
| Machine-readable + human-readable results | predictions.jsonl, metrics.json, manifest.json, trace.jsonl, and report.md per run. |
| Bring-your-own-key (BYOK) | API keys saved to a git-ignored .env (chmod 600), verified against the provider API, never committed, never in run artifacts. |
| Local dashboard | FastAPI-served, Hallmark-designed UI with a dark-first Cobalt theme, side-rail navigation, and a βK jump palette β environment setup with per-provider key entry, key management, dataset browsing, run launching, and results browsing where every run opens a detail modal (report, predictions, downloadable artifacts) plus cross-run comparison. |
| Deterministic dataset generator | Same seed, byte-identical files. Regenerate finance_v1 or add new datasets with scripts/generate_dataset.py. CI regenerates all generated data with make check-data and fails on any difference. |
| User-defined tasks | Define any evaluation task in YAML β input fields, output schema, weighted scoring rules β and run it with forgebench task-run. No code required. |
| Business benefit | How it lands |
|---|---|
| Lower model cost | Cost per 1,000 cases is a headline metric in every run and comparison, computed from actual token usage with dated pricing β pick the cheapest model that clears the risk metrics instead of debating vendor decks. |
| Fewer production incidents | Regression gating on PR: if a prompt or model change degrades a pinned baseline past tolerance, CI fails the change β not your customers. The financially dangerous error (a false "matched") is a first-class metric. |
| Faster model decisions | Swap providers with --provider on the same config; decide between models in an afternoon, not a week of ad-hoc evaluation. |
| No ML infrastructure | No GPU, no database, no paid platform. Python 3.11, flat files, bring-your-own-key β or fully local inference via Ollama with zero egress. |
| Compliance-ready evidence | Deterministic synthetic ground truth (no customer PII in the test path), SHA-256 dataset manifests, prompt hashes, and sanitized self-contained HTML exports safe to attach to an audit file. |
| Team-readable output | CLI for automation, local dashboard for humans, Markdown reports and JSON/CSV exports for whoever needs the numbers. |
The tool itself is proprietary (see LICENSE): the Community tier is free for internal evaluation β you pay only the model provider you configure, exactly for the tokens used, and the zero-cost path (offline mock provider, local Ollama) keeps pipeline validation spend-free.
Credits are the unit of purchase: $0.10 per credit. Metering: a benchmark run costs 30 credits ($3.00) flat β one price whatever the case limit β and a custom-task run costs 1 credit ($0.10) per case (each case is one model call). Runs are counted, not seats. Credits are charged after the data loads and the provider is built, so a run that cannot start costs nothing. A dataset pack can be bought from the same wallet. The Stripe fulfillment loop is wired end-to-end in this repository: checkout β webhook (services/stripe_webhook) β single-use voucher code β forgebench redeem FB-β¦ β local credit ledger (.forgebench/credits.json, git-ignored).
| Package | Credits | Price |
|---|---|---|
| Starter | 50 | $5 |
| Growth | 100 | $10 |
| Scale | 250 | $25 |
| Max | 500 | $50 |
| Domain dataset pack | 250 | $25 |
| Enterprise Unlimited | unlimited | $5,000 one-time |
forgebench credits # balance, mode, grants
forgebench redeem FB-SCALE-250-AB12CD34 # one-time redemption β local balanceStatus: the catalog, webhook listener, voucher store and redeem flow are implemented and tested in this repository; checkout goes live as soon as the Stripe account is configured (payment links + STRIPE_WEBHOOK_SECRET/STRIPE_PRICE_MAP β see services/stripe_webhook/README.md). Until then nothing is billed. Enterprise
also covers SSO/on-prem priorities and shapes the shared-backend roadmap; the fuller business case is in PITCH.md.
Requires Python 3.11+.
git clone https://github.com/seyhunak/forgebench
cd forgebench
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"# 1. Validate the synthetic dataset
forgebench validate-dataset finance_v1
# 2. Save a key (OpenRouter by default), or use a local Ollama daemon
forgebench set-key --provider openrouter
# 3. Run the baseline and a prompt variant
forgebench run --config configs/baseline.yaml --run-id baseline
forgebench run --config configs/experiments/prompt_v1.yaml --run-id prompt-v1
# 4. Compare the two runs
forgebench compare runs/baseline runs/prompt-v1Each run writes predictions.jsonl, metrics.json, manifest.json, trace.jsonl, and report.md under runs/<run-id>/.
Without a key, use a local model instead: forgebench run --config configs/baseline.yaml --provider ollama --model <your-pulled-model> β or omit --model entirely: Ollama picks up OLLAMA_MODEL from your .env, falling back to beam:cloud.
# Save a key interactively (hidden prompt, verified against the provider API).
# OpenRouter is the default provider β pass --provider for another.
forgebench set-key --provider openrouter
forgebench set-key --provider mistral
forgebench set-key --provider openai
forgebench set-key --provider anthropic
forgebench set-key --provider google
forgebench set-key --provider ollama # local/cloud Ollama β key optional
# Save a default model id with the key
forgebench set-key --model ministral-8b-2512 --provider mistral
forgebench set-key --model gpt-4o --provider openai
forgebench set-key --model claude-3-5-sonnet --provider anthropic
forgebench set-key --model gemini-2.5-pro --provider google
forgebench set-key --model anthropic/claude-3.5-sonnet --provider openrouter
# Check masked status
forgebench set-key --show
# Browse the catalog (OpenRouter alone has hundreds of models)
forgebench models --provider openrouter --search claude
forgebench models --provider openrouter --live # straight from openrouter.ai
# Run with a real provider
forgebench run --config configs/baseline.yaml --provider openrouter --run-id baseline
forgebench run --config configs/baseline.yaml --provider openai --model gpt-4o --run-id openai-baseline
forgebench run --config configs/baseline.yaml --provider openrouter --model google/gemini-2.5-pro --run-id openrouter-runAn exported MISTRAL_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY, GOOGLE_API_KEY, or OPENROUTER_API_KEY in your shell takes precedence over .env. Config references like ${MISTRAL_MODEL} resolve from either.
pip install -e ".[ui]"
forgebench dashboard # http://127.0.0.1:8502The dashboard is a single-page app served by FastAPI. Navigate with the left rail, or press βK / Ctrl K to jump straight to any view:
- 01 Setup β the first stop: a BYOK checklist of providers β OpenRouter, Google, Anthropic, OpenAI, Ollama (local, key optional), Mistral β one row each for its
*_API_KEY(and default*_MODEL), with masked set/not-set status and source. Fill in the keys you use and save β everything is written to the git-ignored.env(chmod 600), never committed. Any other environment variable can be added below the provider list. - 02 API key β provider selector (Mock is gone from this list β it needs no key, and runs that name it are upgraded to the selected provider β and Ollama is listed with an install/running check: pick it and the view reports whether the daemon is up and fills the model dropdown with your locally pulled models). OpenRouter leads (the default: one key, every hosted model). Picking a provider fills the model dropdown with its full catalog from models.dev and preselects a sensible model for it (e.g.
openai/gpt-4o-minifor OpenRouter), so a first run works without hunting for an id. Save, verify, or remove a key, with live masked status and cost/context details. (The screenshot shows Mistral and its defaultopen-mistral-nemoβ that is the key saved in the demo environment.) - 03 Datasets β browse datasets and splits, inspect case tables, download JSON.
- 04 Config β edit experiment YAML, validated before saving.
- 05 Run β pick a config, optional case limit, live progress bar, run id.
- 06 Results β run cards that open a detail modal: metrics, the full Markdown report, a predictions table, and Markdown/JSON downloads. Plus N-way comparison with colour-coded deltas and a History view (every run over time, grouped by task/dataset, with per-group deltas).
- 07 Tasks β validate and run user-defined evaluation tasks; provider defaults to OpenRouter with a model prefilled. Each task run writes the same artifacts as a benchmark run (predictions, metrics, manifest, report, trace).
By default the dashboard is a local, single-user tool. To expose it (tail
tunnel, container, LAN), set a token and every /api/* request must carry
it:
FORGEBENCH_TOKEN=$(openssl rand -hex 24) forgebench dashboard
# the UI prompts for the token once (401) and stores it in localStorage;
# API clients send: x-forgebench-token: <token>It ships a dark-first Cobalt theme with a light mode (toggle, top-right), and calls the exact same runner and reporting code as the CLI β so artifacts are identical. (The maintenance cost of keeping a second, parallel dashboard is why src/forgebench/dashboard.py functions as the shared data layer β list_runs, read_predictions, read_trace β consumed by both the FastAPI app and the CLI; the Streamlit entry point inside it is legacy and no longer the primary UI.)
| Command | What it does |
|---|---|
forgebench version |
Print the installed version |
forgebench set-key [--provider mistral|openai|anthropic|google|openrouter] [--model MODEL] [--show|--clear|--verify/--no-verify] |
Save an API key to git-ignored .env (chmod 600), optionally verify against the provider API. Default provider: openrouter |
forgebench models [--provider <p>] [--live] [--search <text>] [--limit N] |
List a provider's models (OpenRouter's full catalog; --live fetches from openrouter.ai) |
forgebench list-datasets |
List available datasets |
forgebench validate-dataset <name> |
Validate a dataset (schema, ids, splits, families, class distribution) |
forgebench task-list [--tasks-root <path>] |
List user-defined evaluation tasks under tasks/ |
forgebench task-validate <name> [--tasks-root <path>] |
Validate a task manifest and its case files |
forgebench task-run <name> [--split <s>] [--provider <p>] [--model <id>] [--models "a,b,c"] [--limit N] [--judge-provider <p>] [--judge-model <id>] |
Run a custom task evaluation end-to-end. With --models, every model runs and a comparison.md matrix lands next to the first run (baseline). Default provider: openrouter |
forgebench run --config <yaml> [--provider <name>] [--model <id>] [--limit N] [--run-id <id>] [--datasets-root <path>] [--output-root <path>] |
Execute one benchmark end-to-end |
forgebench prompt-list [--prompts-root <path>] |
List versioned prompts in the registry (prompts/) |
forgebench prompt-show <name> [--version N] [--prompts-root <path>] |
Print one prompt version (default: latest) |
forgebench prompt-save <name> [--file <path>] [--note <text>] [--label <text>] [--prompts-root <path>] |
Write the next immutable version of a prompt |
forgebench prompt-label <name> <label> <version> [--prompts-root <path>] |
Point a label (e.g. production) at a version; 0 removes the label |
forgebench history [--runs-root <path>] [--limit N] |
Chronological trend of every run, grouped by task/dataset, with deltas |
forgebench compare <run-a> <run-b> [...] [--out <file>] |
N-way comparison with deltas vs. the first run (baseline). Works for benchmark runs and task runs |
forgebench report <run-dir> [--out <file>] |
(Re)generate the Markdown report for a run |
forgebench export <run-dir> [--format json|csv|html] [--out <file>] |
Export predictions + metrics (JSON/CSV) or a standalone sanitized HTML report |
forgebench safety-scan [--split <s>] [--provider <p>] [--model <id>] [--limit N] |
Run the safety red-team task (injection, jailbreak, PII, bias, toxicity) |
forgebench dashboard |
Launch the local web UI |
FORGEBENCH_TOKEN=... forgebench dashboard |
Same, but every /api/* request requires the x-forgebench-token header (the UI prompts for it) |
FORGEBENCH_WEBHOOK_URL=... forgebench run --config ... |
POST a compact run summary to the URL after every completed run |
All providers implement the same predict(case) -> ProviderResponse contract:
| Provider | Adapter | Models | API key |
|---|---|---|---|
| Mock | providers/mock.py |
rule-mock-v1 |
None β offline deterministic rule engine, used by unit tests only. A benchmark run that names it is upgraded to the selected provider, with a CLI warning and a provider_substitution entry in the manifest. task-run --provider mock stays offline, for validating the pipeline only. |
| Mistral | providers/mistral.py |
ministral-8b, ministral-3b, ministral-14b, mistral-small-4, mistral-large, mistral-medium |
MISTRAL_API_KEY |
| OpenAI | providers/openai.py |
GPT-5, GPT-4.1, GPT-4o, o1, o3-mini families | OPENAI_API_KEY |
| Anthropic | providers/anthropic.py |
Claude 4 Opus, Claude 4 Sonnet, Claude 3.5 Sonnet, Claude 3 Opus, Claude 3 Haiku | ANTHROPIC_API_KEY |
providers/google.py |
Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 2.0 Flash, Gemini 1.5 Pro, Gemini 1.5 Flash | GOOGLE_API_KEY |
|
| OpenRouter | providers/openrouter.py |
Every model OpenRouter hosts (hundreds), addressed as vendor/model β e.g. anthropic/claude-3.5-sonnet, openai/gpt-4o, meta-llama/llama-3.1-70b-instruct, google/gemini-2.5-pro |
OPENROUTER_API_KEY |
| Ollama | providers/ollama.py (OllamaProvider) β local daemon (also cloud-served :cloud models) |
Whatever you have pulled. Default: OLLAMA_MODEL env, else beam:cloud |
OLLAMA_API_KEY (optional for local Ollama) / OLLAMA_HOST |
The OpenRouter adapter speaks the REST API directly with the standard library, so it works on a bare install (no extra SDK). Its catalog is available from forgebench models --provider openrouter (the pinned snapshot) or --live (straight from openrouter.ai/api/v1/models).
OpenRouter is the default provider. The dashboard preselects it, and forgebench set-key / forgebench task-run use it unless --provider says otherwise. Config-driven forgebench run takes the provider from the config file (the bundled configs use OpenRouter), and an empty selection falls back to the first provider with a saved key β a run never silently degrades to the offline rule engine.
Model lists are synchronized from models.dev and include pricing (input/output/cache read $/M tokens), context windows, and capabilities.
# Same config, different provider
forgebench run --config configs/baseline.yaml --provider ollama --model beam:cloud --run-id ollama-run
forgebench run --config configs/baseline.yaml --provider mistral --run-id mistral-run
forgebench run --config configs/baseline.yaml --provider openai --model gpt-4o --run-id openai-run
forgebench run --config configs/baseline.yaml --provider anthropic --model claude-3-5-sonnet --run-id anthropic-run
forgebench run --config configs/baseline.yaml --provider google --model gemini-2.5-pro --run-id google-run
forgebench run --config configs/baseline.yaml --provider openrouter --model anthropic/claude-3.5-sonnet --run-id openrouter-run| Dataset | Task | Cases | Splits | Version |
|---|---|---|---|---|
finance_v1 |
Bank/ledger exception classification | 200 | 100 train / 40 dev / 60 test | 1.0.0 |
healthcare_v1 |
Medical claims reconciliation | 125 | 80 train / 30 dev / 15 test | 1.0.0 |
ecommerce_v1 |
Marketplace order vs ERP reconciliation | 125 | 80 train / 30 dev / 15 test | 1.0.0 |
legal_v1 |
Contract clause reconciliation and exception triage | 105 | 42 train / 21 dev / 42 test | 1.0.0 |
insurance_v1 |
Insurance claims adjudication and exception triage | 105 | 42 train / 21 dev / 42 test | 1.0.0 |
support_v1 |
Customer support ticket triage and exception reconciliation | 105 | 42 train / 21 dev / 42 test | 1.0.0 |
payments_v1 |
Payment settlement reconciliation and fraud/exception triage | 105 | 42 train / 21 dev / 42 test | 1.0.0 |
procurement_v1 |
Purchase-order vs. supplier-invoice reconciliation | 105 | 42 train / 21 dev / 42 test | 1.0.0 |
subscriptions_v1 |
SaaS subscription billing reconciliation | 105 | 42 train / 21 dev / 42 test | 1.0.0 |
All nine datasets reuse the same input field names as finance_v1, so they run through the existing pipeline with no code changes. Each has a matching config under configs/ (for example configs/legal_baseline.yaml, configs/payments_baseline.yaml).
All datasets share six scenario families:
| Family | Meaning |
|---|---|
matched |
Amounts, dates, and references agree |
amount_mismatch |
Amounts differ (takes precedence when other fields also differ) |
date_mismatch |
Dates differ, amounts agree |
reference_mismatch |
References differ, amounts agree |
duplicate |
One side shows more occurrences than the other |
ambiguous |
Conflicting weak signals; human review required |
Dataset files live under datasets/<name>/:
dataset.jsonβ manifest (name, version, task, license, families, splits, SHA-256 hashes)dataset_card.mdβ human-readable documentationtrain.jsonl,dev.jsonl,test.jsonlβ JSONL split files with ground-truth labels
View any dataset in the dashboard's Datasets view, or via the API:
curl http://localhost:8502/api/datasets
curl http://localhost:8502/api/datasets/healthcare_v1
curl http://localhost:8502/api/datasets/finance_v1/splits/testRegenerate datasets deterministically:
# finance_v1 (and its variants)
python scripts/generate_dataset.py --seed 42 --train 100 --dev 40 --test 60
# legal_v1, insurance_v1, support_v1, payments_v1, procurement_v1, subscriptions_v1 (fixed seed 2026)
python scripts/generate_multi_dataset.py
# saas_billing_v1, telecom_v1, travel_v1
python scripts/generate_more_datasets.py
# Regenerate everything with a generator, and fail if the committed data differs
make check-datahealthcare_v1 and ecommerce_v1 have no generator in this repository, so a test pins their split hashes instead.
| Metric | Definition | Why it matters |
|---|---|---|
| Accuracy | Correct / schema-valid classifications | Overall performance |
| Macro-F1 | Mean F1 across classes with ground truth | Common classes can't dominate |
| Critical false-match rate | Truly non-matching cases wrongly called matched |
The financially dangerous error β a false "matched" means money is assumed reconciled when it isn't |
| Escalation recall | Cases requiring review that were flagged for review | Missed exceptions that need human attention |
| Schema validity | Structured outputs passing the Prediction schema |
Integration reliability β models that can't emit valid JSON are useless in pipelines |
| Latency (p50/p95) | Median and tail response time | Operational performance |
| Estimated cost per 1,000 cases | Provider pricing Γ tokens used | Budget comparison across providers/models |
Denominator conventions and evaluation rules are specified in docs/benchmark_protocol.md.
A comparison answers "did candidate B beat baseline A". History answers the question you ask a month later: is this task or dataset getting better as I keep working on it?
forgebench history # every run, grouped by task/dataset, with deltas
curl localhost:8501/api/history # the same data as JSON (the dashboard's History view renders it)Each row carries the headline score (accuracy for benchmarks, valid rate for
task runs) and Ξ vs previous β the delta against the previous run of the
same group. Click a row in the dashboard to open that run's report.
forgebench report runs/<id> --out report.md # Markdown (per-case rows include the test data)
forgebench export runs/<id> --format html --out report.html # standalone sanitized HTML page
forgebench export runs/<id> --format csv --out results.csv
forgebench export runs/<id> --format json > results.jsonSet FORGEBENCH_WEBHOOK_URL (a Slack incoming-webhook URL, CI listener, or
any collector) and every completed run β CLI or dashboard β POSTs one compact
JSON summary (run_id, model, dataset/task, cases, accuracy). Delivery
failures warn on stderr and never fail the run.
Pin one run per dataset as the baseline, then open any later run and choose Vs baseline in the run modal:
# via the API (the dashboard's pin button posts exactly this)
curl -X POST localhost:8501/api/baselines \
-H 'Content-Type: application/json' \
-d '{"dataset": "finance_v1", "run_id": "runs/baseline-run-id"}'
curl localhost:8501/api/runs/new-run-id/regressionThe comparison answers both questions a metric delta can't: which scalar
metrics moved, and which cases flipped (improved / regressed / still
correct / still wrong, plus cases added or dropped). Pins live in
runs/baselines.json β local state, never committed. The CLI equivalent is
forgebench compare runs/baseline runs/new for a Markdown report.
Experiments are YAML files under configs/. Example β configs/baseline.yaml:
experiment:
name: baseline
dataset: finance_v1
dataset_version: "1.0.0"
split: test
provider:
name: openrouter
model: openai/gpt-4o-mini
options:
temperature: 0.0
inference:
temperature: 0
max_retries: 2
timeout_seconds: 30
max_tokens: 1024
prompt:
name: reconciler # optional: pull the system prompt from prompts/reconciler/
version: null # null/omitted = latest version
system: null # inline text; ignored when prompt.name is set
few_shot_count: 0
evaluation:
metrics:
- accuracy
- macro_f1
- critical_false_match_rate
- escalation_recall
- schema_validity
- latency
- estimated_costConfig values support ${ENV_VAR} interpolation (e.g. "${MISTRAL_MODEL}"). Missing variables fail loudly β never silently.
The provider is optional in practice: an empty/missing selection falls back to
the first provider with a saved key, and the model falls back to the *_MODEL
value saved with forgebench set-key.
Prompts live as immutable files under prompts/<name>/vN.md with named labels:
forgebench prompt-list
forgebench prompt-save reconciler --file my_system_prompt.md --note "tighter rules" --label production
forgebench prompt-show reconciler --version 2
forgebench prompt-label reconciler production 3 # 0 removes the labelA config that sets prompt: {name: reconciler} runs whatever the registry
resolves (highest version unless version: pins one), and the run's manifest
records the prompt's SHA-256 plus the digest of the text the provider actually
sent β so two runs can be proven to have used the same instructions.
Configs can be edited in the dashboard's Config view, validated with Pydantic before saving.
forgebench/
βββ README.md # This file
βββ LICENSE # Proprietary β ForgeBench Commercial License
βββ CODE_OF_CONDUCT.md # Community covenant
βββ CONTRIBUTING.md # How to contribute
βββ SECURITY.md # Security policy
βββ Makefile # install, test, lint, data, demo, dashboard
βββ pyproject.toml # Project metadata, dependencies, scripts
βββ .env.example # Environment variable template (git-ignored .env)
βββ .gitignore # Python, secrets, run artifacts
βββ configs/ # Experiment YAML configs
β βββ baseline.yaml # OpenRouter reference baseline
β βββ ecommerce_baseline.yaml # OpenRouter baselines for the ecommerce, healthcare,
β βββ healthcare_baseline.yaml # insurance, legal, payments, procurement,
β βββ insurance_baseline.yaml # saas_billing, subscriptions, support, telecom
β βββ legal_baseline.yaml # and travel datasets
β βββ payments_baseline.yaml
β βββ procurement_baseline.yaml
β βββ saas_billing_baseline.yaml
β βββ subscriptions_baseline.yaml
β βββ support_baseline.yaml
β βββ telecom_baseline.yaml
β βββ travel_baseline.yaml
β βββ experiments/
β βββ prompt_v1.yaml # Prompt variant
β βββ fewshot_v1.yaml # Few-shot variant
βββ prompts/ # Versioned prompt registry (immutable vN.md + labels.json)
β βββ reconciler/ # Provider default system prompt, pinned as v1
βββ datasets/ # Versioned synthetic datasets
β βββ finance_v1/ # Bank/ledger exception classification
β βββ healthcare_v1/ # Medical claims reconciliation
β βββ ecommerce_v1/ # Marketplace order vs ERP
β βββ legal_v1/ # Contract clause reconciliation
β βββ insurance_v1/ # Insurance claims adjudication
β βββ support_v1/ # Support ticket triage
β βββ payments_v1/ # Payment settlement / fraud triage
β βββ procurement_v1/ # Purchase order vs supplier invoice
β βββ subscriptions_v1/ # SaaS subscription billing
βββ tasks/ # User-defined evaluation tasks (task.yaml + JSONL)
β βββ sentiment_analysis/ # Classification with confidence and key phrases
β βββ summarization_qa/ # Summary accuracy and hallucination checks
β βββ code_review/ # Bug/security/style review with severity
β βββ intent_classification/ # Support intent, confidence and entities
β βββ named_entity_extraction/ # Typed entity extraction from free text
β βββ contract_clause_detection/ # Clause type, risk level, counsel flag
βββ docs/ # Architecture, protocol, integration notes, dashboard screenshots
βββ services/ # Business services (not part of the harness)
β βββ stripe_webhook/ # Stripe checkout β voucher β credit fulfillment (devctl-managed)
βββ scripts/ # Deterministic dataset generators
β βββ generate_dataset.py # Regenerate finance_v1
β βββ generate_multi_dataset.py # Generate the six 105-case datasets
βββ src/forgebench/ # Source code
β βββ __init__.py # Package, version
β βββ cli.py # Typer CLI
β βββ config.py # Pydantic experiment config
β βββ runner.py # Load β validate β infer β score β record
β βββ task_runner.py # Custom task evaluation loop
β βββ tasks.py # Task registry (locate, load, hash, validate)
β βββ registry.py # Dataset registry (locate, load, hash, validate)
β βββ schemas.py # BenchmarkCase, Prediction, Category, TaskDefinition enums
β βββ credentials.py # BYOK storage (.env, chmod 600)
β βββ judge.py # G-Eval style LLM-as-judge scoring
β βββ evidence.py # Evidence field validation
β βββ parsing.py # Structured output parsing
β βββ providers/ # Provider adapters
β β βββ __init__.py
β β βββ base.py # ModelProvider protocol
β β βββ mock.py # Deterministic rule engine
β β βββ mistral.py # Mistral API adapter
β β βββ openai.py # OpenAI API adapter
β β βββ anthropic.py # Anthropic API adapter
β β βββ google.py # Google API adapter
β β βββ openrouter.py # OpenRouter adapter (OpenAI-compatible, stdlib HTTP)
β β βββ verify.py # Shared provider API-key verification
β βββ metrics/ # Metric computation (classification, performance, pricing, reliability, rag)
β βββ reporting/ # Markdown + JSON report renderers
β βββ api.py # FastAPI backend for the dashboard
β βββ dashboard.py # Legacy Streamlit entry point (kept for compat)
β βββ static/ # Dashboard frontend (index.html, style.css, app.js, shell.js, models.json)
βββ tests/ # Unit + end-to-end tests (no API key needed)
βββ runs/ # Local run artifacts (demo/evidence runs may be committed)
pip install -e ".[dev]"
pytestThe suite runs offline: tests that exercise the run pipeline inject a deterministic fake at the provider seam, so nothing spends API credits or flakes on the network. One smoke test hits a real provider and skips unless OPENROUTER_API_KEY is set.
See CONTRIBUTING.md. Good first contributions:
- Additional provider adapters (Cohere, Cohere Command, xAI Grok, etc.)
- Synthetic datasets for new domains (legal, insurance, supply chain)
- New metrics
- Independently reproduced runs with published
report.mdoutputs - Dashboard improvements (light/dark theme refinements, export formats)
All contributions follow the Contributor Covenant.
API keys are never stored in the repository. See SECURITY.md for the full policy, including how keys are stored (git-ignored .env, chmod 600), verified against provider APIs, and excluded from run artifacts.
ForgeBench implements LLM evaluation capabilities inspired by common open-source evaluation frameworks:
Subjective scoring criteria evaluated by any configured provider (Mistral, OpenAI, Anthropic, Google, OpenRouter, Ollama). Define criteria and chain-of-thought evaluation steps per scoring rule:
scoring_rules:
- field: summary_quality
method: llm_judge
weight: 2.0
threshold: 0.7
criteria: "Determine whether the summary captures all key facts without hallucination."
evaluation_steps:
- "Compare the summary against the source document."
- "Penalize any claim not supported by the source."
- "Score highly when meaning and key details match."
judge: # which provider/model judges the llm_judge rules
provider: openai
model: gpt-4o-miniBenchmark runs can also add a judge on top of the rule-based metrics via
the evaluation.judge section β every schema-valid prediction's reason
gets scored against your criterion, and metrics["judge"] (avg score, pass
rate) appears in the report and comparisons:
evaluation:
metrics: [accuracy, macro_f1, escalation_recall, schema_validity]
judge: # optional LLM-as-judge (G-Eval style)
provider: openrouter
model: openai/gpt-4o-mini
# options: {} # provider options (e.g. the mock judge takes none)
# criteria: >- # override the default reason-quality criterionThe judge is advisory: a broken or unparseable judge never fails a run β
failures are counted in metrics["judge"]["n_failures"] and the run
continues on the rule-based metrics alone.
Deterministic lexical metrics for retrieval-augmented generation, no LLM calls needed:
faithfulnessβ share of the answer grounded in retrieved contextanswer_relevancyβ vocab overlap between answer and questioncontextual_relevancyβ share of retrieved chunks relevant to the question
scoring_rules:
- field: answer
method: faithfulness
threshold: 0.8
- field: answer
method: answer_relevancy
threshold: 0.5Set retrieval_context on each test case (in the JSONL expected block or case input) to enable context-aware scoring.
Define arbitrary evaluation tasks in YAML without code β input fields, output schema (string/number/boolean/enum/list[string]), scoring rules with weights and thresholds, plus G-Eval criteria:
forgebench task-list
forgebench task-validate sentiment_analysis
forgebench task-run sentiment_analysis --split test --provider openrouter --judge-provider openai --judge-model gpt-4o-miniFourteen tasks ship with the repo under tasks/:
| Task | What it evaluates | Scoring methods |
|---|---|---|
sentiment_analysis |
Feedback sentiment with confidence and key phrases | exact_match, contains, f1 |
summarization_qa |
Summary accuracy, completeness, hallucination list | boolean, exact_match, f1 |
code_review |
Bug/security/style review with severity and fix flag | exact_match, boolean, contains |
intent_classification |
Support intent with confidence and extracted entities | exact_match, contains, f1 |
named_entity_extraction |
Typed entities from free text with count and primary type | f1, exact_match |
contract_clause_detection |
Clause type, risk level and counsel flag from contract text | exact_match, boolean |
email_classification |
Email category (spam/ham/phishing) with confidence and reason | exact_match, contains, contains |
topic_classification |
Article topic with confidence | exact_match, contains |
question_type |
Question type and whether external context is needed | exact_match, boolean |
safety_redteam |
Red-team probes: prompt injection, jailbreak, PII, bias, toxicity | boolean, exact_match |
expense_reimbursement |
Expense claim review: approve/reject/flag, reimbursable amount | exact_match, contains |
invoice_matching |
Invoice vs. purchase order: match status and review flag | exact_match, boolean, contains |
purchase_order_approval |
Purchase order decision and approver routing | exact_match, contains |
ticket_triage |
Support ticket priority, owning team and response SLA | exact_match, contains |
A red-team workflow is available via forgebench safety-scan (prompt injection, jailbreaking, PII leakage, bias, toxicity); --provider mock runs it fully offline for pipeline validation.
templates/regression.yml is a copy-in GitHub Actions template: an offline
gate (lint, tests, dataset reproducibility) plus a task regression gate that
compares a pinned baseline run against a fresh candidate run and fails the PR
beyond a tolerance.
See CHANGELOG.md for what shipped in each version.
All linked markdown files in the docs/ folder:
| Document | Description |
|---|---|
architecture.md |
System architecture, data flow, and component design |
benchmark_protocol.md |
Evaluation methodology, metric definitions, and denominator conventions |
forge_integration.md |
Integration guide for Forge platform |
OPPORTUNITY-FRAMEWORK.md |
Strategic opportunity analysis and framework |
PITCH.md |
Business case, value proposition, and commercial model |
results.md |
Benchmark results and analysis |
WORKFLOW.md |
Development workflow, CI/CD, and release process |
CHANGELOG.md |
Version history and release notes |
CODE_OF_CONDUCT.md |
Community code of conduct (Contributor Covenant v2.1) |
CONTRIBUTING.md |
Contribution guidelines and development setup |
SECURITY.md |
Security policy and vulnerability reporting |
05_execute_rag.md |
RAG execution guide and examples |
ForgeBench is proprietary software at MVP stage β see LICENSE. A final commercial license has not been published: today there are no licenses on sale and no commercial use terms; the software is free for internal evaluation, while redistribution, a competing managed service and team-scale deployment need an agreement with the copyright holder. The pricing section above describes planned tiers, not an active offer. Model weights and third-party datasets/APIs carry their own terms β check each model card and license independently. Synthetic data in datasets/ was generated by this repository and follows the same terms.
ForgeBench β measure domain adaptation, not just model intelligence.












