EvalForge

English | 简体中文

Clean-room Evaluation Infrastructure

EvalForge turns offline evaluation specifications into auditable records, metrics, and reports through one verified infrastructure path.

EvalForge is an installable Python evaluation infrastructure for identifiable, repeatable, and auditable offline runs. It separates configuration, component resolution, execution, typed run evidence, evaluation, aggregation, persistence, and reporting. The verified v0.2 path includes a deterministic no-key fixture, canonical config hashing, structured failures, atomic JSON persistence, and optional HTML reporting—without claiming real-model quality or production readiness.

Project boundary: EvalForge is engineering infrastructure. It is not a benchmark, leaderboard, or reliability optimization framework.

Why EvalForge?

Current workflow

Evaluation projects often begin with a practical collection of scripts: one runs a system, another judges outputs, a metrics module calculates scores, and a final step writes results.

Pain point

As runs accumulate, their meaning becomes distributed across filenames, dictionaries, call order, local conventions, and operator memory. Basic questions become difficult:

• Which configuration produced this result?
• Which backend generated this output?
• Did execution fail, or did evaluation fail?
• Does the report represent the same record set?
• Can another engineer reproduce the run without inheriting the original environment?

Engineering challenge

An evaluation system must coordinate configuration, component selection, execution, recording, scoring, aggregation, persistence, and presentation without collapsing those responsibilities into one implicit procedure.

EvalForge

EvalForge treats each run as an auditable engineering object. Explicit contracts make component resolution, execution evidence, configuration identity, result authority, and report provenance visible in one small verified path.

Key Features

Feature	What it provides
Installable CLI	A stable evalforge run entry point for offline experiments
Registry-based resolution	Named dataset, backend, evaluator, and metric selection
Typed contracts	ExperimentSpec, ScenarioSpec, RunRecord, TraceSchema, and EvaluationResult
Deterministic offline backend	A no-key regression fixture for repeatable infrastructure checks
Config Hash	SHA-256 identity for canonical parsed configuration content
Result Store	Atomic local persistence of the canonical result.json payload
Structured errors	Categorized config and backend failures with persisted scenario evidence
JSON / HTML reports	Machine-readable JSON and an optional human-readable HTML summary

Architecture

flowchart TB
    subgraph C["Configuration Layer"]
        CLI["CLI"]
        SPEC["ExperimentSpec / ScenarioSpec"]
        HASH["Config Hash"]
        CLI --> SPEC
        SPEC --> HASH
    end

    subgraph R["Resolution Layer"]
        REG["Registry"]
        BUILTINS["Dataset / Backend / Evaluator / Metric Resolution"]
        REG --> BUILTINS
    end

    subgraph X["Execution Layer"]
        RUNNER["Runner / Orchestration"]
        BACKEND["Backend Adapter"]
        RECORD["RunRecord / TraceSchema"]
        RUNNER --> BACKEND --> RECORD
    end

    subgraph E["Evaluation Layer"]
        EVALUATOR["Evaluator"]
        METRICS["Metrics"]
        AGG["Aggregation"]
        EVALUATOR --> METRICS --> AGG
    end

    subgraph P["Persistence Layer"]
        RESULT["EvaluationResult"]
        STORE["Result Store"]
        RESULT --> STORE
    end

    subgraph V["Presentation Layer"]
        JSON["JSON Report"]
        HTML["HTML Report"]
    end

    SPEC --> REG
    BUILTINS --> RUNNER
    RECORD --> EVALUATOR
    AGG --> RESULT
    STORE --> JSON
    STORE --> HTML

Loading

Configuration declares intent; resolution selects components; execution produces evidence; evaluation interprets it; persistence establishes the authoritative result; and presentation exposes that result without recomputing metrics.

Main Pipeline

CLI
  ↓
ExperimentSpec
  ↓
Registry
  ↓
Runner / Orchestration
  ↓
Backend Adapter
  ↓
RunRecord / TraceSchema
  ↓
Evaluator
  ↓
Metrics
  ↓
Aggregation
  ↓
Result Store
  ↓
JSON / HTML Report

The pipeline separates side-effecting execution from scoring and gives each boundary an explicit input, output, and evidence artifact. In v0.2, run_experiment performs the synchronous Runner/orchestration role; there is no separately verified Runner class or production scheduler.

Quick Start

git clone git@github.com:dimpleoVo/EvalForge.git
cd EvalForge

python3 -m venv .venv
. .venv/bin/activate

python -m pip install -e '.[test]'
evalforge --help
evalforge run --config configs/offline_demo.yaml --html

Run the complete test suite:

pytest

The compatibility entry point remains available:

python run_eval.py run --config configs/offline_demo.yaml

Demo

This is the verified offline command and its CLI artifact map:

$ evalforge run --config configs/offline_demo.yaml --html
{"html": "outputs/offline_demo/report.html", "report": "outputs/offline_demo/report.json", "result": "outputs/offline_demo/result.json"}

The deterministic fixture produces three RunRecord entries. The verified result has status completed, zero errors, and aggregate values of 2/3 for both exact match and edit similarity.

Artifact	Purpose
outputs/offline_demo/result.json	Canonical persisted EvaluationResult
outputs/offline_demo/report.json	Machine-readable report from the same result payload
outputs/offline_demo/report.html	Optional human-readable summary

Exercise the persisted backend-error path without an external API key:

$ evalforge run --config configs/offline_error_demo.yaml
{"html": null, "report": "outputs/offline_error_demo/report.json", "result": "outputs/offline_error_demo/result.json"}

That run records one backend_error, preserves a null failed output, and reports completed_with_errors. The CLI exits 0 because the infrastructure completed the run and persisted the scenario failure.

Repository Layout

EvalForge/
├── evalforge/
│   ├── cli.py
│   ├── engine.py
│   ├── config.py
│   ├── registry.py
│   ├── builtins.py
│   ├── schema.py
│   ├── backends.py
│   ├── evaluators.py
│   ├── store.py
│   └── reporting.py
├── configs/
│   ├── offline_demo.yaml
│   └── offline_error_demo.yaml
├── tests/
├── pyproject.toml
└── run_eval.py

Path	Responsibility
evalforge/cli.py	CLI contract and top-level infrastructure error handling
evalforge/engine.py	Verified synchronous main-chain orchestration
evalforge/config.py	Experiment loading, validation, and canonical config hash
evalforge/registry.py	Named component registration and resolution
evalforge/builtins.py	Offline dataset, backend, evaluator, and metric registration
evalforge/schema.py	Typed experiment, scenario, trace, record, and result contracts
evalforge/backends.py	Backend interface and deterministic offline fixture
evalforge/evaluators.py	Evaluation, per-record metrics, and aggregation
evalforge/store.py	Atomic local JSON result persistence
evalforge/reporting.py	JSON and optional HTML report projection
configs/	Verified offline success and backend-error experiment specs
tests/	Six tests covering config, registry, main chain, errors, and CLI behavior

The repository also retains historical analysis, scheduler, Docker, leaderboard, and OpenAI-compatible modules for auditability. Their presence does not make them verified v0.2 main-chain capabilities.

Verified Capabilities

Status	Capability	Evidence boundary
✅	Installable Python package	Clean-clone installation of evalforge 0.2.0
✅	CLI help and offline run	evalforge --help and both offline configs
✅	Registry resolution	Dataset, backend, evaluator, and metric resolution in the main chain
✅	Typed run evidence	RunRecord, TraceSchema, and EvaluationResult artifacts
✅	Stable Config Hash	Same canonical hash across repeated identical runs
✅	Deterministic backend	Stable normalized records, evaluator rows, metrics, and status
✅	Evaluation and aggregation	Per-record values and aggregate metrics persisted in JSON
✅	Result Store	Atomic local result.json write
✅	JSON / HTML reports	Reports derived from the same EvaluationResult payload
✅	Structured backend error	Persisted backend_error with completed_with_errors status

The frozen evidence commit is a7d150412df947eb070b1f6352c958a0c441d3c1.

Design Philosophy

• Evidence First: a directory, interface, or documented intention is not treated as proof that a capability participates in the verified runtime.
• Typed Contracts: explicit experiment, scenario, trace, record, and result types make lifecycle handoffs visible.
• Small Verified Core: one complete, inspectable path is preferred over a broad surface with uneven evidence.
• Capability Boundary: implemented main-chain behavior is separated from partial, stubbed, historical, and not-established paths.

License

License follows the repository. No License file is currently declared; repository owners should choose one only after confirming ownership and distribution intent.