LocalEval

LocalEval is a benchmark harness and web workbench for evaluating workflow-oriented model behavior in a controlled local environment.

It includes:

• a Go backend that runs code in isolated Docker containers
• a PostgreSQL-backed benchmark runner with scaffold-aware execution, grading, and reporting
• a Next.js web workbench for browsing suites, launching runs, and reviewing results
• a structured catalog for models, tasks, scaffolds, tools, and artifact expectations

What This Repository Does

LocalEval is designed for benchmark workflows where outcome quality and process quality both matter.

The current implementation supports:

• isolated code execution in Docker
• LM Studio and customer-hosted OpenAI-compatible model endpoints
• URL-based model connections with configurable authentication headers
• scaffold-aware benchmark comparisons
• artifact-aware grading and report generation
• a web interface for run operations, authoring, and inspection

Repository Layout

• cmd/evaluator - HTTP server, benchmark CLI entrypoint, and runtime wiring
• internal/ - execution, scoring, PostgreSQL repositories, model adapters, and control-plane primitives
• web/ - Next.js workbench UI and related client components
• db/ - Drizzle schema and checked-in PostgreSQL migrations

Prerequisites

• Go 1.24+
• Docker and Docker Compose
• Node.js 20+ and pnpm if you want to run the web workbench locally
• Python 3 if you want to run the Compose smoke-test script
• A running Docker daemon

Setup

git clone <repository-url>
cd go-exec-sandbox
go mod download
cd web && pnpm install

Configuration

PostgreSQL

PostgreSQL is the only runtime persistence backend. The API and worker require DATABASE_URL; the one-shot migrate service uses the separate MIGRATION_DATABASE_URL bootstrap role. PostgreSQL stores catalog, run, execution, evidence, event, job, and metadata state. Local files are reserved for artifact bytes behind the storage abstraction.

Model Connections

Model connections are external HTTP endpoints. LM Studio is the supported local OpenAI-compatible integration; deployed or self-hosted customer endpoints use the same synchronous protocol. Authentication is configured through secret references and declared request headers. The benchmark control plane stores typed connection revisions and never stores returned secret values.

Running the Backend

Benchmark API

go run ./cmd/evaluator

This starts the API when the required PostgreSQL configuration is available.

Docker Compose

docker compose up -d --build
docker compose ps
docker compose logs -f api worker

Compose starts exactly five services:

1. postgres — pinned PostgreSQL with the named postgres_data volume;
2. migrate — one-shot Drizzle migration service using the bootstrap database role;
3. api — Go API using the application database role;
4. worker — Go execution worker using the application database role and customer-provided model or agent endpoints;
5. frontend — Next.js workbench on http://localhost:3000.

migrate waits for healthy PostgreSQL. The API and worker wait for successful migration completion and verify schema readiness before serving. Neither service applies migrations at startup. The API and worker share only the artifact-byte volume; PostgreSQL is the source of truth for all control-plane and execution state.

Set database credentials through environment variables or secret files: POSTGRES_PASSWORD_FILE, POSTGRES_APP_PASSWORD_FILE, DATABASE_URL, and MIGRATION_DATABASE_URL. EXPECTED_SCHEMA_HASH is required by the API and worker readiness checks.

The browser sends same-origin /api/v1 requests through the Next.js proxy. The public NEXT_PUBLIC_API_BASE_URL value (http://localhost:8080 in Compose) is available to the UI and operator-facing links. The Next.js /api/v1 route handler forwards requests at runtime using BACKEND_ORIGIN=http://api:8080, including live event streams and request bodies. Internal service names are never sent to the browser, and changing the backend origin does not require rebuilding the frontend image.

To stop the stack without deleting named volumes:

docker compose down

Validate the service isolation without starting containers:

bash scripts/test-compose-layout.sh
bash scripts/compose-inspect.sh docker-compose.yml

Run the full Compose smoke test when Docker is available:

bash scripts/test-compose-e2e.sh

The E2E stack uses isolated host ports 28080 (API) and 23000 (frontend) so it can run alongside the default application Compose stack on 8080/3000.

Provision isolated databases for development verification:

bash scripts/setup-test-database.sh   # TEST_DATABASE_URL for live contracts
bash scripts/setup-app-database.sh    # application Compose stack database

Running the Web Workbench

cd web
pnpm dev

The web app runs on the default Next.js port and provides:

• dashboard views
• suite and task authoring
• run launch forms
• live run progress
• comparison pages
• run artifact and event inspection

API Surface

Health

GET /ping

Sandbox Execution

POST /execute

Executes source code inside an isolated Docker container.

Metrics

GET /metrics

Returns request and error counts.

Benchmark Run

POST /benchmark/run

Runs the configured benchmark and returns a JSON report.

Versioned API

The web workbench also uses versioned backend routes under /api/v1/ for run, catalog, artifact, comparison, and import/export operations.

Project Structure

go-exec-sandbox/
├── cmd/
│   └── evaluator/              # HTTP server, benchmark CLI, and wiring
├── db/                          # Drizzle schema and PostgreSQL migrations
├── docs/                        # Specs, plans, and research notes
├── internal/
│   ├── accesspolicy/            # Visibility and access rules
│   ├── attachments/             # Artifact and attachment handling
│   ├── benchmark/               # Benchmark orchestration and reporting
│   ├── controlplane/            # Run records, versions, and event plumbing
│   ├── grading/                 # Outcome and judge evaluation logic
│   ├── httpapi/                 # Versioned backend routes and handlers
│   ├── modeladapter/            # Provider adapters
│   ├── modelruntime/            # External model endpoint resolution and checks
│   ├── prompting/              # Prompt rendering and parsing
│   ├── runner/                 # Execution modes and task runners
│   ├── sandbox/                 # Docker-based code execution
│   ├── scoring/                 # Scoring and judge logic
│   ├── solver/                  # Model execution strategies
│   └── tenancy/                # Scoped storage and credentials handling
├── web/                         # Next.js workbench
├── docker-compose.yml           # Local multi-service setup
├── Dockerfile                   # Backend container image
└── README.md

Development

Verification overview

PostgreSQL contract tests require a separate isolated test database. The application Compose database is provisioned independently and must never be used as TEST_DATABASE_URL.

# Isolated contract-test database (host port defaults to 55433)
bash scripts/setup-test-database.sh
source .test-db.env
export EXPECTED_SCHEMA_HASH="$(awk -F= '$1 == "EXPECTED_SCHEMA_HASH" { print $2; exit }' .env.example)"

# Application Compose database (default project: localeval)
bash scripts/setup-app-database.sh
source .app-db.env

# Contract, schema, and repository verification
bash scripts/test-postgres-contract.sh

# Full acceptance matrix (provisions test DB, static checks, Compose E2E)
bash scripts/test-acceptance.sh

# Cleanup isolated databases when finished
bash scripts/cleanup-postgres-environments.sh all

scripts/test-postgres-contract.sh provisions a fresh test database by default, exports redacted connection metadata, runs live postgres_contract tests with -p 1 -shuffle=off, and fails if TEST_DATABASE_URL is missing. It never silently skips live database coverage.

scripts/setup-test-database.sh and scripts/setup-app-database.sh print redacted URLs and the checked-in EXPECTED_SCHEMA_HASH. Secrets are written only to gitignored .test-db.env / .app-db.env files.

The app-role replay privilege contract is opt-in and requires explicit DATABASE_URL, MIGRATION_DATABASE_URL, and TEST_DATABASE_URL inputs. It canonicalizes all three URLs before any mutation, runs sabotage and replay in a single transaction, and rolls back the transaction before checking the app connections. After sourcing the isolated test environment, run it with:

RUN_APP_ROLE_RECONCILIATION_CONTRACT=1 bash scripts/test-postgres-contract.sh

This contract fails closed unless all three URLs use exactly one loopback host (localhost, 127.0.0.1, or ::1) and each connected database name matches localeval_test_*. It requires exact equality of the connected canonical database identity across app, migration, and test URLs; equal URL strings alone are never considered safe. It deliberately repairs injected stale grants twice with the canonical db/provision/reconcile-app-role.sql, runs every mutating probe and effective privilege check inside rollback-only transactions, and performs only read-only cleanup verification after rollback. The production runtime does not require the test database marker.

Backend Tests

go test ./...
go test -race ./...
go vet ./...

Live PostgreSQL repository contracts:

source .test-db.env
export EXPECTED_SCHEMA_HASH="$(awk -F= '$1 == "EXPECTED_SCHEMA_HASH" { print $2; exit }' .env.example)"
go test -tags=postgres_contract -p 1 -shuffle=off ./internal/postgres/... -count=1

Web Tests

bash scripts/test-web.sh

Or manually:

cd web
pnpm test
pnpm lint
pnpm build

Database schema checks

source .app-db.env
bash scripts/test-database-migrations.sh
bash scripts/test-database-migrations-docker.sh
pnpm --dir db test
pnpm --dir db schema:check

Compose verification

bash scripts/test-compose-layout.sh
bash scripts/compose-inspect.sh docker-compose.yml
bash scripts/test-compose-e2e.sh
bash scripts/scan-active-policy.sh

The Compose E2E path uses the guided import fixture in testdata/compose-e2e/guided-import/ and a deterministic OpenAI-compatible HTTP model fixture. It launches through the frontend /api/v1 proxy and asserts scoped preview, mapping, publication idempotency, typed run creation, and durable run completion.

Build

go build -o evaluator ./cmd/evaluator
cd web && pnpm build

License

MIT License.