Forge

Synthetic Data Engine for Real-World AI

Forge generates training data for autonomous agents and reasoning models. It builds mock application environments from descriptions, writes multi-turn tasks with verified solutions, simulates human colleagues, runs agents in sandboxes, and exports datasets in SFT, DPO, and RL/GRPO formats.

Tip

Looking to train and benchmark models on your data? Use Autotune to fine-tune (LoRA, QLoRA, GRPO) and benchmark models on your data.

Example RL Tasks: Example RL tasks for Forge live in OpenGym.

What Forge Does

Stage	What you get	Where
Environment generation	Docker sandboxes for apps, shells, and browsers, with state snapshots and a Gymnasium API	Environment Types, Custom Generation Pipeline
Synthetic task synthesis	Task batches with solutions tested over fresh runs, reviewed by a separate model; exported as RL tasks, SFT data, or preference pairs	Task Factory, Synthetic Data Engine
Dynamic personas	Simulated colleagues and users with distinct traits, communication habits, and response times	Personas
Agent execution	Agent rollouts through application APIs, logging steps, state hashes, and tool calls	Agent Runs & Data Collection
Grading & Verifiers	Verifier scores with checks for reward hacking and grader independence	Reward Engine, Verifiers
Dataset export	SFT chat pairs, preference pairs, GRPO rollouts, failure datasets, and trajectories with `dataset_manifest.json`	Dataset Export

Environment Types

Type	What runs	Good for
Premade	Pre-built Gmail or Slack replica	Ready-to-use evaluation; seeded with realistic emails, threads, and DMs
Custom	LLM-generated FastAPI app	Simulate any business app from a plain-English description or by dropping a product URL (where the user research agent inspects the real product)
CLI	Ubuntu 22.04 shell, no network, frozen clock	Shell scripting, sysadmin, and file tasks
Browser	Chromium + KasmVNC, no internet	Automating pages the environment serves, form filling, navigation
Repo	A permissively licensed GitHub repo at a pinned commit, dependencies installed, no network, frozen clock	Code repair and implementation tasks graded by the repo's own tests

Environment Contracts

Every Forge environment implements a shared set of interfaces in forge/contracts/, one per concern:

Contract	Answers
`TaskSource`	What problems should the model solve?
`InitialStateProvider`	How is per-episode state set up?
`PromptTemplate`	How is the task presented to the model?
`ToolProvider`	What can the model do?
`ObservationEncoder`	What does the model see back?
`ExecutionBackend`	Where do actions actually run?
`StateManager`	How is state tracked across turns?
`Rubric` / `Verifier`	How is the model's behavior scored?
`TerminationPolicy`	How does the episode end?
`EpisodeController`	Who drives the multi-turn loop?
`Transport`	How does the model talk to the environment?
`PersonaDriver` / `PersonaScheduler`	Who else is in the world, and when do they act?

Environment composes the concerns that describe the world. EpisodeController stays outside because a rollout worker or agent run drives an environment; Verifier remains separate from Rubric because deciding whether an objective was met is different from assigning its reward.

Both ForgeEnv and container-backed environments implement this facade. Their hot paths call the injected collaborators: reset uses InitialStateProvider, actions run through ExecutionBackend, state is owned by StateManager, observations pass through ObservationEncoder, final rewards use Rubric, and every completed step consults TerminationPolicy. The reserved submit control action ends an episode without being sent to a domain backend. Container episode controllers use the same facade rather than duplicating reset, state, action, or termination plumbing with direct HTTP calls. At reset, TaskSource selects an explicit task id or deterministically distributes seeded episodes across its task set. make_agent(..., environment=env) binds that selected task, the environment prompt, and full tool schemas to OpenAI, Anthropic, or vLLM adapters, so the model-facing context cannot drift from the environment.

Premade Environments

Premade environments ship with realistic seed data that resembles real products. They're ready to evaluate agents immediately with no configuration needed.

Gmail

42 emails across Inbox, Sent, Drafts, Archive, and Trash, tagged with custom labels (Work, Urgent, Personal, Newsletter, Receipts)
19 contacts with names and addresses
5 labels with color coding
Send, receive, reply, archive, label, star, and delete actions
Auto-reply simulation: sending or replying triggers a response from the recipient
POST /receive endpoint lets evaluators inject new emails mid-episode
Baseline snapshot saved on boot for drift detection

Slack

7 channels: #general, #engineering, #product, #random, #design, #ops-infra, #announcements
38 messages across channels
88 thread replies stored with parent references
43 reactions across messages
12 direct message threads
Auto-responders simulate team activity when an agent posts
Post, reply, react, DM, and pin actions

Both apps serve the same Forge protocol (/forge/health, /forge/state, /forge/dump, /forge/reset, /forge/snapshot, /forge/restore/{slot}, /forge/restore-state) from one shared module, docker/premade/_shared/forge_protocol.py, along with the virtual clock and deterministic ids. Each build copies the app's folder and that module into a temporary build context, so building never modifies the tracked files.

Core Features

Environment Creation

4-option creation flow: CLI, Browser, Custom, and Premade on the new-environment page
Headless or UI environments: custom environments are API-only by default; opt into With UI on the creation form to also generate a browsable single-page app. A headless build skips the UI specialist entirely: no ui.html, no /ui route, and no UI review gate, so it builds faster and exposes only the RL surface. The toggle applies to custom environments; CLI is always headless, and Browser and Premade always ship a UI.
Custom environment generator: describe any app in plain English or drop a real product URL (where the user research agent inspects the product to extract workflows and specs); choose headless or with-UI
User research agent: given an original product name and URL, fetches documentation to extract workflows, data requirements, UI states, and verified citations, then publishes role-specific briefs for downstream specialists
Prompt planner agent: creates a dependency-aware task graph for dedicated backend, UI, telemetry, state-bridge, policy, reward, correctness, and review agents
Agent-to-Agent context protocol: specialists exchange typed task and artifact messages while scoped channels expose only each task's declared inputs
Reviewer quality gate: static checks and semantic review verify syntax, required APIs, UI action coverage, RL artifacts, code quality, and the original user requirements before files are written
Determinism correctness specialist: a dedicated gate audits generated code for wall-clock access, unseeded randomness, and nondeterministic identifiers, requiring a counter-based virtual clock (forge_now()) and sequential IDs (_next_id()) before artifacts are written; after the container boots, a runtime validator proves /forge/reset restores the exact initial universe (rows, IDs, counters, database included) and that snapshot/restore round-trips, hard-failing the build on any drift
Real-time build progress: WebSocket stream shows agent completion, Docker build phase, and live worker logs
Compiler review: inspect and edit LLM-generated compiler input before the build starts
Self-healing /start: detects stale image tags, missing port bindings, and crash-looped containers; clears bad state and auto-recovers
10-environment cap: enforced at UI and API level; expired environments cleaned up automatically

MicroVM & Container Sandboxing

Sandboxed rollouts in Docker: agent rollouts run in hardened Docker containers.
Experimental Firecracker backend: FORGE_SANDBOX_RUNTIME=firecracker boots KVM microVMs under the Jailer. It has no guest networking or image store yet, so it cannot run worker episodes, and auto-detection never picks it. A VM that fails to boot raises instead of being reported as running.
Ephemeral rootfs disks: each episode runs on a fresh copy-on-write sparse disk image that is discarded on teardown to prevent state contamination.
LLM drift guardrails: every generated file is post-processed before build: base image normalised, port forced to 8000, install step reduced to the hashed runtime lock
Registry fallback: four-tier fallback when Docker Hub flakes: canonical pull → AWS ECR → GCR → direct HTTPS via httpx. Digest-pinned images skip the HTTPS tier, since it rebuilds the manifest and cannot keep the pinned digest
Worker pre-warm: Celery pulls base images and builds the CLI image on boot so user builds and CLI runs always hit cache
Crash-loop detection: restart_policy=on-failure with 3-attempt cap; status flips to error automatically

Custom Generation Pipeline

Custom environment generation separates planning, implementation, assembly, and review:

User prompt + compiler input + optional original product research
          │
          ▼
   UserResearchAgent? ─→ backend / UI / RL / review briefs
          │                 (role-pruned + size-bounded)
          ▼
   PromptPlannerAgent ──→ typed todo DAG + acceptance criteria
          │
          ├── BackendBuilderAgent ─┐
          ├── UIBuilderAgent ──────┴─→ AppAssemblyAgent → TelemetryAgent → StateBridgeAgent
          ├── ScenarioBuilderAgent    (seeded scenarios + source-bound verifier milestones)
          ├── PolicyAgent
          └── RewardAgent
                    │
                    ├─→ EnvironmentCorrectnessAgent   determinism audit gate
                    └─→ ReviewerAgent                 static + semantic checks
                    │
                    ▼
              RepairLoop  ──→ routes each finding back to the owning
                    │         specialist as a correction task, up to
                    │         FORGE_ENVGEN_MAX_REPAIR_ROUNDS rounds
                    ▼
             approved artifacts → docker build/run
                    │
                    ├─→ CorrectnessValidator (post-boot)
                    │     reset fidelity + snapshot/restore round-trip
                    └─→ PostGenerationValidator (post-boot)
                          exercises declared actions against the live
                          container and scores state-manifest coverage

When enabled for a custom environment, UserResearchAgent reads the extracted application spec, the required original product name and URL, optional reference URLs, and a small web search when references are not provided. It synthesizes the target product's workflows, functionality, UI states, data, rules, RL observations, and edge cases. Full raw pages are discarded inside the research task, but each retained source carries a bounded verbatim passage checked against the fetched document. Backend, UI, RL, and review specialists receive only their relevant sections and these verified evidence excerpts under a hard character budget. When disabled, the planner omits the research task and downstream specialists run with the application spec alone.

ScenarioBuilderAgent uses that evidence to bind every verifier-facing required action, forbidden action, and expected answer to a source when documentation justifies the constraint. A milestone source records the fetched document title, URL, and exact passage; a citation counts only when all three match the pruned research context. If no supplied passage supports the constraint, the generator must leave source null instead of inventing a citation. Legacy scenario files containing bare strings remain readable and are classified as unattributed.

Every generated custom/scenarios.json includes an attribution_report. It reports total milestones, source-attributed and model-invented counts and fractions, and analyzes the two groups separately. Source-attributed milestones are documented ground truth. Unattributed or unverifiable milestones are retained as generator findings rather than silently promoted to ground truth.

{
  "scenarios": [
    {
      "scenario_id": "archive_inbox_message",
      "required_actions": [
        {
          "value": "archive_message",
          "source": {
            "title": "Acme Mail guide",
            "url": "https://docs.example.test/mail",
            "passage": "Archived messages leave the inbox."
          }
        }
      ]
    }
  ],
  "attribution_report": {
    "total_milestones": 3,
    "source_attributed_count": 2,
    "model_invented_count": 1,
    "source_attributed_fraction": 0.6666666667,
    "model_invented_fraction": 0.3333333333
  }
}

TaskExecutor runs independent tasks concurrently and waits on declared dependencies. Each task receives a scoped artifact channel. The A2A protocol records assignment, completion, failure, review, and artifact-availability messages with correlation IDs, without copying large generated files into message payloads. The reviewer blocks artifact writes when generated code or requirement coverage fails. A dedicated correctness specialist runs alongside it and blocks writes when generated code is nondeterministic, wall-clock reads, unseeded randomness, nondeterministic IDs, or a /forge/reset that fails to re-initialize the virtual clock and ID counters, while exempting telemetry event-envelope timestamps that never reach /forge/state. When either gate rejects the artifacts, RepairLoop does not fail the build immediately: FindingRouter attributes each review issue to the specialist that owns the offending artifact, RepairPlanner turns the attributed issues into typed correction tasks, and the executor re-runs only those specialists before re-review. It runs at most FORGE_ENVGEN_MAX_REPAIR_ROUNDS (default 2) rounds, de-duplicates findings by fingerprint so an unfixable issue is not retried forever, and raises UnrepairableFinding when a finding cannot be attributed to any agent.

Once the container is built and running, CorrectnessValidator exercises the live endpoints to prove reset fidelity (two resets and a mutate→reset both return the byte-identical pristine baseline) and a snapshot→mutate→restore round-trip before the environment is accepted; any drift hard-fails the build. PostGenerationValidator then boots the same container, calls /forge/reset, exercises the declared actions, and checks the observed state against the StateSchemaManifest, reporting missing fields and a coverage score so a generated app that never populates part of its declared state surfaces before agents run against it.

Generation prompts are grouped behind prompt catalog classes, and the shared LLM client appends an explicit Pydantic output contract to every structured call. EnvGenConfig centralizes model token budgets, research limits, context budgets, and reviewer excerpt sizes; each value can be changed through its FORGE_ENVGEN_*, FORGE_RESEARCH_*, FORGE_SPECIALIST_*, or FORGE_REVIEW_* environment variable. GenerationErrorHandler normalizes specialist failures and retains task/agent error records for orchestration and A2A diagnostics.

Agent execution and data collection are separate layers. Runtime agents choose actions, and ForgeEnv emits immutable snapshots through the storage-agnostic TelemetrySink protocol. Backend EpisodeDataCollector owns SQLite and JSONL persistence; runtime code does not import backend models or database libraries.

Agent Runs & Data Collection

Five agent adapters: random, scripted:<path>, anthropic:<model>, openai:<model>, vllm:<model>
AgentContext: per-episode agent memory with a compact deterministic digest for prompt injection, stuck-vs-context-limit diagnosis, and automatic pruning of error spam and revisited-state noise
Trajectory recording: every step's state, action, and reward persisted to JSONL and DB
Post-episode objective scoring: container and browser runners call ObjectiveScorer once on the final state; CLI uses its final tiered grader. Cheap state-hash and loop monitors may stop stuck runs without exposing grader feedback to the agent
Cross-run episode selection: pick episodes from multiple runs, export as a single merged dataset
Parallel rollouts (local and distributed): launch batched episode rollouts across any compiled environment via the global Rollouts page, Python API, or CLI. Both execution modes are supported:
- Local parallel runs: create_rollout_runner(..., mode="local") / ParallelRolloutRunner runs the same task across isolated environment copies in parallel threads on a single host for sub-second iteration and local testing.
- Distributed parallel runs: create_rollout_runner(..., mode="distributed") / DistributedParallelRolloutRunner coordinates rollouts across multi-worker pools with durable task queuing, burst absorption, backpressure protection, 10-minute TTL, 3-retry dead-letter queuing (DLQ), and deterministic round-robin seed assignment with batch aggregation. Both runners share identical taxonomy outcome classification (success, failure, partial_success, edge_case) and interface contracts. A rollout always runs the task it names: if the environment does not declare that task, each episode fails with failure type configuration and lists the declared tasks, instead of running a different one.
Per-environment dashboard: pass rate, average reward, step efficiency, termination-reason breakdown

Observability & Replay

Live event feed: real-time observability panel streamed from the running container
Unified per-run trace: AgentRunLogger records the agent's LLM layer (prompt, chosen tool call, response) alongside every action, result, and state change as one ordered, step-correlated trace; persisted per run even when the run aborts mid-flight
Per-run loss analysis: LossAnalyzer classifies why a run failed into a fixed seven-mode taxonomy (instruction-following, hallucination, tool-sequencing, early-stopping, context-loss, reward-hacking, surface-overfitting) from the run trace and verifier result, emits a per-run report with evidence and confidence, and aggregates modes across runs. A clean, correct run yields no failure modes
Episode replay: re-run any recorded episode step-by-step from stored trajectory
Branch replay: fork from any step index and try alternate action sequences
Failure clustering: groups failed episodes by trajectory diff similarity
Cross-run anomaly detection: POST /api/sandbox/{env}/detect loads recent completed episodes, compares trajectories across runs, and returns severity-tagged findings in five categories (reward_hacking, distribution_drift, policy_gaming, anomalous_pattern, reward_collapse), surfaced on the per-environment Violations page next to the rule-based audit log
Environment graph: visual entity/action relationship map

Reward Engine

Tiered reward with configurable partial credit, completion bonuses, and efficiency scaling. Mix and match scoring methods per environment:

Method	How it works
LLM-as-judge	Claude Haiku evaluates each trajectory against your requirements. Most flexible.
Sentence Embeddings	Cosine similarity via `all-MiniLM-L6-v2`. Fast, no LLM calls.
ROUGE-L	Longest common subsequence overlap. Deterministic.
BLEU	N-gram precision. Best for short, structured outputs.

If any configured method cannot score (the judge errors, a scorer's library is missing), the episode is left ungraded. It fails as an infrastructure crash (provider_error) and is retried, so a grader outage never shows up as a 0.0 reward.

Verifiers

Six built-in verifier types compose into the final EpisodeEvaluation and its auditable RewardBreakdown:

Verifier	Checks
`ExactStateVerifier`	Specific state field values
`EventVerifier`	Required events appeared in the trajectory
`TemporalVerifier`	Event ordering and timing constraints
`NegativeVerifier`	Forbidden events did not occur
`PolicyVerifier`	Python expressions against current state
`SemanticVerifier`	LLM-based semantic correctness (with embedding cache)

Layered verification: LayeredVerifier composes five layers into one verdict: final-state checks, invariant milestone checks (none skipped, correct order), trajectory checks (necessary tool calls made, no unnecessary ones), LLM-as-judge rubrics for creative tasks, and negative checks for unintended side effects.

Per-environment verifier composition: VerifierComposer builds a configured LayeredVerifier for each task from its declared success/failure conditions and scenario ground truth, mapping them onto the five tiers (the LLM judge stays off by default). Provenance metadata does not affect runtime matching: the composer unwraps each milestone's value before checking action presence, order, and forbidden calls, while also accepting legacy strings and serialized milestone dictionaries. It then scores an episode's result into a RewardBreakdown under either mode: binary (full credit only when every tier passes) or partial (weighted per-tier mean, so a partially-correct trajectory earns graded credit). A right answer reached by an unauthorized side effect or the wrong tool order still fails.

Verifier Trust Boundary & Group Adapters: Instead of writing one-off verification code for every task, VerifierTrustBoundary isolates evaluation inside a protected trust boundary. It checks trajectory integrity (monotonic step ordering, state hash continuity, and policy compliance) and delegates checks to domain group adapters:

CUA (Computer-Use Agents): verifies DOM presence, URL navigation routes, visual milestones, and forbidden UI interactions.
TU (Tool-Use Agents): verifies tool call sets, parameter constraints, execution sequences, and forbidden calls.
Code (Coding & Terminal Agents): verifies exit codes, test suite pass rates, git diff boundaries, and syntax validity.
Data (Database & SQL Agents): verifies table row counts, SQL query assertions, schema invariants, and mutation safety.
Artifact versioning & test isolation: prompts, verifiers, data, and initial test suites are cryptographically versioned via SHA-256 digests (ArtifactVersionRegistry) so rollouts can be differentiated and debugged across runs. Grading evaluates against the authoritative initial test suite, while TestIsolationBoundary completely isolates all tests from the agent to prevent test leakage and cheating.

Reward-hacking audit: RewardHackingAuditor is a separate audit agent that asks whether a passing verdict was earned: it flags passes with skipped milestones, suspiciously short episodes, redundant call patterns, and supports a pluggable LLM audit client. RewardHackingAuditor.for_verifier(...) inherits the milestone list straight from a LayeredVerifier.

Reward ablation presets: set reward_preset in an experiment YAML; both VerifierComposer and TieredRewardEngine resolve the same named contract:

Preset	Verifier and scoring behavior
`full_layered_partial`	All five verifier layers, weighted partial credit, and reward-hacking audit penalties
`binary_final_state`	Final-state checks only; reward is exactly 0 or 1, with no efficiency or partial-credit adjustment
`judge_only`	LLM judge score only; structured state, trajectory, and negative checks are excluded
`full_no_auditor`	Same layered partial reward as the full preset, while the auditor remains post-hoc and does not alter pass/reward

Verification Independence

Forge's environments are authored by an LLM. If the same model family then graded the agent, the grade would not be independent evidence, the generator would be marking its own homework. Forge's answer is structural: the verdict is computed, not asked for.

Grading path	Who issues the verdict	Contaminable
Final-state checks	Assertions over recorded state	No
Invariant / milestone checks	Order and presence over the trajectory	No
Trajectory checks	Necessary vs. unnecessary tool calls	No
Negative checks	Forbidden side effects over the trajectory	No
Reward-hacking audit	Milestone, length, and call-pattern rules	No: the optional LLM audit client is off by default
LLM judge (`judge` layer, semantic checks)	A model	Yes
Objective progress scoring (reward shaping only)	A model	Yes

Five of the seven paths never consult a model, and a sixth only does so when explicitly given a client. VerifierComposer leaves the judge layer off unless a task explicitly declares a semantic check, and the binary_final_state preset removes it entirely, so the default reward path is fully structural and needs no independence guarantee at all. This is the reason the layered verifier is built the way it is, not an incidental design choice.

For the paths that do consult a model, independence is configurable:

Separate judge client: FORGE_JUDGE_MODEL / FORGE_JUDGE_PROVIDER configure grading independently of generation. Every grading call site (TieredRewardEngine, ObjectiveScorer, trajectory anomaly detection) resolves through get_judge_client(); generation keeps using get_client(). Unset, grading falls back to the generation model, a workable local default, and an explicitly non-independent one.
Family-level comparison: model_family() collapses tiers, so grading Sonnet-authored environments with Haiku does not count as separation. Swapping tiers changes cost, not provenance.

Verdict Quorum

A Jury can vote on top of the structural verdict. Members implement one small interface and may be LLM-backed or deterministic, so a statistical test and a Gemini judge are the same kind of thing to the jury.

Setting	Meaning
`FORGE_QUORUM_MODELS`	Comma-separated `provider:model` list. Members must come from different families, and none may share a family with the generator. Empty means a single judge.
`agreement_threshold`	Unanimity among voting members by default; `0.67` accepts 2-1 on a three-member jury
`max_abstention_rate`	Ceiling on undecided episodes (default `0.2`)

Majority decides, but a split below the threshold is indeterminate: the episode leaves the pass-rate denominator instead of being counted as a failure. An abstaining member, a provider outage is missing evidence, not evidence of failure, is excluded from the agreement denominator rather than counted as a vote against.

Interaction Contracts

Every environment declares which capabilities the agent has access to, meaning the actions an agent can take, across every interaction modality, via env.capabilities(). Each capability validates actions against its schema before anything touches the environment, so a hallucinated tool, unknown endpoint, or out-of-bounds click never executes. Not every environment needs every modality; each advertises only the ones it attaches:

Capability	Interacts with	Action shape	Schema enforces
ToolUse	API endpoints / functions of the environment	`{"type", …}`	tool exists, required params present, param types match
MCPUse	MCP server tools	`{"tool", "arguments"}`	tool exists, required arguments present and typed
RESTUse	HTTP endpoints	`{"method", "path", "input"}`	method+path is a declared endpoint, required params present
ORPCUse	Typed RPC procedures	`{"procedure", "input"}`	procedure exists, required input present and typed
ComputerUse	The VM / OS (Linux, macOS, Windows)	`{"action_type", …}`	allowed primitives (`exec`, `screenshot`), non-empty commands
BrowserUse	The browser	`{"action_type", …}`	allowed primitives (`click`, `type`, `press`, `navigate`, `scroll`), viewport bounds

Every ForgeEnv exposes ToolUse (env.tool_use.execute(...) is a schema-validated step()); attach the others with EnvBuilder.with_mcp_use(...) / .with_rest_use(...) / .with_orpc_use(...) / .with_computer_use(...) / .with_browser_use(...)
env.capability_surface() returns {modality: [ToolSpec]}: the full set of actions the agent can take, grouped by modality, so every attached interface is discoverable through one tool surface
CLI environments grant ComputerUse (os="linux"); browser environments route every agent action through BrowserUse

Determinism

Environments are deterministic by default: the same seed and same trajectory produce the same observations and the same score:

Launch-time determinism check (default): two identically seeded rollouts are hashed (observations + rewards + termination flags); a mismatch raises DeterminismError and aborts the launch. It runs in the backend env loader, forge run / forge export, and EnvBuilder.build()
EnvBuilder + DeterminismConfig: virtual clock, seeded RNG and UUIDs, canonical sorted-key JSON, float rejection (integers only), serialized transitions, network and filesystem guards inside the env, and fresh-universe startup (factory caches dropped every reset)
Seed control: the seed threads end to end (reset(seed) → POST /forge/reset {"seed": …} → STATE.seed_state(seed)), so the same seed reproduces the same starting universe and a different seed produces a different-but-reproducible one; an unseeded reset restores the fixed baseline
Generated-app determinism contract: custom LLM-generated apps must use a counter-based virtual clock (forge_now()) and sequential IDs (_next_id()) in place of wall-clock timestamps and random UUIDs, and build the universe from a random.Random(seed); a static correctness specialist audits this before artifacts are written, and a post-boot CorrectnessValidator proves /forge/reset restores a byte-identical initial universe (rows, IDs, counters, DB included), that snapshot/restore round-trips, and that the same seed reproduces identical state while distinct seeds diverge, hard-failing the build on any violation
Replayable episodes: every step records the tool call, emitted events, state diff, hashes, and reward; replay_episode(env, seed, steps) re-executes any recording and verifies every state hash and reward against it. Container/CLI/browser trajectories are written incrementally (each step flushed as it happens), so a run that crashes mid-episode still leaves a durable, replayable partial trace
Pinned builds: every base image is pinned by digest, and every container installs one hashed lock (forge/envgen/runtime_requirements.lock) with pip --require-hashes. The LLM's requirements list never reaches pip, and the correctness gate rejects imports outside the lock
Fixed clocks, zones, and hash order: premade Gmail and Slack stamp time from a SQLite-backed virtual clock and mint sequential IDs, so reset, snapshot, and restart all carry them. Containers run with TZ=UTC and PYTHONHASHSEED=0, browser pages see a fixed Date.now(), and the CLI image preloads libfaketime so every command starts at the same fixed epoch (file mtimes still come from the kernel clock, and statically linked binaries ignore the preload)
One episode per container at a time: a run's episodes execute in order, and a per-environment Redis lock keeps separate runs from driving the same container at once
Fast, clean starts: app containers stay warm and reset in place. A CLI run snapshots its environment once (terminal setup included), each episode forks a fresh container from that snapshot, and a pre-started spare means the next episode starts in about 10 ms. Browser episodes each get a fresh context in the already-running Chromium, so no cookies, storage, or tabs carry over
Grader separation: agent and persona actions can never reach /forge/* control endpoints, so an agent cannot write the state its grader reads. CLI assertions never run in the agent's shell: the finished container is committed, a fresh copy starts with no network and none of the agent's processes, and each assertion runs through a static BusyBox toolbox mounted read-only from a pinned image
Generous fixed timeouts: 60 s per environment request, 120 s per shell command or assertion, and 120 s for a container to come up. The browser waits for the page to settle instead of sleeping a fixed second
Flake-free UI: premade UIs ship a CSS no-motion override and browser sessions force prefers-reduced-motion + injected no-animation styles
SQLite as source of truth: premade and generated apps persist state in SQLite; verification reads /forge/state (DB-backed), never the UI
Enforced separation of concerns: architecture tests keep environment, agents, verifiers, and training code from importing across boundaries
UI-determinism gate: tests/architecture/test_ui_determinism.py asserts every premade UI ships the forge-no-motion kill switch, that the browser runner disables motion on every page it opens, and that verification reads the DB rather than the rendered UI
Test-scenario-diversity gate: a static analyzer (tests/architecture/diversity_audit.py) parses every test module and fails the suite if one asserts only the happy path; each behavior must pair its happy case with a negative case (invalid input / error path) and a false-positive guard (a look-valid input that must be rejected, detected via pytest.raises, a differential/exclusion assertion, or a rejection-named test)

For the determinism ablation only, set FORGE_DETERMINISM=off before building, starting, training, and evaluating environments. This skips the correctness and launch gates, swaps the virtual clock for wall time, stops applying experiment seeds to runtime RNGs, UUIDs, policies, and training libraries, and omits seeds from container resets. Existing containers must be recreated so they receive the flag. Omitting the variable (or setting it to on) preserves the default behavior.

Security & Policy

PolicyEngine DSL: policy and verifier expressions use a restricted AST evaluator instead of eval; violations block transitions and return 0.0 reward
Network and process isolation: AST-based scanning blocks network modules, subprocess access, shell execution, and dynamic imports in generated envs (bypass with FORGE_DEV_NETWORK=true). CLI containers run with no network. App and browser containers each sit on their own internal Docker network with no route out, and a pinned socat gateway per environment publishes the loopback ports and forwards only to that environment. A relay inside the browser's network namespace re-serves DevTools, which Chromium binds to its own loopback. Compiled in-process packages are scanned for network imports before the worker loads them. An agent's action target must be a path on its environment, so it cannot point a request at another host
Generated-code validation: compiler checks run in an isolated subprocess with time and output limits; generated paths are confined to the configured environment root
Credential-safe logging: bearer values and URL query strings are redacted before HTTP, Docker pull, or worker errors reach logs; signed CDN URLs are never emitted intact
Local-only default: run.sh binds the backend to 127.0.0.1 unless FORGE_HOST is explicitly changed
PII redaction: strips emails, phone numbers, and SSNs from LLM input before code generation
Dataset safety gate: quarantines adversarial, adult, and non-permissively licensed rows, redacts secrets, and scrubs PII from internet and human data (see Safety Gate)
RBAC observation filtering: removes or restricts state fields per role, applied in reset() and step()
Policy Violation Viewer: global filterable table of violations by environment, episode, and severity

Environment Customization

A generated environment package can override compiled behaviour without editing generated code. Any custom/*.py file in the package is imported at load time and its decorated functions are registered by CustomizationLoader:

Hook	Overrides
`@override_transition(action_name)`	The transition function for one action
`@verifier(task_name)`	The verifier for one task
`@reward(task_name)`	The reward function for one task
`@observation_transform(name)`	The observation shown to the agent
`@policy_rule(name)`	An additional policy rule evaluated on every step

EnvConfig carries the per-environment knobs the UI's Config page edits, RewardConfig (base success, step penalty, policy-violation penalty, invalid-action penalty, reward clamps, semantic weight), ObservationConfig (mode, actor role, visible/hidden entities for RBAC filtering), and PersonaPopulation (see Simulated People). Overrides live in the package's custom/ directory, which is included in the runnable source export.

Simulated People

Most workflows worth learning are not performed alone. An environment that models only records teaches an agent to edit records; one where a supervisor has to approve the discharge, and will not approve anything undocumented, teaches the job. Environments can be populated with simulated humans who act alongside the agent.

The split between what is reproducible and what is lifelike is the whole design:

Concern	Deterministic?	Owned by
Who is in the cast, and what they are like	Yes, resolved from the episode seed	`PersonaPopulation`
When each person acts, and how often	Yes, from an engine-owned RNG	`PersonaScheduler`
What each person actually does	No, a model decides in character	`PersonaDriver`
What they are permitted to do	Yes, declared up front	`PersonaBehavior.allowed_actions`

A driver returns a proposal. ActionGuard checks it against that persona's declared action space, the environment's real action surface, and the action's required parameters before anything touches state; a proposal that fails is recorded as a blocked turn and never applied. So a model can choose freely without being able to do anything its author did not grant it.

Personas execute through the same ExecutionBackend the agent does, so their writes hash, diff, and replay identically, acting within the agent's step, before it observes, so a colleague's reply lands in the same step's diff rather than appearing out of nowhere in the next one.

Configure a cast in custom/config.yaml, or visually at /environments/<name>/personas:

personas:
  enabled: true
  count: 4                  # above the roster, archetypes are cloned to fill it
  driver: anthropic:claude-sonnet-5   # or `scripted`, free, offline and replayable
  max_actions_per_step: 1
  roster:
    - id: supervisor
      name: Alan Whitmore
      role: shift supervisor
      backstory: Answerable for the whole shift, so approves nothing undocumented.
      traits: { responsiveness: 45, diligence: 90, formality: 85 }
      behavior:
        allowed_actions: [send_message, approve_discharge]
        wake_on: [send_message]
        latency_steps: 2          # shortened by their responsiveness trait
        cooldown_steps: 3
        max_actions_per_episode: 5
  archetypes:                     # cloned to fill `count`
    - id: caller
      name: Caller
      role: patient
      behavior: { allowed_actions: [send_message] }

Six integer traits (responsiveness, initiative, verbosity, diligence, formality, patience) render into the driver's prompt as instructions rather than numbers, so a low-diligence colleague actually answers half the question.

People are written, not chosen from a list. The builder and the personas page both start from an empty roster: an author describes who each person is, what they want, how they write, and where their six dials sit. No premade cast is offered, because whatever shipped in one would quietly become the people every environment has.

For environments defined in Python, forge/personas/archetypes.py provides archetype(), eight reusable dispositions (a gatekeeper, a busy expert, an unreliable third party, someone who cuts corners and does not say so) that supply traits, cadence, and voice while leaving identity to the caller. A test keeps them domain-neutral: none may name an industry, so the library cannot quietly push environments toward one. It is a code convenience and a YAML archetype: shorthand, not a product surface. Every entry ships inert, so an unbound persona can never act.

EnvBuilder.with_personas(...) wires a cast into an in-process environment and ContainerEnvBase(personas=...) into a containerized one; both reject at build time a persona bound to an action the environment does not implement. CLI and browser environments take no cast because there is no notion of a colleague inside a shell session.

In the product, the cast is written before the environment exists and bound after it does. Step 05 of the custom builder (/environments/new/custom) is where people are described; it deliberately offers no action picker, because at that moment the environment has no actions; granting one is refused at the request with that as the reason. The build writes the roster into custom/config.yaml, and because a cast that arrives inert is only a safe default if it is impossible to miss, the binding step is surfaced three times: in the build log, as a Needs actions badge on the environment hub, and as a callout at the top of the personas page. From then on the switch that turns the cast on and off sits in the New Agent Run dialog, where runs are actually launched; it edits the environment rather than the run, since personas are resolved when the environment is built, and says so.

One subtlety worth stating because it is invisible: ContainerEpisodeRunner does not call ContainerEnvBase.step; it drives the environment's collaborators directly, so the persona tick lives in its _execute_action as well. Wiring step alone would have produced a cast that existed on the environment, passed every unit test, and never acted in a real run.

Because a model-backed persona cannot be replayed, build() runs its determinism probe with the scripted driver swapped in: the cast is still present and still acts on the same steps, so the probe proves what it exists to prove, that the environment's own machinery is reproducible.

Synthetic Data Engine

Generate training data without running live agents:

Goal suggestion: LLM proposes research goals tailored to the env's policy and reward; de-duplicates against existing goals
Difficulty scaling: five tiers (Trivial → Expert) with concrete step-count ranges
Edge case injection: inject boundary_conditions, permission_errors, missing_deps, conflicting_state, or recovery scenarios into generated trajectories
Replay manifest: saved to generated_envs/<env>/synthetic_replay.json; active epochs replace live LLM inference in agent runs

Task Factory

Create versioned batches of realistic, executable tasks for any environment, apart from environment building. Open Generator (/generator, the synthetic data generator), pick an environment, choose how many tasks you want (1 to 20,000) and how many times each golden solution must pass (k, 1 to 10), and start a batch. Each batch runs four steps in a Celery job and streams progress to the page.

Taxonomy. The writer model maps 4 to 12 categories of real work in this environment (up to 40 for large batches), across at least 3 difficulty levels. Code checks the spread (level coverage, duplicate names, overlapping descriptions, tools the environment lacks) and asks for one retry before failing the batch. Every batch builds a fresh taxonomy.
Writer. Code turns the taxonomy into an exact slot plan, so 30 requested tasks means 30 slots spread evenly across levels and categories. Each round runs 10 slots at a time: write, check, run, review. Results land steadily, and the environment lock is held for one chunk's golden runs at a time. The writer fills five slots per call and sees the last 50 titles from the same categories, and a draft repeating an accepted title is rejected. Each task carries an objective, seed data, a golden solution, machine-checkable checks, and for difficulty 4 and 5, at least 3 reflection points where the agent has to stop and rethink.
Validation, in three layers:
- A static check in code: the task fits its schema, names only real tools and state, and its golden solution fits the difficulty (1: 1 to 5 steps, 2: 5 to 12, 3: 12 to 30, 4 and 5: 30 or more).
- Golden pass^k: the golden solution runs k times, each from a fresh seeded start. Before it runs, at least one outcome check must fail, so no task starts solved. After it runs, every check must pass, and every run must end in the same final state.
- LLM review by a model from a different family than the writer, judging each task realistic, fair, and sensible. The job refuses to start, before any LLM call, if the two models share a family or a provider's Python package is missing (openai and google-genai are not installed by default).
A draft whose only problem is its golden solution's length goes back to the writer in the same round, with the draft and a target length, to widen or narrow its scope without padding. Other rejected slots go back to the writer with the reason, for up to 3 rounds. A batch that still falls short is saved as short with every rejection reason, never padded.
Registry. One transaction saves the batch as the next version for its environment, with its creation date, the models that ran it, the accepted tasks, every rejection, and its own copy of the taxonomy. A saved batch never changes. Deleting one keeps its version number retired.

What a task's seed, golden solution, and checks look like depends on the environment:

Environment	Seed	Golden steps	Checks
In-process, custom apps, premade apps	Rows merged by id into the reset state	Actions with arguments	Row exists, row absent, row count, value at a path, actions called (optionally in order), actions never called
CLI	Setup shell commands	Shell commands that exit 0	Shell assertions, run in the networkless grading sandbox
Browser	1 to 3 static HTML pages and a start page	goto, click, fill, select, press, check	DOM text, value, or attribute, element exists or absent, page path
Repo	Setup commands that plant a bug or stub a function in `/repo`	Shell commands that repair the source	The repo's own stable tests, run in the networkless grading sandbox

Golden runs reuse the episode machinery: a fresh in-process environment, an app reset in place, a CLI fork from a snapshot taken for the round, or a fresh browser context. Browser tasks need no server. Playwright answers requests to http://task.local from the task's stored pages and aborts everything else. Seeds merge into, and checks read, an app's full restorable state. Premade apps expose it at GET /forge/dump, because their /forge/state is a view that hides archived and sent mail. A round holds the environment's lock, so agent runs on that environment wait until it finishes. An app's state is saved before each round and restored after it.

A freshly compiled in-process environment has stub transitions that leave the state unchanged until custom/transitions.py overrides them (see Environment Customization). The factory rejects every task for such an environment, because no golden solution changes anything.

Each version page (/generator/<batch>) shows the taxonomy, every task with its golden solution, checks, and the validator's reasons, and every rejection with its stage. Export JSON downloads the batch as <env>-v<version>-<date>.json. An SFT batch adds each task's objective and golden commands as a prompt and completion. A preference-pair batch pairs each accepted task with the drafts rejected for the same slot, with the rejection reason attached. A task accepted on its first try has nothing to compare against, so it yields no pair rather than an invented one. Agent runs do not draw from batches yet.

Settings (/settings) edits the platform settings in four groups: models (generator, judge, validation quorum, task validator, Ollama server), runtime, containers, and generation budgets. Any model field whose provider is Ollama lists every model the local server has pulled, with its family and size, disables the generator's family for the judge and validator, and labels Ollama cloud models. Saving checks every value and the model families together, refuses a judge, quorum member or validator from the generator's family, and rewrites only the changed lines in backend/.env through a temporary file swapped into place. The task validator applies to the next batch. Every other setting is read when the API and workers start, so the page marks it "restart pending" until you restart them. API keys, the database URL, file locations and container-internal values are not on the page. Keys stay in backend/.env.

GitHub Repo Environments

Turn permissively licensed GitHub repos into coding environments. You describe what you want, approve a shortlist, and each approved repo becomes a repo environment the task factory can target.

Discover (POST /api/repos/discover). The LLM turns your brief, such as "Python HTTP clients with real test suites", into a few search phrases. Searches run through GitHub's MCP server. Code adds and re-checks the hard filters on every result: a permissive license (the same allowlist as the Safety Gate), not archived, not a fork, enough stars, a recent push, and Python, TypeScript, or JavaScript. READMEs pass the safety gate before the LLM ranks the candidates for fit and test coverage. Repos without a test suite are dropped, and every rejection keeps its reason.
Approve (POST /api/repos/shortlists/{id}/approve). Each approved repo is pinned to its branch head SHA and queued as a sandbox. Approvals count against FORGE_SANDBOX_LIMIT like every other environment.
Snapshot. A Celery job downloads the tarball for that SHA (size-capped) and extracts it through tarfile's data filter, which refuses path escapes, links outside the folder, and device files. Every source file passes the safety gate as internet data. Code skips the person-name model, because it reads identifiers like a package name as people. Regex PII is still pseudonymized, consistently, so tests comparing equal strings still pass. One flagged file, such as a vendored GPL file, rejects the whole repo.
Build. Forge writes its own Dockerfile beside the repo, so the repo's Dockerfile and .dockerignore are never used. It installs dependencies with network (pip for Python, npm, pnpm, or yarn by lockfile for Node), then runs the tests twice in fresh containers with no network and a frozen clock. Tests that pass both runs become the stable set. A repo is dropped, with its build log, when the install fails or fewer than FORGE_REPO_MIN_STABLE_TESTS tests are stable.
Use. A ready repo runs like a CLI environment. The task factory's writer gets the repo, its test command, and its stable tests, and writes tasks that plant a bug or stub a function. Checks run the affected stable tests in the grading sandbox.

GET /api/repos lists every repo environment with its status (queued, snapshotting, building, ready, rejected, dropped, or failed) and reason.

Dataset Export

Seven export formats from the per-environment Export Dataset page:

Format	File	Use with
SFT Pairs	`sft_pairs.jsonl`	TRL SFTTrainer, OpenAI fine-tuning, Axolotl
Preference Pairs	`preference_pairs.jsonl`	TRL DPOTrainer, LlamaFactory
RL Trajectories	`grpo_rollouts.parquet`	Forge batch-GRPO, custom RL pipelines
Failure Dataset	`failure_dataset.jsonl`	Contrastive training, red-teaming
Raw Trajectories	`trajectories.jsonl`	Custom pipelines
Rewards	`rewards.jsonl`	Analysis, custom reward models
Verifier Results	`verifier_results.jsonl`	Debugging, custom reward models

Safety Gate

Every row passes one gate before it is written, in every format above. Generator batch downloads and agent-run trajectory downloads pass it too, row by row, with flagged rows quarantined under the environment's exports/downloads/ directory. Raw rows never touch disk.

Each pipeline declares where its data came from. Model-generated data (Forge rollouts, task-factory batches) is screened for harm but keeps its text. Internet and human data is also PII-scrubbed.

Check	Model data	Internet / human data
Threat, adult, and copyright checks	yes	yes
Secret redaction	yes	yes
PII pseudonyms	no	yes

Threat rules run on the raw text: prompt injection, jailbreak phrasing, destructive commands (rm -rf /, fork bombs, mkfs, dd to a disk), curl | sh, and secret exfiltration. rm -rf ./build passes.
Copyright rules keep only permissively licensed code (MIT, Apache-2.0, BSD-2/3-Clause, ISC, Unlicense, CC0, 0BSD). They flag other SPDX-License-Identifier values, GPL-family license terms, and proprietary notices in comments. A BSD header's "All rights reserved" and an email footer pass.
Adult content is scored by a local DistilBERT classifier. A row is scored as one text, since a lone field like the folder name trash reads as explicit, and every 1,500-character window counts.
Secrets (OpenAI, Anthropic, AWS, GitHub, Slack keys, bearer tokens, private keys, URL passwords) become [REDACTED_SECRET].
PII (internet and human data; code skips the name model) becomes a stable keyed pseudonym, so one person stays one person across a trajectory. Regex covers email, phone, SSN, card (Luhn), IBAN (mod-97), public IPs, and US street addresses. Person names come from GLiNER2-PII, run locally. With your key, sarah@acme.com becomes something like user_3f9a2c1d@example.invalid, and Sarah Chen becomes Person_8b04e6aa.
LLM judge (optional, FORGE_SAFETY_LLM=1) reviews the scrubbed row through the FORGE_JUDGE_* client and adds adult and copyrighted-passage categories. It has the final say on rows only the adult classifier flagged, since the classifier misreads sex-education text. It never clears a rule finding, and a judge error blocks the row.

A flagged row goes to safety_quarantine.jsonl with its reasons, so you can audit false positives. The quarantine is left out of the manifest's files list. dataset_manifest.json gains a safety block with the source, rows seen, kept, and quarantined, replacements by kind, and quarantined rows by rule.

Variable	Default	Purpose
`FORGE_SCRUB_KEY`	none	HMAC key for pseudonyms. Required for internet and human data.
`FORGE_NER_MODEL`	`fastino/gliner2-privacy-filter-PII-multi`	GLiNER2 checkpoint for names. Downloaded once from the Hugging Face Hub.
`FORGE_NER_THRESHOLD`	`0.8`	Name confidence cutoff, between 0 and 1. Lower catches more names and more product names.
`FORGE_ADULT_MODEL`	`eliasalbouzidi/distilbert-nsfw-text-classifier`	Text classifier with an `nsfw` label.
`FORGE_ADULT_THRESHOLD`	`0.9`	Adult score cutoff, between 0 and 1.
`FORGE_SAFETY_LLM`	`0`	`1` adds the LLM judge.

All environment-specific episode results convert to the shared forge.contracts.RolloutRecord. Runtime collectors, Parquet export, and training loaders therefore exchange one rollout shape while retaining richer controller results internally.

Runnable source export: separately from the datasets, any generated environment can be downloaded as a self-contained zip (GET /api/envs/{env_name}/download, or the download action on the environment page). build_source_bundle packages the generated app, container_env.py, reward_fn.py, state_schema.json, and any custom/ overrides alongside a README and a docker-compose.yml, while filtering runtime artifacts (episodes/, __pycache__, .pyc). It runs with one command so an environment can be published, reviewed, or archived independently.

Training

Forge does not train models. Training is handled by Autotune. Export your verified rollouts or preference datasets from Forge, then train your policy:

# SFT or GRPO training with automated LoRA/QLoRA recipe selection
uv run autotune train exports/grpo_rollouts.parquet --objective grpo --output-dir ./policy_checkpoint

Autotune also benchmarks the trained model against its base, on held-out exports or on Harbor tasks.

Web UI

The responsive Next.js control surface uses an industrial foundry visual system with active navigation, environment inventory telemetry, live status indicators, and accessible reduced-motion behavior.

Page	What it does
Generator	Synthesize task taxonomies and datasets with automated golden pass^k validation and dated batch versioning
Export	Configure and export SFT, DPO, and GRPO datasets with dataset manifests ready for Autotune

Status & Known Gaps

Everything documented above is implemented and covered by the test suite, including distributed parallel runs across multi-worker pools with deterministic seed assignment, burst queues, retries, dead-letter queues, and run-level aggregation.

Model benchmarking lives in Autotune. See TASKS.md for the ranked work queue.

Architecture

Browser / API Client
        │
        ▼
  Next.js Frontend (:3000)
        │  REST / WebSocket
        ▼
  FastAPI Backend (:8000)
        │
        ├── Centralized Platform Services
        │     ├── Config:        forge.config.yaml (versioned prompts, verifiers, datasets)
        │     ├── Observability: DevLogger (contextual binding, timers, redaction)
        │     └── Resilience:    CentralizedErrorHandler (system-wide error catalog)
        │
        ├── SQLite (forge.db)    environments, runs, episodes, audit log
        │
        ├── Distributed Parallel Runner (Multi-Worker Pool)
        │     ├── DurableTaskQueue: burst queue, 10m TTL, 3-retry Dead-Letter Queue (DLQ)
        │     ├── Seed Partitioner: deterministic round-robin worker assignment
        │     └── Aggregation:      DistributedRolloutBatch & run-level statistics
        │
        └── Celery Worker        async background tasks
              │
              ├── Execution Runtimes (Dual-Engine)
              │     ├── Firecracker MicroVM: Jailer hardware virtualization & sandbox
              │     └── Docker Container:    fallback container isolation
              │
              ├── Workloads:
              │     ├── CLI:       ubuntu:22.04 MicroVM / container
              │     ├── Browser:   chromium MicroVM / container
              │     ├── Premade:   pre-built apps (Gmail, Slack)
              │     └── Custom:    User Researcher → Planner → Specialist DAG → Reviewer
              │
              ▼
  Reverse Proxy → Sandbox Hub (App / Terminal / Observability)
              │
              ▼
  Environment facade (Gymnasium-compatible)
  ┌────────────────────────────────────────────────────────┐
  │  reset()                                               │
  │    InitialStateProvider                                │
  │    StateManager                                        │
  │    ObservationEncoder (RBAC)                           │
  │                                                        │
  │  step(action)                                          │
  │    ActionValidator                                     │
  │    PolicyEngine ───────────────► AuditLog              │
  │    ExecutionBackend (Firecracker / Docker)             │
  │    PersonaEngine ──────────────► ActionGuard           │
  │    Trust Boundary Adapters (CUA, TU, Code, Data)       │
  │    Verifier → Layered Rubric & Independent Grader      │
  │    TerminationPolicy                                   │
  │    TelemetryClient                                     │
  └────────────────────────────────────────────────────────┘
              │
              └──► Export (7 formats: SFT / DPO / RL / Failure / Raw)

Project Structure

forge/
  config.py            # CentralizedConfig: forge.config.yaml registry & versions
  dev_logger.py        # DevLogger: contextual binding, timers, PII/secret redaction
  jsonl.py             # Streaming JSONL object reader with strict and tolerant modes
  contracts/           # The interfaces every environment family implements
    types.py           # Shared shapes: Task, Action, Observation, ToolSpec, results
    environment.py     # Environment facade composing ten of the eleven contracts
    dataset.py         # TaskSource; initial_state.py, prompting.py, tools.py
    observation.py     # ObservationEncoder; backend.py, state.py, transport.py
    reward.py          # Verifier and Rubric; termination.py, episode.py
    rollout.py         # RolloutRecord: the shared collector → exporter → trainer record
  runtime/             # Gymnasium env, state, trajectory, verifiers, agents
    distributed/       # Distributed parallel runs: DurableTaskQueue, DLQ, partitioner, runner
    trust_boundary/    # Domain evaluation adapters (CUA, TU, Code, Data)
    versioning/        # Prompt, verifier, and dataset version registry
    error_handling.py  # CentralizedErrorHandler catalog and tracking
    parallel_rollout.py # ParallelRolloutRunner, execute_single_rollout, classify_rollout_outcome
    prompting.py       # ForgeAgentPromptTemplate: the default PromptTemplate
    tools.py           # Spec / Capability / OpenAPI ToolProviders
    tasks.py           # Seeded, reproducible task selection from a TaskSource
    interaction.py     # Capability contracts: tool / MCP / REST / oRPC / computer / browser use
    verifier_composer.py  # Per-task LayeredVerifier composition + binary/partial scoring
    agent_logger.py    # Unified per-run trace (LLM calls + actions + state changes)
    loss_analysis.py   # Per-run failure-mode taxonomy + cross-run aggregation
    reward_hacking.py  # RewardHackingAuditor
    reliability/       # Retries, snapshots, replay, quarantine, env versioning, failure types
    clustering.py      # FailureClusterer
  extraction/          # LLM pipeline, PII redactor, schemas
  compiler/
    generators/        # Jinja2 compiler, package builder
  envgen/              # LLM orchestration, container runtime, episode/CLI/browser runners
    agents/            # Backend, UI, assembly, telemetry, state, scenario, policy, reward, correctness, reviewer
    research.py        # Dedicated User Research agent: product profiling, personas, user journeys
    planning.py        # Typed task plans and dependency validation
    executor.py        # Dependency-aware specialist task execution
    a2a.py             # Typed Agent-to-Agent messages and scoped context permissions
    artifact_bus.py    # Async artifact publish/await between dependent specialists
    repair.py          # RepairLoop: route review findings back to owning specialists
    correctness_validator.py  # Post-boot reset-fidelity + snapshot/restore validation
    post_generation_validator.py  # Post-boot action exercise + state-manifest coverage
    source_bundle.py   # Package a generated env as a standalone runnable zip
    objective.py       # ObjectiveScorer: LLM 0–1 progress score toward a stated goal
    error_handling.py  # GenerationErrorHandler: normalized specialist failures
    episode_runner.py  # Container episode loop; cli_runner.py / browser_runner.py
    telemetry/         # Telemetry client and collectors
    firecracker/       # Firecracker MicroVM runtime, rootfs, kernel, Jailer sandboxing
    container.py       # ContainerRuntime lifecycle; docker_cli.py, docker_images.py,
    │                  #   dockerfile.py, container_specs.py, sandbox_network.py,
    │                  #   episode_snapshots.py hold the injected collaborators
    tiered_reward.py   # TieredRewardEngine with partial credit and multi-method scoring
    ml_reward.py       # SentenceEmbeddingScorer, NGramScorer (ROUGE-L / BLEU)
  contracts/           # Interface contracts: TaskSource, Rubric, StateManager
  customization/       # Per-env overrides: decorator hooks, EnvConfig, loader
  personas/            # Simulated humans: population, scheduler, guardrails, drivers, engine
  schema/              # StateSchemaManifest and related schemas
  validation/          # Verdict quorum, jury, and statistical trajectory detectors
  taskfactory/         # Versioned synthetic task batches: taxonomy, writer, validation, top-up
    schemas.py         # Task, check, seed, golden step, verdict, and difficulty rules
    taxonomy.py        # TaxonomyBuilder + spread check; slots.py plans exact slots
    writer.py          # TaskWriter; review.py: cross-family TaskReviewer
    static_check.py    # Rejects drafts that cannot run; state_checks.py seeds and checks state
    pass_k.py          # Golden solution must pass k fresh runs with one fingerprint
    pipeline.py        # Rounds, top-up, rejections; registry is the caller's
    runners/           # in_process, container_app, cli, browser task runners
  settings.py          # Process-wide settings: determinism mode, seeds, paths, URLs
  reward_presets.py    # Canonical reward-ablation presets shared by every reward path
  grading_provenance.py  # Model families, for keeping generator and grader models independent
  logging_utils.py     # Credential-safe redaction for logs
  safety/              # Dataset safety gate: source policy, PII/secret scrubbing, threat, adult, and copyright checks, LLM judge
  repogen/             # GitHub repo environments: MCP discovery, safe snapshots, toolchains, builds, task notes
  paths.py             # Confined-path helpers for generated-env file access

  cli/
    main.py            # forge CLI: compile, validate, run, replay, diagnose
    diagnosis.py       # forge diagnose; replay_render.py renders forge replay
backend/
  app/
    api/               # FastAPI routers: sandbox, envs, personas, episodes, agent_runs,
    │                  #   synthetic, evaluate, exports, audit, rollouts, detect, compile,
    │                  #   task_factory, settings
    services/
      task_registry.py # Versioned task batches; task_factory_targets.py opens runners
      env_file.py      # Rewrites named keys in backend/.env for the Settings page
      settings_registry.py # Every setting the Settings page edits, with its checks
      export_writers/  # sft_pairs, preference_pairs, grpo_rollouts, failure_dataset, ...
    worker/            # Celery tasks, one module per job; tasks.py re-exports them
    │                  #   sandbox_build_tasks.py, rollout_tasks.py, agent_run_tasks.py,
    │                  #   task_factory.py
    models.py          # SQLAlchemy models: SandboxEnvironment, Episode, AgentRun,
                       #   AuditLog, ...
frontend/
  app/
    dashboard/         # Cross-environment stats: pass rate, reward, failure clusters
    rollouts/          # Global parallel episode rollout launcher
    violations/        # Global policy audit log (filterable by env / episode / severity)
    compiler-review/
      [job_id]/        # Inspect and edit LLM compiler output before build
    generator/         # Synthetic data generator: new batch, live progress, versions
      [batchId]/       # One version: taxonomy, tasks, rejections, export
    settings/          # Platform settings: models, runtime, containers, budgets
    environments/
      new/             # 4-option landing page (CLI / Browser / Custom / Premade)
        custom/        # Prompt form, user-research toggle, and the simulated-people step
        premade/       # Gmail / Slack picker
      [env_name]/
        sandbox/       # Tabbed hub: App / Terminal / Observability
        progress/      # Real-time planner, specialist, review, and Docker build progress
        agent/         # Agent runs + cross-run episode selection
        dashboard/     # Pass rate, reward distribution, step efficiency
        config/        # Environment config editor
        personas/      # Simulated people: roster, trait dials, action space, seed preview
        policy/        # Policy requirements editor
        reward/        # Reward requirements + scoring method selector
        evaluate/      # Policy and reward evaluation viewer
        synthetic/     # Synthetic data: goal suggestion, difficulty, edge cases
        export/        # Dataset export
        violations/    # Per-environment policy audit log
        replay/        # Episode step-through viewer
        graph/         # Visual entity/action relationship map
examples/
  gmail_env/           # Reference in-process environment built on the contracts
docker/
  premade/
    _shared/           # forge_protocol.py: /forge/* routes, virtual clock, ids, snapshots
    gmail/             # Gmail-like environment (seeded with 42 emails, 19 contacts)
    slack/             # Slack-like environment (seeded with 7 channels, 88 thread replies)
tests/
  runtime/             # Kernel, verifier, policy, RBAC, network isolation, PII tests
  backend/             # API integration tests, E2E sandbox + agent-runs tests
  envgen/              # ContainerRuntime, normalisation, pull/mirror/HTTPS fallback tests
  cli/                 # forge CLI command tests (run determinism, replay)
  customization/       # Hook registry, loader, and config tests
  personas/            # Cast resolution, cadence, guardrails, drivers, both env families,
  │                    #   the container run path, and the build-flow wiring
  gmail_env/           # Premade Gmail determinism, transition, and verifier tests
  premade/             # Premade Gmail and Slack app query-count and state tests
  architecture/        # Separation-of-concerns, UI-determinism, and test-diversity gates

Getting Started

Prerequisites: Python 3.11+, Node.js 18+, Docker, Redis

# 1. Clone and install
uv venv && source .venv/bin/activate
uv pip install -e ".[dev]"
npm --prefix frontend install

# 2. Start everything
./run.sh        # Redis + Celery worker + backend (:8000) + frontend (:3000)

# 3. Stop everything
./kill.sh

The development runner configures Redis with a TCP backlog of 128 to match the default macOS kernel limit and avoid local startup warnings.

Service	URL
Frontend	http://localhost:3000
Backend API	http://localhost:8000
Swagger Docs	http://localhost:8000/docs

Run tests:

uv run pytest                                  # full Python suite
uv run pytest tests/architecture               # boundary, UI-determinism, and diversity gates only
uv run pytest tests/runtime tests/envgen       # kernel + generation pipeline
npm --prefix frontend test                     # polling lifecycle tests
npm --prefix frontend run lint
npm --prefix frontend exec -- tsc --noEmit --project frontend/tsconfig.json
npm --prefix frontend run build

The Python suite mocks LLM calls and avoids container builds. Runtime depends on your machine and installed optional dependencies. tests/architecture doubles as a review gate: it fails the build on cross-layer imports, animated premade UIs, and tests that assert only the happy path.

Environment Variables

Infrastructure

Variable	Default	Description
`FORGE_GENERATED_ENVS_DIR`	`generated_envs`	Where compiled environments are written
`FORGE_DB_URL`	`sqlite:///./forge.db`	Backend database URL
`FORGE_SANDBOX_LIMIT`	`10`	Maximum active sandbox environments
`FORGE_HOST`	`127.0.0.1`	Backend bind host used by `run.sh`
`FORGE_DETERMINISM`	`on`	Set to `off` only for the determinism ablation; disables virtual time, runtime/training seeding, and determinism gates
`FORGE_DEV_NETWORK`	`false`	Set to `true` to bypass network isolation in generated envs
`FORGE_DISABLE_PREWARM`	unset	Set to `1` to skip base-image and CLI-image pre-warm on worker boot
`REDIS_URL`	`redis://localhost:6379/0`	Redis URL for Celery and build progress pub/sub; `run.sh` replaces this with a runtime-generated authenticated URL
`FORGE_REDIS_PASSWORD`	generated at startup	Optional local Redis password override used by `run.sh`; must be at least 32 hexadecimal characters and is never logged
`NEXT_PUBLIC_API_URL`	`http://localhost:8000`	Backend URL used by the frontend
`CORS_ORIGINS`	`http://localhost:3000`	Comma-separated origins the backend accepts browser requests from
`CELERY_BROKER_URL` / `CELERY_RESULT_BACKEND`	`REDIS_URL`	Override the Celery broker or result store separately from progress pub/sub
`FORGE_DEBUG`	unset	Set to any value for debug-level backend logs

GitHub Repo Environments

Variable	Default	Description
`GITHUB_TOKEN`	-	GitHub token with public read access. Required for repo discovery and downloads
`FORGE_GITHUB_MCP_URL`	`https://api.githubcopilot.com/mcp/`	GitHub MCP server endpoint (streamable HTTP)
`FORGE_REPO_MAX_TARBALL_MB`	`200`	Largest repo download accepted. Extraction is capped at 5 times this
`FORGE_REPO_MIN_STABLE_TESTS`	`10`	Fewest tests that must pass both baseline runs for a repo to be kept

Container Images & Resource Limits

Variable	Default	Description
`FORGE_PYTHON_BASE_IMAGE`	`python:3.12-slim`	Base image every generated custom environment is built on
`FORGE_CLI_IMAGE`	`ubuntu:22.04`	Image used for CLI environments
`FORGE_BROWSER_IMAGE`	`lscr.io/linuxserver/chromium:latest`	Image used for browser environments
`FORGE_CONTAINER_MEMORY`	`1g`	Memory cap for generated custom containers
`FORGE_CLI_MEMORY`	`1g`	Memory cap for CLI containers
`FORGE_BROWSER_MEMORY`	`2g`	Memory cap for browser containers
`FORGE_CONTAINER_NANO_CPUS`	`1000000000`	CPU quota in nano-CPUs (1e9 = 1 core)
`FORGE_CONTAINER_PIDS`	`256`	PID limit per container

Generation Budgets

EnvGenConfig centralizes every model token budget, research limit, context budget, and reviewer excerpt size; each field maps to one environment variable. The full list with defaults lives in .env.example, the ones most often tuned:

Variable	Default	Description
`FORGE_ENVGEN_MAX_REPAIR_ROUNDS`	`2`	Reviewer-driven repair rounds before the build fails
`FORGE_ENVGEN_CAPABLE_TOKENS`	`8192`	Output budget for capable-tier generation calls
`FORGE_ENVGEN_TELEMETRY_TOKENS`	`32768`	Output budget for the telemetry specialist
`FORGE_RESEARCH_SEARCH_RESULTS`	`3`	Pages fetched by the user researcher when no reference URLs are given
`FORGE_RESEARCH_DOCUMENT_CHARS`	`20000`	Per-document character cap on fetched research
`FORGE_SPECIALIST_CONTEXT_CHARS`	`12000`	Hard character budget on the context handed to each specialist
`FORGE_REVIEW_FILE_CHARS`	`16000`	Per-file excerpt size the reviewer sees

Credentials

forge/credentials.py declares every credential Forge reads: ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, GITHUB_TOKEN, and FORGE_SCRUB_KEY. No other module reads them. .env.example sets each one to mock_key, which counts as unset. A feature that needs a missing credential refuses to run and names the variable. SDK clients get their key passed in, so none of them picks up a placeholder on its own.

LLM Provider

All LLM calls go through a single get_client() factory; swap providers or models without touching code.

Variable	Default	Description
`FORGE_LLM_PROVIDER`	`anthropic`	LLM backend: `anthropic`, `ollama`, `openai`, or `gemini`. The last two need their SDKs installed (`openai`, `google-genai`)
`FORGE_LLM_MODEL`	`claude-haiku-4-5-20251001`	Standard-tier model (faster, cheaper)
`FORGE_LLM_MODEL_CAPABLE`	`claude-sonnet-4-6`	Capable-tier model (code generation, complex reasoning)
`ANTHROPIC_API_KEY`	-	Required when `FORGE_LLM_PROVIDER=anthropic`
`OPENAI_API_KEY`	-	Required for the `openai` provider and `openai:<model>` agents. The `vllm:<model>` agent never sends it
`GEMINI_API_KEY`	-	Required for the `gemini` provider. `GOOGLE_API_KEY` is accepted too
`OLLAMA_BASE_URL`	`http://localhost:11434`	Ollama server URL
`VLLM_BASE_URL`	`http://localhost:8000/v1`	OpenAI-compatible endpoint for the `vllm:<model>` agent adapter
`FORGE_JUDGE_PROVIDER`	falls back to `FORGE_LLM_PROVIDER`	Provider used for LLM grading only
`FORGE_JUDGE_MODEL`	falls back to `FORGE_LLM_MODEL`	Model used for LLM grading only. Set this to a model outside the generating family to make LLM-graded runs independent, see Verification Independence
`FORGE_TASK_VALIDATOR_PROVIDER`	-	Provider for the task factory's validator (`anthropic`, `openai`, `gemini`, `ollama`). Required to create tasks. Set it here or on the Settings page
`FORGE_TASK_VALIDATOR_MODEL`	-	Task factory validator model. Must come from a different family than `FORGE_LLM_MODEL_CAPABLE`, the task writer

Run fully locally with Ollama:

FORGE_LLM_PROVIDER=ollama FORGE_LLM_MODEL=gemma4:12b ./run.sh

CLI

forge compile --input spec.json --output generated_envs   # Extract + compile an environment
forge validate generated_envs/<env_name>                  # Tests + override validation on a package
forge run --env <env_name> --task <verifier_id> \
  --seed 42 --steps 10               # Run one episode with a seeded random policy
forge export --env <env_name> --seed 42 --out exports     # Run + write the trajectory as JSONL
forge replay <episode_id> [--json]   # Replay a recorded episode (ep_* gym or cep_* container)
forge diagnose <env_name> [--json]   # Analyse episode quality across all runs

forge run and forge export drive the environment with a seeded random policy (seeded_random_policy), not an LLM adapter; LLM/scripted agents run through the backend agent-run API and the Agent page. Both commands run the launch-time determinism check before the first reset().

License

MIT. See LICENSE.

Built by Mostofa Shakib.

Name		Name	Last commit message	Last commit date
backend		backend
docker/premade		docker/premade
examples		examples
forge		forge
frontend		frontend
generated_envs		generated_envs
specs		specs
tests		tests
.env.example		.env.example
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
forge.config.yaml		forge.config.yaml
kill.sh		kill.sh
pyproject.toml		pyproject.toml
run.sh		run.sh
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation

Forge

What Forge Does

Environment Types

Environment Contracts

Premade Environments

Gmail

Slack

Core Features

Environment Creation

MicroVM & Container Sandboxing

Custom Generation Pipeline

Agent Runs & Data Collection

Observability & Replay

Reward Engine

Verifiers

Verification Independence

Verdict Quorum

Interaction Contracts

Determinism

Security & Policy

Environment Customization

Simulated People

Synthetic Data Engine

Task Factory

GitHub Repo Environments

Dataset Export

Safety Gate

Training

Web UI

Status & Known Gaps

Architecture

Project Structure

Getting Started

Environment Variables

Infrastructure

GitHub Repo Environments

Container Images & Resource Limits

Generation Budgets

Credentials

LLM Provider

CLI

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages