fuseraft sandbox

An eval sandbox for fuseraft's multi-agent orchestration pipelines. The test subject is a small "weather CLI" project that different orchestration teams build and extend from a set of task briefs, so pipeline behavior can be exercised and scored end to end.

Prerequisites

• The fuseraft CLI on PATH (build it from fuseraft-cli with ./build.sh).
• jq, for run-evals.sh's summary generation.
• An XAI_API_KEY environment variable — not a fuseraft requirement (it works with any OpenAI-compatible endpoint, e.g. a LiteLLM proxy in front of Bedrock), just what these particular sandbox configs happen to be pinned to (grok-4.3 via the x.ai endpoint in every Model: block under config/*/agents/). Point Endpoint/ApiKeyEnvVar elsewhere if you want to eval a different provider.

Layout

.fuseraft/
├── config/                 # Orchestration + per-agent prompts/settings (fixed)
│   ├── greenfield/         # Greenfield: Preflight -> Planner -> Developer -> Tester -> Reviewer
│   ├── swe/                # SWE: Planner -> PlannerCritic -> Developer -> Tester -> Reviewer (+ Verifier)
│   └── brownfield/         # Brownfield: Archaeologist -> Planner -> Developer -> Reviewer (graph)
├── evals/
│   ├── suite.yaml          # Eval cases: which orchestration + task brief + pass criteria (fixed)
│   ├── tasks/              # Task briefs (markdown) given to the agent team (fixed)
│   └── results/            # Per-case run results (jsonl) — generated, gitignored, kept across runs
├── tests/                  # Tester agent's test scripts + fixtures — generated, cleared between runs
├── knowledge/              # Agent-accumulated decisions/objectives/repository notes — generated
├── docs/                   # Agent-authored reference docs (e.g. worked examples) — generated
└── artifacts/              # Generated reports (e.g. test-report.json) — generated, cleared between runs

config/ and evals/ (suite + task briefs) are the sandbox's fixed design — hand-authored and not touched by runs. Every agent in config/*/agents/*.yaml explicitly declares Isolation: Shared, so it sees the full shared session transcript rather than fuseraft-cli's Isolation: Fresh default (a synthesized handoff directive plus only whatever Context: sources it declares) — these configs were authored and tuned against full-transcript visibility, so the declaration is required to keep results comparable across fuseraft-cli versions.

Everything else is per-run agent output, written under the write scopes granted by FileSystemPermissions in each orchestration.yaml: agents may write to workspace/ (the actual CLI code, created by the run), .fuseraft/tests/ (the Tester agent's test scripts and fixtures), and .fuseraft/artifacts/ (reports like test-report.json). .fuseraft/knowledge/ and .fuseraft/docs/ accumulate agent notes and reference docs across runs.

run-evals.sh (see below) clears .fuseraft/tests/, .fuseraft/artifacts/, and workspace/ automatically after every run, via cleanup.sh. .fuseraft/evals/results/ is left alone on purpose — each run's JSONL + summary accumulate there as history; run ./cleanup.sh --all if you want to discard that history too.

Eval cases

Case	Orchestration	Task	Tags	What it exercises
weather-cli-greenfield	Greenfield	weather-cli.md	greenfield, smoke	Build the CLI from scratch
weather-cli-swe	SWE	weather-cli-json-flag.md	swe, smoke	Extend an existing CLI with full safeguards
weather-cli-brownfield	Brownfield	weather-cli-add-city.md	brownfield, smoke	Targeted one-entry change to existing code
weather-cli-multi-city	SWE	weather-cli-multi-city.md	swe, complex	Multi-value args, per-item loop, partial failure, aggregate exit code
weather-cli-datafile-refactor	Brownfield	weather-cli-datafile-refactor.md	brownfield, complex	Pure refactor (hardcoded dict -> external JSON) with byte-identical behavior
weather-cli-unit-flag	SWE	weather-cli-unit-flag.md	swe, complex	Data-schema extension, argparse choices validation, flag-conditional JSON keys
weather-cli-case-insensitive	Brownfield	weather-cli-case-insensitive.md	brownfield, complex	Case-insensitive normalization + first-occurrence dedup

The smoke cases are the fast baseline (fuseraft eval run .fuseraft/evals/suite.yaml --filter smoke) — run these first when checking a fuseraft-cli change didn't regress anything before spending the time on the complex cases.

Model selection

Every agent pins its own Model: block (see any config/*/agents/*.yaml), so nothing stops you from running a heterogeneous team — a cheap/fast model on mechanical roles and a stronger one on judgment-heavy ones. All 17 agents here are currently pinned directly to grok-4.3; each orchestration.yaml also has a commented-out Models: alias block (fast / reasoning) near the bottom as a starting point — define real model configs there once and reference them as Model: { ModelId: fast } from each agent instead of repeating ModelId/Endpoint/ApiKeyEnvVar in every file.

What to weigh per role:

• Mechanical/leaf roles (Preflight, Archaeologist's one-time recon, Approved) — a fixed, low-ambiguity job (check the environment, list files, emit a one-line confirmation) with light reasoning. Good candidates for your cheapest/fastest model.
• Planning roles (Planner, PlannerCritic) — need to hold the whole task in mind and produce or critique a complete, unambiguous brief. Underpowered here shows up as an incomplete files_to_change/execution_checklist the Developer can't satisfy, or a Critic that rubber-stamps a bad brief. Worth your strongest reasoning model.
• Developer — the highest tool-call volume role (read/patch/write/shell_run in a loop) and the most exposed to instruction drift: in these evals we've directly observed a model skip its own explicit "always re-run verify_command" instruction and hand off on a fabricated "complete" self-report (see Latest results below). Prioritize reliable function-calling and instruction adherence over raw code quality — a merely competent coder that actually runs its own verification beats a strong one that assumes success.
• Gate roles (Reviewer, Verifier, Tester) — these exist to catch the other roles' mistakes, so a model that's too agreeable defeats the point. Look for models that resist premature approval and reliably produce the exact structured output the validators expect (Reviewer here must emit a fenced json review block before APPROVED; a model that drops formatting under pressure burns retries or aborts — see the MaxRevisits comment in brownfield/orchestration.yaml).

Two things to budget for regardless of model:

• Re-tune the model-specific knobs, not just ModelId. MaxRevisits/MaxEscalations, MaxInTurnContextTokens, and validator retry counts were tuned against grok-4.3's specific failure modes (e.g. brownfield/orchestration.yaml's comment on why the back-edge limit absorbs grok-4.3's occasional empty-response turns). Swapping the model without revisiting these compares apples to a differently-tuned harness, not the models.
• Run each case more than once before comparing models. Same model, same config, same task: weather-cli-greenfield passed in 5, 8, and 10 turns across different runs today, and failed outright in a fourth. A single pass/fail per case isn't enough signal to say one model outperforms another — average over a handful of runs per case.

Latest results

Run 20260824-211413, full suite, against fuseraft-cli feature/nextgen @ 778bf4a:

Case	Result	Turns	Duration	Tokens (in/out)
weather-cli-greenfield	PASS	5	44.5s	176,385 / 1,998
weather-cli-swe	PASS	19	244.3s	710,147 / 5,409
weather-cli-brownfield	PASS	12	268.2s	356,545 / 2,056
weather-cli-multi-city	FAIL	11	277.3s	381,222 / 3,628
weather-cli-datafile-refactor	PASS	11	89.4s	261,820 / 2,678
weather-cli-unit-flag	PASS	11	285.0s	607,885 / 4,460
weather-cli-case-insensitive	PASS	29	297.6s	990,077 / 8,431

6/7 passed. Total duration ~1506s; total tokens in 3,484,081 / out 28,660.

The one failure (weather-cli-multi-city) was Developer non-compliance, not a suite defect: it skipped its required verify_command run, wrote a false "complete" summary to session context, and kept re-asserting it on retry. The ImplementationComplete contract correctly rejected the unverified handoff each time until retries were exhausted — the guardrail working as intended, not a bug to fix.

Full run history (JSONL + per-run summary) accumulates in .fuseraft/evals/results/, gitignored.

Running

Run the full suite with:

./run-evals.sh

This runs every case in .fuseraft/evals/suite.yaml, writes a timestamped <run>.jsonl and <run>-summary.md to .fuseraft/evals/results/, prints the summary, then clears .fuseraft/tests/, .fuseraft/artifacts/, and workspace/. It exits non-zero if any case failed. Pass an alternate suite file as the first argument: ./run-evals.sh path/to/other-suite.yaml.

For finer control (filtering to one case/tag, a per-case timeout, etc.), call the fuseraft CLI directly:

fuseraft eval run .fuseraft/evals/suite.yaml --filter smoke

The generated CLI's own test report (from the Tester agent, not the eval harness) lands in .fuseraft/artifacts/test-report.json during the run — run-evals.sh clears it afterward along with the rest of the ephemeral output dirs.

To clear the ephemeral output dirs without running the suite (e.g. to reset the sandbox between manual experiments), run:

./cleanup.sh

Add --all to also wipe .fuseraft/evals/results/ and discard prior run history.