diff --git a/docs/benchmarks/paired-agent-setup.md b/docs/benchmarks/paired-agent-setup.md new file mode 100644 index 0000000000..60903834f4 --- /dev/null +++ b/docs/benchmarks/paired-agent-setup.md @@ -0,0 +1,203 @@ +# Paired coding-agent benchmark setup + +A coding-agent comparison needs an isolated repository checkout, a matching code +index and an instrumented agent runner. This guide explains how to qualify that +setup before measuring tool choice, task resolution, tokens, cost and elapsed +time. Use it with the [benchmark methodology](methodology.md) and +[running instructions](running.md). + +## Pair attempts by task and repeat + +Use `(task_id, repeat)` as the comparison key and +`(task_id, repeat, arm)` as the unique attempt key. Every arm in a pair starts +from the same complete source commit, problem statement and public inputs. + +| Arm | Agent tools | Context policy | +| ---------------------- | --------------------------------------------------- | ----------------------------------------------------- | +| Baseline | Source read, search, edit and fixed tests | No Loom backend, schemas, instructions or hooks | +| Optional retrieval | The same source tools, plus the declared Loom tools | The model chooses; zero calls is a valid outcome | +| Orchestrated retrieval | The same source tools | The application retrieves at declared workflow stages | + +Freeze the provider, exact model identifier, API settings, dependency images, +visible tests, hidden graders and resource limits across paired attempts. +Adding tool definitions consumes input tokens; include that overhead in the +retrieval arm's cost. Do not add unused retrieval schemas to the baseline to +make its prompt the same length. + +Persist the complete randomized schedule before generation. Keep every scheduled +failure, nonuse, abstention and budget stop. Do not select only successful +attempts or compare the subset that chose retrieval against the whole baseline. +Use identical baseline A/A copies to measure variation in the runner and model. + +## Keep the checkout, SDK and runner separate + +The runner operates on a fresh source-only checkout for each attempt. Its working +directory can be elsewhere if every source operation is explicitly rooted in +that checkout. The checkout provides the read, search, edit and test capabilities +that both arms share. + +The SDK sends scoped operations to Loom. It does not give a remote model direct +filesystem access. In an API runner, the application advertises tool schemas, +receives structured tool requests, executes them and returns their results to the +model. Installing a coding-agent plugin does not add that loop to a raw model API. +Plugin instructions, hooks and automatic context retrieval are separate treatment +choices and must be declared if included. + +Keep hidden tests, reference patches, provider credentials and controller files +outside the agent's readable paths. Do not expose a shell, Git object database, +parent checkout or arbitrary SDK operation through a source tool. + +Scripted replay checks the runner's control flow. It provides no measurement of +live model quality, tool adoption or savings. A working provider credential also +does not qualify the tool loop, grading path or budget enforcement. + +## Prepare private indexes at the task bases + +Use a private Loom installation and database for the experiment. Follow +[local development](../development/local-development.md) and +[code-index operations](../operations/code-index-operations.md); do not copy a +running deployment's credentials or migrate a database found on an occupied port. +Record the server build and exact resource names so cleanup affects only the +experiment's processes and containers. + +Each historical task needs an index at its starting commit. A current-branch +index may already contain the solution. Keep separate namespaces for different +bases, disable automatic ingestion after qualification and use one viewer token +per task namespace. The runner must bind the namespace itself; the model supplies +only the arguments of the advertised retrieval operations. + +For a tree-sitter-only experiment, disable the compiler-tier workers: + +```dotenv +LOOM_BACKGROUND_WORKERS=off +LOOM_CODE_SCIP_APPLY_ENABLED=false +LOOM_CODE_SCIP_DELTA_ENABLED=false +LOOM_CODE_SCIP_FULL_ENABLED=false +LOOM_CODE_SCIP_PY_DELTA_ENABLED=false +LOOM_CODE_SCIP_TS_DELTA_ENABLED=false +LOOM_CODE_SCIP_GO_DELTA_ENABLED=false +LOOM_CODE_EMBED_RECOVERY_ENABLED=false +``` + +Configure an entry for each repository/base in `LOOM_CODE_REPOS_JSON`, with a +separate namespace, its bare mirror and the implementation path prefixes the +experiment allows. Make the pinned commit available in that mirror first. +The index phase extracts chunks, tree-sitter symbols and edges without embedding +when `--embed` is absent: + +```bash +loom code backfill \ + --org "$BENCH_ORG" --namespace "$BENCH_NAMESPACE" \ + --phase index --commit "$BENCH_COMMIT" --no-fetch --concurrency 1 +``` + +Set the `BENCH_*` shell variables to the configured task's org, namespace and +complete base commit. Do not run the embed phase for an exact-retrieval +experiment. Empty vectors in source chunks do not establish semantic-search +quality. Indexing cost and time belong in a separate setup measurement; +include them in any amortized cost comparison. + +Use Linux for the bounded parser and process supervision. Importing a grammar +package is insufficient: a failed parse can leave searchable chunks with no +symbols. Check `code.index_errors`, then request a known definition through +`code.symbol` and verify that its tier is `treesitter`, its body is available, +and its path and line span match the task checkout. `code.index_status` must +identify the expected repository and full commit. Check the compiler-tier fields +as well; disabling a worker does not erase an existing compiler index. + +Separate read-time cache updates from source or watermark drift. +`code.index_status` can refresh the shared `status_snapshot` row in +`code_index_state`. Record that cache activity separately when comparing index +contents before and after an attempt. + +An elided body is a context-budget outcome. An absent definition is an abstention. +Neither proves that the index is stale. Record these separately from bodies that +differ from the local source, and retain the affected task in the comparison. + +## Qualify the evaluator independently + +Pin dependency images by digest and record their installed package locks. Select +a runtime compatible with each historical base. If the image is not available, +rebuild and requalify it; a copied digest or earlier receipt does not qualify a +different environment. A Dockerfile base needs a repository-qualified manifest +digest, not a bare local image identifier. + +Resolve source and evaluator paths absolutely before applying patches in nested +checkouts. A patch command can exit successfully without applying an out-of-scope +patch. Verify that the required test definitions exist, collect and execute. + +Run these controls on each exact task image and grader path: + +1. Visible tests pass on the unmodified base. +2. Hidden regression tests fail on the base, while regression-preserving tests pass. +3. The reference patch passes every selected hidden test. + +Reject missing selectors, collection errors, skipped tests and ambiguous or +repeated test identifiers. Fix an incompatible runtime at its source; do not +suppress warnings or weaken assertions to qualify it. + +Containers and read-only source mounts do not, by themselves, protect grading +authority. Candidate code executing inside the test interpreter can affect its +exit status or writable reports. Qualify that boundary before treating reports +as independent outcomes. An edit filter is an additional task constraint: freeze +and disclose it, and do not describe it as a sandbox for arbitrary Python. +Keep evaluator tests and configuration fixed when applying candidate source edits. + +## Record the model loop and physical transports + +Record the native provider response before converting tool blocks. Preserve +opaque or signed blocks when continuing the conversation. Disable automatic +retries and alternate-model fallback unless the protocol explicitly budgets them. + +Exercise application hooks when qualifying the native message format. A +verification hook can add context to the same user message as tool results. If the +provider requires results first, preserve that ordering. A baseline-only transport +probe does not test this path. + +| Evidence | What to retain | +| ------------------ | ---------------------------------------------------------------------------------------------------- | +| Attempt identity | Pairing key, arm, source and patch hashes, provider/model, protocol digest and terminal status | +| Model request | Request identity, input hash, dispatch and response times, stop reason and raw usage fields | +| Tool choice | Provider tool-call ID, parent turn, tool name and arguments | +| Tool execution | Matching call ID, result or error, elapsed time and whether cached evidence was returned | +| Retrieval | Caller (model or application), operation, scope, accepted/rejected spans, abstention and request IDs | +| Physical transport | Every dispatch, response, retry, failure, auth or metadata request | +| Evaluation | Visible and hidden outcomes, grader identity, timeout and cleanup receipts | + +Count every model-selected tool request, including one served from a local cache. +Keep that count separate from logical backend reads and physical HTTP requests. +For example, a read with one preflight scope check and one postflight scope check +uses three physical requests. A cached tool choice can use none. + +Require zero physical Loom requests in baseline attempts, including failed +requests and ambient SDK traffic. A zero logical counter alone is insufficient. +Disable ambient runs and mentor requests when using the SDK in a controlled loop. +Correlate client and server records through request IDs without logging credential +headers or arbitrary response headers. + +## Freeze spending and interpret the report + +Record input, output, cache-read and cache-write usage using the provider's field +names. Usage belongs to model requests; a tool result's byte size is not an +independently measured per-tool token charge. A token-counting endpoint may return +an estimate; a character limit is not +a verified token bound. Qualify the bound required by the protocol, reserve cost +before dispatch, and stop when usage or pricing cannot be reconciled. Count token +counting requests and their latency separately from generation requests. + +Keep model generation, retrieval provider overhead, index setup and infrastructure +costs separate. State whether prices are list-price estimates or reconciled +charges. Missing billing evidence stays unknown. Cost per successful resolution +includes failed attempts and has no finite estimate when nothing resolves. + +Measure each attempt from setup through independent grading and cleanup. Report +provider, retrieval and test latency alongside that total; do not add components +that overlap. Preserve in-flight requests and reservations after a timeout. + +Average repeats within tasks and resample paired task clusters for intervals. +Report A/A variation and every task category, including regressions. Missing +attempts suppress complete-cohort intervals. A small historical pilot measures +the declared workflow; it does not establish general adoption, quality or savings. +An exact-lookup tool set also does not measure the full code plane or a plugin's +behavior. Publish results only after the live path and all required receipts +qualify. diff --git a/docs/benchmarks/running.md b/docs/benchmarks/running.md index 7680141a64..36308d64f9 100644 --- a/docs/benchmarks/running.md +++ b/docs/benchmarks/running.md @@ -126,6 +126,9 @@ calibration JSON to any gated bench with `--judge-calibration`. ### Code index +For coding agents, use [paired setup and qualification](paired-agent-setup.md) +to separate source tools, optional retrieval and application-owned context. + The code benches need a server that indexes the fixture's repository (ClickHouse/ClickHouse for the default fixture) and a real embedder. Org and namespace come from `LOOM_CODE_ORG` / `LOOM_CODE_NAMESPACE` or `--org` / `--namespace`. diff --git a/llms.txt b/llms.txt index 8c958c7a15..21f1fb884d 100644 --- a/llms.txt +++ b/llms.txt @@ -185,6 +185,7 @@ Start with [the docs map](docs/index.md). To install Loom, a coding agent reads - [Adding a benchmark case, dataset or suite](docs/benchmarks/adding-a-case.md): The committed sets are small (set sizes), so a new case is the cheapest way to improve the suite. - [Benchmark coverage](docs/benchmarks/coverage.md): This matrix shows which parts of Loom's memory, knowledge and code flows have a reproducible public measurement. - [Benchmark methodology](docs/benchmarks/methodology.md): This page covers what the harnesses compute, how a delta is judged against noise, when two numbers compare, and where the numbers are biased. +- [Paired coding-agent benchmark setup](docs/benchmarks/paired-agent-setup.md): A coding-agent comparison needs an isolated repository checkout, a matching code index and an instrumented agent runner. - [Benchmark results](docs/benchmarks/results.md): This page lists only numbers that a committed file reproduces. - [Running the benchmarks](docs/benchmarks/running.md): Run every command from the repository root with the checkout's interpreter and the root first on `PYTHONPATH`: - [SDK issue resolution](docs/benchmarks/sdk-issue-resolution.md): The SDK issue harness measures proposed patches with and without Loom code lookup.