Skip to content

Document paired coding-agent benchmark setup - #1080

Open
zlareb1 wants to merge 3 commits into
mainfrom
docs/paired-factory-benchmark-setup
Open

zlareb1 wants to merge 3 commits into
mainfrom
docs/paired-factory-benchmark-setup

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Related: #1050
Related: #1052

Changelog category (leave one):

  • Documentation (changelog entry is not required)

Changelog entry

Not required.

Why

A paired coding-agent benchmark can produce invalid comparisons when the index contains future source, the evaluator silently skips a patch, or a logical tool counter misses physical requests. Operators need setup checks before spending on model attempts.

Scope and risk

Add a setup guide for paired attempts, private indexes at task bases, tree-sitter qualification, evaluator controls, transport accounting, spending and interpretation. Link it from the running instructions and regenerate llms.txt.

The guide distinguishes scripted qualification from live measurements and describes the limits of candidate edit filters and small pilots. It separates status-cache refreshes from source drift and checks native tool-result ordering through application hooks. It reports no benchmark results. Runtime code, authentication and stored data are unchanged.

Compatibility, rollout and rollback

  • Wire: none.
  • Operator steps: none for this documentation change. The guide lists the checks required when preparing an experiment.
  • Rollback: remove the guide and its link, then regenerate llms.txt. No stored data or migrations remain.

Tests

  • LOOM_ENV_FILE= .venv/bin/python -m pytest tests/unit/test_doc_links.py tests/unit/test_public_copy.py tests/unit/test_docs_reference.py -q: 291 passed.
  • .venv/bin/python -m scripts.codegen.docs_reference --check: generated references are current.
  • .venv-docs/bin/mkdocs build --strict: passed with the pinned documentation dependencies.
  • npx --yes prettier@3.9.9 --check docs/benchmarks/paired-agent-setup.md docs/benchmarks/running.md: passed.
  • python3 .claude/skills/loom-writing/scripts/check_prose.py docs/benchmarks/paired-agent-setup.md: passed.
  • git diff --cached --check: passed.

The integration part 4 job fails because the replicated-erasure test fixture does not create _migration_lock. The same two tests fail on main: main failure, PR failure. This diff changes documentation and its generated index only; the runtime failure is outside its scope.

AI assistance in the code

  • AI-generated: Codex wrote the guide, checked its configuration and CLI examples against the implementation, and ran the checks listed above. Human review is pending.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • make format-check, make lint, make typecheck and make test-unit pass; the integration suite passes if server behaviour changed.
  • Generated documentation is regenerated, not edited by hand.
  • No migration or plugin behaviour changes.
  • No frontend changes.
  • Docs describe the system as it is and name the setup checks.
  • Repository text passes the public-copy gate.
  • Commits are signed off (git commit -s).

Describe the index, evaluator and transport checks required before comparing tool adoption, cost and elapsed time.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
@vercel

vercel Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
loom-docs Ready Ready Preview Nov 20, 2026 10:51pm UTC

Request Review

Status reads can refresh a shared cache without changing source rows, revision watermarks or extractor stamps.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Verification hooks can add context beside tool results, so the native ordering contract needs a check beyond a baseline probe.

Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant