Repository navigation
Conversation
Describe the index, evaluator and transport checks required before comparing tool adoption, cost and elapsed time. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Status reads can refresh a shared cache without changing source rows, revision watermarks or extractor stamps. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
Verification hooks can add context beside tool results, so the native ordering contract needs a check beyond a baseline probe. Signed-off-by: Lareb Zafar <lareb.zafar@clickhouse.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: #1050
Related: #1052
Changelog category (leave one):
Changelog entry
Not required.
Why
A paired coding-agent benchmark can produce invalid comparisons when the index contains future source, the evaluator silently skips a patch, or a logical tool counter misses physical requests. Operators need setup checks before spending on model attempts.
Scope and risk
Add a setup guide for paired attempts, private indexes at task bases, tree-sitter qualification, evaluator controls, transport accounting, spending and interpretation. Link it from the running instructions and regenerate
llms.txt.The guide distinguishes scripted qualification from live measurements and describes the limits of candidate edit filters and small pilots. It separates status-cache refreshes from source drift and checks native tool-result ordering through application hooks. It reports no benchmark results. Runtime code, authentication and stored data are unchanged.
Compatibility, rollout and rollback
llms.txt. No stored data or migrations remain.Tests
LOOM_ENV_FILE= .venv/bin/python -m pytest tests/unit/test_doc_links.py tests/unit/test_public_copy.py tests/unit/test_docs_reference.py -q: 291 passed..venv/bin/python -m scripts.codegen.docs_reference --check: generated references are current..venv-docs/bin/mkdocs build --strict: passed with the pinned documentation dependencies.npx --yes prettier@3.9.9 --check docs/benchmarks/paired-agent-setup.md docs/benchmarks/running.md: passed.python3 .claude/skills/loom-writing/scripts/check_prose.py docs/benchmarks/paired-agent-setup.md: passed.git diff --cached --check: passed.The integration part 4 job fails because the replicated-erasure test fixture does not create
_migration_lock. The same two tests fail on main: main failure, PR failure. This diff changes documentation and its generated index only; the runtime failure is outside its scope.AI assistance in the code
AI-generated: Codex wrote the guide, checked its configuration and CLI examples against the implementation, and ran the checks listed above. Human review is pending.
I can explain every line of this diff and how it interacts with the rest of Loom.
Checklist
make format-check,make lint,make typecheckandmake test-unitpass; the integration suite passes if server behaviour changed.git commit -s).