Skip to content

docs(bench): add an 'Adding a benchmark case' guide - #51

Closed
wufangyong973 wants to merge 1 commit into
7vignesh:mainfrom
wufangyong973:gfi/39-bench-guide
Closed

wufangyong973 wants to merge 1 commit into
7vignesh:mainfrom
wufangyong973:gfi/39-bench-guide

Conversation

@wufangyong973

Copy link
Copy Markdown
Contributor

Closes #39.

bench/README.md explained what the benchmark does but not how to add a
case, even though the project's standing rule is to measure retrieval
changes before merging and to keep a change only if the numbers support
it.

Adds an "Adding a benchmark case" section covering:

  • the shape of the committed eval set (bench/datasets/repo-memory.json),
    with concrete examples of a memory entry and a query entry, plus the
    recency-order and no-write-path-inference constraints that keep the
    numbers honest;
  • how to run npm run bench (with --verbose to see every miss) and which
    of the two reported tables is the retrieval-quality number;
  • how npm run bench:check works (fails on drift, does not write) and
    when to update the committed snapshot in bench/results/.

Docs-only, 92 lines added to one file.

Signed-off-by: wfy18 <wfy18056200731@163.com>
@vercel

vercel Bot commented Oct 12, 2026

Copy link
Copy Markdown

@wufangyong973 is attempting to deploy a commit to the vignesh's projects Team on Vercel.

A member of the Team first needs to authorize it.

@7vignesh

Copy link
Copy Markdown
Owner

Superseded by your revised version in #55 - the more concise rewrite is the one we are taking. Closing as superseded, not rejected. Thanks for tightening it up. (#55 closes #39.)

@7vignesh 7vignesh closed this Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bench/README: add a "how to add a benchmark case" guide

2 participants