Skip to content

docs(bench): explain how to add a benchmark case to the eval set - #55

Merged
7vignesh merged 2 commits into
7vignesh:mainfrom
wufangyong973:gfi/39-bench-add-case
Oct 13, 2026
Merged

7vignesh merged 2 commits into
7vignesh:mainfrom
wufangyong973:gfi/39-bench-add-case

Conversation

@wufangyong973

Copy link
Copy Markdown
Contributor

Closes #39.

Fossel has a strong measure retrieval changes before merging culture, but bench/README.md stopped at describing what the benchmark does and never explained the contribution side — how to add a new query/expected-result pair to the eval set and interpret the result.

This adds a single Adding a benchmark case section, placed between the existing Datasets and Tuning sweeps sections, that covers:

  • the shape of a dataset entry — memories[] ({key, note, type, tags}) and queries[] ({id, query, relevant, category, rationale}) in bench/datasets/repo-memory.json, cross-referenced by key
  • the four-step add-a-case flow: add a memory if needed, add a query, re-run npm run bench to refresh the committed snapshot under bench/results/, then verify with npm run bench:check
  • how to read the output (the search vs context surfaces, byCategory, hit@k / MRR, and the per-mode miss list as the design tool)
  • the merge rule: keep a retrieval change only when the numbers support it

Everything is grounded in files that already exist — the committed dataset, bench/run.ts (the --check / --no-write snapshot logic) and the current README.md. Pure documentation, no harness change. A contributor should now be able to add a case and interpret the result without reading the harness source.

Add an 'Adding a benchmark case' section to bench/README.md between the
@vercel

vercel Bot commented Oct 13, 2026

Copy link
Copy Markdown

@wufangyong973 is attempting to deploy a commit to the vignesh's projects Team on Vercel.

A member of the Team first needs to authorize it.

@7vignesh
7vignesh merged commit da7a94f into 7vignesh:main Oct 13, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bench/README: add a "how to add a benchmark case" guide

2 participants