Repository navigation
docs(bench): explain how to add a benchmark case to the eval set - #55
Merged
Merged
Conversation
Add an 'Adding a benchmark case' section to bench/README.md between the
|
@wufangyong973 is attempting to deploy a commit to the vignesh's projects Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #39.
Fossel has a strong measure retrieval changes before merging culture, but
bench/README.mdstopped at describing what the benchmark does and never explained the contribution side — how to add a new query/expected-result pair to the eval set and interpret the result.This adds a single Adding a benchmark case section, placed between the existing
DatasetsandTuning sweepssections, that covers:memories[]({key, note, type, tags}) andqueries[]({id, query, relevant, category, rationale}) inbench/datasets/repo-memory.json, cross-referenced bykeynpm run benchto refresh the committed snapshot underbench/results/, then verify withnpm run bench:checksearchvscontextsurfaces,byCategory, hit@k / MRR, and the per-mode miss list as the design tool)Everything is grounded in files that already exist — the committed dataset,
bench/run.ts(the--check/--no-writesnapshot logic) and the currentREADME.md. Pure documentation, no harness change. A contributor should now be able to add a case and interpret the result without reading the harness source.