benchmarks: Add zvec-grep to benchmarks - #268
Merged
Merged
Conversation
Adds benchmarks/baselines/zvecgrep.py following the existing baseline pattern (index + hybrid query, parsed from its markdown CLI output since it has no --json flag), plus results from a full 63-repo run (avg NDCG@10 0.670, index 3.95s, query p50 386ms) folded into the README tables, by-language breakdown, and speed/quality plots. Claude-Session: https://claude.ai/code/session_013GB3H6NiUFKsyAVC3aQNZf
Regenerate the conflicting cold plot from the merged plot.py (main's new semble index time plus the zvec-grep point). Label zvec-grep left of its point so it no longer collides with ColGREP (cold) or runs off the right edge (warm), and note in the benchmark README that it was measured on v0.2.1 in direct mode, one process per query like the other CLI baselines. Claude-Session: https://claude.ai/code/session_01N5WnCfeaj66TgZTbqm2Zew
Enlarge the figure to 10x6.25 so zvec-grep and ColGREP no longer collide, keep every label right of its point, and extend the warm axis to 1 s (with a tick) so zvec-grep's label stays inside the plot. Claude-Session: https://claude.ai/code/session_01N5WnCfeaj66TgZTbqm2Zew
The offset is fixed in cube-root(ms) space, so the wider canvas turned the same gap into ~48px instead of ~37px. The warm plot is unaffected: its axis grew with the canvas. Claude-Session: https://claude.ai/code/session_01N5WnCfeaj66TgZTbqm2Zew
The old offset was a fixed distance in cube-root(ms) space, so the visible gap depended on marker size, axis range and figure size — labels drifted far from the smaller bubbles. Offsetting by the marker's radius plus a fixed 3pt keeps the gap identical everywhere, which also removes the two per-plot delta constants. With labels attached to their bubbles the extra canvas is no longer needed to keep zvec-grep and ColGREP apart, so the figure drops back to 9x5.6 and labels go to 9.5pt to restore the density of the original plots. Claude-Session: https://claude.ai/code/session_01N5WnCfeaj66TgZTbqm2Zew
Codecov Report✅ All modified and coverable lines are covered by tests. 🚀 New features to boost your workflow:
|
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Address review comments: pin the reproduction install to @0.2.1 and record `zg --version` in the results file, since the runner parses zg's human-readable output and a CLI change would silently alter the numbers. Also correct the model references the default bump to potion-code-16M-v2 left behind: the methodology line and four ablation rows still named potion-code-16M, while both ablation result files record v2. Claude-Session: https://claude.ai/code/session_01N5WnCfeaj66TgZTbqm2Zew
Its latency came from benchmarks.baselines.zvecgrep (63 repos, 3 runs per query), while the published speed figures use speed_benchmark's 19 repos and 5 consecutive runs. Add a zvec-grep bench there and generalise --semble-only into --tools/--repo so one tool can be re-measured without re-running the slow baselines. Re-measured: index 4.0s -> 3.4s, query p50 386ms -> 391ms. The p50 barely moves because direct mode spawns a zg process per query, but the row is now comparable to the others in the table. Also fix _AsymmetricWrapper.encode, which rejected the max_length argument semble has passed since #264 — CodeRankEmbed crashed any full run. Claude-Session: https://claude.ai/code/session_01N5WnCfeaj66TgZTbqm2Zew
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds zvec-grep to our benchmarks.