Repository navigation
docs(benchmarks): add Gemma 3 12B replication results - #38
Merged
Vedantgitbot merged 1 commit intoNov 16, 2026
Merged
Conversation
Owner
|
Thanks @camerontjs-dot, this is careful and well disclosed. I recomputed the accuracy, per-dataset, latency and net-recovery figures from your tables and they all hold. Merging as is. One note: the “Accuracy by dataset” table in the generated summary has a separator-row bug from the runner, which I’ll fix separately, so it may not render as a table on GitHub. Your numbers are unaffected. If you’d like to continue, the follow-up you suggested (matched numeric questions with raw replies retained) would be very welcome. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change?
Adds the Gemma 3 12B results requested in #32:
RESULTS_gemma3-12b.md: the write-up as posted, byte for byte, including the generated summary, exact commands, source commit, Codex disclosure and separate interpretation.results_gemma3-12b.jsonl: the original 200 per-question records (69,542 bytes), unchanged.Why?
This puts the second-model run alongside the existing benchmark results. The run used unchanged upstream
a51f30c173776bfb6a47781a1e52b03f39b258a2, BOOTH 0.5.8, local Ollamagemma3:12b, fifty questions and one repetition. It repeats the protocol on a smaller sample; it does not reproduce Qwen's exact 100-question measurements.Verification
summary.mdbyte for byte.(rep, qid, arm)records are unique and complete; regrading from the saved manifest reproduced the reported scores.Record SHA-256:
c78e27c7eaf2d7f1764cdb787799b6885883ae736bca4bcdc61af0fbef5382f9.Checklist
Anything else worth knowing?
As requested,
run_example.py,RESULTS.mdand.gitignoreare unchanged. Raw provider replies were not retained by the original runner; the JSONL contains its per-question records. There is no new inference in this PR. The suggested numeric follow-up remains explicitly marked as not run.