Skip to content

docs(benchmarks): add Gemma 3 12B replication results - #38

Merged
Vedantgitbot merged 1 commit into
Vedantgitbot:mainfrom
camerontjs-dot:codex/gemma3-12b-results
Nov 16, 2026
Merged

Vedantgitbot merged 1 commit into
Vedantgitbot:mainfrom
camerontjs-dot:codex/gemma3-12b-results

Conversation

@camerontjs-dot

Copy link
Copy Markdown
Contributor

What does this change?

Adds the Gemma 3 12B results requested in #32:

  • RESULTS_gemma3-12b.md: the write-up as posted, byte for byte, including the generated summary, exact commands, source commit, Codex disclosure and separate interpretation.
  • results_gemma3-12b.jsonl: the original 200 per-question records (69,542 bytes), unchanged.

Why?

This puts the second-model run alongside the existing benchmark results. The run used unchanged upstream a51f30c173776bfb6a47781a1e52b03f39b258a2, BOOTH 0.5.8, local Ollama gemma3:12b, fifty questions and one repetition. It repeats the protocol on a smaller sample; it does not reproduce Qwen's exact 100-question measurements.

Verification

  • The document matches the published comment, and its generated summary matches the original summary.md byte for byte.
  • The JSONL matches the original saved records. All 200 (rep, qid, arm) records are unique and complete; regrading from the saved manifest reproduced the reported scores.
  • All 93 sealed original artifacts still match their recorded hashes.
  • The full current test suite passed: 436 passed, 6 skipped.
  • The diff contains only these two additions. Added files passed the staged checks for local paths and secret patterns.

Record SHA-256: c78e27c7eaf2d7f1764cdb787799b6885883ae736bca4bcdc61af0fbef5382f9.

Checklist

  • I ran the full test suite locally and it passes (436 passed, 6 skipped).
  • New regression test: not applicable; this adds saved results.
  • Existing tests are unchanged.
  • Changelog entry and bug reproduction: not applicable; this adds saved results.

Anything else worth knowing?

As requested, run_example.py, RESULTS.md and .gitignore are unchanged. Raw provider replies were not retained by the original runner; the JSONL contains its per-question records. There is no new inference in this PR. The suggested numeric follow-up remains explicitly marked as not run.

@Vedantgitbot

Copy link
Copy Markdown
Owner

Thanks @camerontjs-dot, this is careful and well disclosed. I recomputed the accuracy, per-dataset, latency and net-recovery figures from your tables and they all hold. Merging as is. One note: the “Accuracy by dataset” table in the generated summary has a separator-row bug from the runner, which I’ll fix separately, so it may not render as a table on GitHub. Your numbers are unaffected. If you’d like to continue, the follow-up you suggested (matched numeric questions with raw replies retained) would be very welcome.

@Vedantgitbot
Vedantgitbot merged commit eceb8ba into Vedantgitbot:main Nov 16, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants