Repository navigation
Description
Anyone adopting BOOTH wants to understand the trade-off between answer reliability and computational cost.
We currently have manually measured numbers (e.g., 3 samples resulting in approximately 3× the model calls in #29), but no controlled, reproducible comparison.
This issue aims to create a small benchmark that runs the same questions through a plain model call and BOOTH, comparing accuracy, latency, and model-call overhead.
What to build
Create a self-contained example:
examples/{github_username}_latency_accuracy/
├── run_example.py
├── RESULTS.md
└── .gitignore1. Question dataset
Use 100 questions by default, with configurable sample sizes of 50, 100, or 200.
Suggested distribution:
| Dataset | Questions | Purpose |
|---|---|---|
| TriviaQA | 30 | General knowledge |
| TruthfulQA | 30 | Misconceptions and misleading questions |
| GSM8K | 40 | Mathematical reasoning |
Also include:
Per-repetition results.
Grading and normalization methodology.
Dataset IDs or a reference to the committed question manifest.
Brief observations, failures, and limitations.
Important: Commit actual benchmark output, not fabricated or illustrative measurements. Report negative or unexpected results honestly.
8. Implementation constraints
No changes to
src/booth/.No new BOOTH dependencies.
No committed API keys or secrets.
Keep the example self-contained and configurable.
Use
.gitignorefor temporary files and local configuration.
Acceptance criteria
Add the three requested files under
examples/{github_username}_latency_accuracy/.Support 50, 100, and 200 questions (100 by default).
Sample from all three suggested datasets using a fixed seed.
Preserve question IDs and reference answers.
Implement deterministic grading.
Implement all three execution arms.
Run at least 3 repetitions with randomized or alternating order.
Report accuracy, latency, runtime, model calls, and all outcome categories.
Include actual results and reproducibility details in
RESULTS.md.Verify the example works with a local model without an API key.
Do not modify BOOTH source code or add dependencies.
Out of scope
Changes to the BOOTH library.
Adding timing to
BoothResult.Calibration analysis (tracked in Measure whether BOOTH's self-reported confidence predicts correct answers (calibration example) #24).
Building a general-purpose benchmark framework.
Contribution claim
To claim this issue, comment with:
Proposed provider.
Model and version.
Local or hosted execution (local preferred).
Claim window: 7 days.
The goal is an honest, reproducible demonstration of BOOTH's accuracy, latency, and model-call trade-offs.
Results from 100 questions are illustrative and should not be treated as general performance claims.
# Example: Benchmark BOOTH's Latency and Accuracy Overhead (100 Questions)Background
Anyone adopting BOOTH wants to understand the trade-off between answer reliability and computational cost.
We currently have manually measured numbers (e.g., 3 samples resulting in approximately 3× the model calls in #29), but no controlled, reproducible comparison.
This issue aims to create a small benchmark that runs the same questions through a plain model call and BOOTH, comparing accuracy, latency, and model-call overhead.
What to build
Create a self-contained example:
examples/{github_username}_latency_accuracy/
├── run_example.py
├── RESULTS.md
└── .gitignore
1. Question dataset
Use 100 questions by default, with configurable sample sizes of 50, 100, or 200.
Suggested distribution:
| Dataset | Questions | Purpose |
|---|---|---|
| [TriviaQA](https://huggingface.co/datasets/mandarjoshi/trivia_qa) | 30 | General knowledge |
| [TruthfulQA](https://huggingface.co/datasets/EleutherAI/truthful_qa_mc) | 30 | Misconceptions and misleading questions |
| [GSM8K](https://huggingface.co/datasets/openai/gsm8k) | 40 | Mathematical reasoning |
Requirements:
- Use a fixed sampling seed.
- Preserve dataset IDs and reference answers.
- Check dataset licences and attribution requirements.
- Use deterministic grading; no LLM judge.
- Document answer normalization and numerical comparison rules.
2. Compare three execution arms
Run the same questions using the same model:
| Arm | Configuration |
|---|---|
| A | Plain model call without BOOTH |
| B | booth.check() with max_retries=0 |
| C | booth.check() with default retries |
Optionally include a fourth arm using check_with_evidence() where suitable evidence is available.
3. Metrics
Report the following for every arm:
| Metric | Description |
|---|---|
| Accuracy | Correct answers (%) |
| Mean / median latency | Seconds per question |
| Total runtime | Total wall-clock time |
| Model calls | Total calls made |
| Rejected | Number of rejected answers |
| AMBIGUOUS | Number of ambiguous results |
| UNCERTAIN | Number of uncertain results |
| Failures | Number of failed calls |
Clearly explain the accuracy denominator. Count unsuccessful responses separately and do not silently exclude them from grading.
4. Fair comparison
- Run at least 3 repetitions by default, configurable by the user.
- Randomize or alternate execution order.
- Use identical model and generation settings across all arms.
- Apply consistent timeout and error-handling policies.
- Report individual repetitions and aggregated results.
- Distinguish setup/warm-up time from benchmark execution time.
5. Model and provider
Support at least one local model so the benchmark can run without an API key.
Suggested configuration:
- Provider: Ollama
- Model:
qwen2.5:7b-instructor another suitable local model. - Temperature: 0.
- Fixed seed where supported.
Keep provider integration simple: isolate model invocation in one function so contributors can switch providers easily.
Hosted providers are welcome, but API keys must come from environment variables.
Use Python's standard library for HTTP requests where possible.
6. Reproducibility
Document and pin the following:
- Latest published
boothpyversion at implementation time (record the exact installed version). - Model tag and digest/revision.
- Provider and generation settings.
- Python version and operating system.
- Hardware (CPU, GPU, and RAM).
- Dataset versions and question-sampling seed.
- Number of questions and repetitions.
7. RESULTS.md — Expected format
Use a consistent reporting format:
Experiment configuration
| Configuration | Value |
|---|---|
| BOOTH version | |
| Python / OS | |
| Provider / Model | |
| Model digest | |
| Temperature / Seed | |
| Dataset versions | |
| Questions / Repetitions | |
| Hardware |
Benchmark comparison
| Metric | Plain model | BOOTH (no retries) | BOOTH (default retries) |
|---|---|---|---|
| Accuracy (%) | |||
| Mean latency (s) | |||
| Median latency (s) | |||
| Total runtime (s) | |||
| Model calls | |||
| Rejected | |||
| AMBIGUOUS | |||
| UNCERTAIN | |||
| Call failures |
Also include:
- Per-repetition results.
- Grading and normalization methodology.
- Dataset IDs or a reference to the committed question manifest.
- Brief observations, failures, and limitations.
Important: Commit actual benchmark output, not fabricated or illustrative measurements. Report negative or unexpected results honestly.
8. Implementation constraints
- No changes to
src/booth/. - No new BOOTH dependencies.
- No committed API keys or secrets.
- Keep the example self-contained and configurable.
- Use
.gitignorefor temporary files and local configuration.
Acceptance criteria
- Add the three requested files under
examples/{github_username}_latency_accuracy/. - Support 50, 100, and 200 questions (100 by default).
- Sample from all three suggested datasets using a fixed seed.
- Preserve question IDs and reference answers.
- Implement deterministic grading.
- Implement all three execution arms.
- Run at least 3 repetitions with randomized or alternating order.
- Report accuracy, latency, runtime, model calls, and all outcome categories.
- Include actual results and reproducibility details in
RESULTS.md. - Verify the example works with a local model without an API key.
- Do not modify BOOTH source code or add dependencies.
Out of scope
- Changes to the BOOTH library.
- Adding timing to
BoothResult. - Calibration analysis (tracked in Measure whether BOOTH's self-reported confidence predicts correct answers (calibration example) #24).
- Building a general-purpose benchmark framework.
Contribution claim
To claim this issue, comment with:
- Proposed provider.
- Model and version.
- Local or hosted execution (local preferred).
Claim window: 7 days.
The goal is an honest, reproducible demonstration of BOOTH's accuracy, latency, and model-call trade-offs.
Results from 100 questions are illustrative and should not be treated as general performance claims.
Activity
Vedantgitbot commentedon Nov 3, 2026
I'll take the lead on the first implementation of this benchmark.
Planned setup:
qwen2.5:7b-instruct)examples/Vedantgitbot_latency_accuracy/check()withmax_retries=0and default retries, compared against a plain model call without BOOTH.I'll use 100 questions by default, run 3 repetitions, and measure accuracy, latency, and model-call overhead across all three approaches.
I'll include the runnable script,
.gitignore, and actual benchmark results with the environment and model details inRESULTS.md.I'm starting with this implementation, but the issue will remain open for others who want to contribute alternative models, approaches, or benchmark results.
camerontjs-dot commentedon Nov 12, 2026
Gemma 3 12B: local four-arm run, 50 questions
I ran the existing four-arm benchmark with
gemma3:12b(Q4_K_M) using local Ollama on an M3 MacBook Air with 24 GiB RAM. Codex handled execution and the checks described below.The source was unchanged upstream
mainat a51f30c173776bfb6a47781a1e52b03f39b258a2. I used an isolated Python environment withboothpy 0.5.8installed from that checkout, and verified that the runner imported that copy.I followed the updated run instructions: smoke first, then four arms and one repetition. The ten-question smoke passed but took 25.8 minutes excluding setup, projecting about 4.3 hours for 100 questions, so I chose the permitted
--n 50before the full run. The model tag and digest were fixed before inference and stayed unchanged.This repeats the protocol on a second model with a smaller sample. It does not reproduce the published Qwen run's exact 100-question measurements. All four Gemma arms used the same 50 questions: 15 TriviaQA, 15 TruthfulQA and 20 GSM8K. Their IDs are the ordered per-dataset prefixes of the published list; no dataset fallback occurred.
Commands below use
pythonto denote the isolated environment's interpreter;RUN_DIRwas a fresh absolute evidence directory. Neither command used--resume.Codex checked all 200 full-run records for complete, unique
(rep, qid, arm)coverage, reconstructed the sampled inputs from the retained dataset cache, checked the randomized execution order, independently recomputed the grades, and reconciled the summary. Source, imports and model identity checks passed. All questions, including rejections, remain in the denominator. Raw records, manifests, cached source rows and command logs are retained locally.Unedited generated
summary.mdfollows.Experiment configuration
gemma3:12bf4031aab637d1ffa37b42570452ae0e4fad0314754d17ded67322e4b95836f8amainat run time (2026-11-12); see Datasets belowArms
booth.check()with default settings.booth.check()with BOOTH's suffix swapped (viacall_fn) for one that lets the model reason before the JSON; same schema.Benchmark comparison
Pooled over 1 repetition(s) x 50 questions = 50 graded question-runs per arm. Accuracy denominator is ALL question-runs: rejected (AMBIGUOUS/UNCERTAIN), call-failed and unanswerable runs count as not correct.
Derived (not part of the required table):
Per-repetition results
Accuracy by dataset (%, pooled over reps)
| Dataset | Questions | Plain model | BOOTH prompt, no gate | BOOTH (default) | BOOTH + reasoning room |
|---|---|---|---|---|
| triviaqa | 15 | 60.0 | 60.0 | 53.3 | 60.0 |
| truthfulqa | 15 | 86.7 | 80.0 | 33.3 | 73.3 |
| gsm8k | 20 | 100.0 | 85.0 | 85.0 | 90.0 |
Mean output tokens per question, by dataset
Datasets
mandarjoshi/trivia_qaEleutherAI/truthful_qa_mcopenai/gsm8kSampling:
random.Random(f"{seed}:{dataset}").sample(...)with seed 42. Question IDs and reference answers are inoutput/manifest.json.Sampled question IDs
triviaqa: sfq_7483, jp_925, qg_2508, bt_2652, qw_9695, odql_9445, sfq_22425, jp_3307, bt_2624, sfq_380, sfq_23048, qw_6473, wh_326, sfq_4851, odql_389
truthfulqa: truthfulqa_mc-validation-600, truthfulqa_mc-validation-267, truthfulqa_mc-validation-248, truthfulqa_mc-validation-179, truthfulqa_mc-validation-323, truthfulqa_mc-validation-163, truthfulqa_mc-validation-51, truthfulqa_mc-validation-318, truthfulqa_mc-validation-632, truthfulqa_mc-validation-151, truthfulqa_mc-validation-236, truthfulqa_mc-validation-280, truthfulqa_mc-validation-641, truthfulqa_mc-validation-312, truthfulqa_mc-validation-21
gsm8k: gsm8k-test-948, gsm8k-test-198, gsm8k-test-248, gsm8k-test-1137, gsm8k-test-1071, gsm8k-test-401, gsm8k-test-3, gsm8k-test-467, gsm8k-test-490, gsm8k-test-785, gsm8k-test-1156, gsm8k-test-135, gsm8k-test-888, gsm8k-test-417, gsm8k-test-359, gsm8k-test-396, gsm8k-test-374, gsm8k-test-827, gsm8k-test-1277, gsm8k-test-681
Interpretation, separate from the generated summary
Possible follow-up, not run: matched numeric questions that change one unit or comparison, such as percent versus percentage points, while retaining raw replies and rejection reasons.
Vedantgitbot commentedon Nov 13, 2026
Thanks @camerontjs-dot, this is very useful. Running an unchanged
mainon a second model, pinning the commit and digest, and re-checking all 200 records and the grades is exactly the kind of replication we needed, and it changed my plan for 0.5.9 (see Discussion #34).Would you open a PR adding this run to the example folder,
examples/Vedantgitbot_latency_accuracy/?RESULTS_gemma3-12b.md, with your write-up as posted: the unedited generated summary, the exact commands, the commit SHA, and your separate interpretation section.results_gemma3-12b.jsonl(the per-question records) if it's small.run_example.py,RESULTS.mdor.gitignore. Other results depend on the runner staying as it was when they were produced, and I'll make any runner changes myself.Please keep the generated summary unedited and keep the Codex disclosure as you wrote it. I'll review and merge, and add a link to your file from
RESULTS.mdafterwards.If you'd rather not do the PR, just let me know and I'll add it with your name credited.