Repository navigation
Description
Background
This came out of #32 (latency/accuracy benchmark example, examples/Vedantgitbot_latency_accuracy/). The first run on qwen2.5:7b-instruct showed BOOTH at 42.0% accuracy against 82.0% for the plain call. I assumed the gate was to blame. Two diagnostic arms added to the example (BOOTH's exact prompt with no gate, and a reasoning-first suffix) show that most of the gap comes from the prompt instead.
This is one 7B model, one seed, 100 questions (30 TriviaQA, 30 TruthfulQA-MC, 40 GSM8K). It's enough to document a pitfall, but not enough to decide what the library should ship, so I'd like others to try other models.
Evidence (boothpy 0.5.8, full tables in the example's RESULTS.md)
| Arm | Overall | GSM8K | TruthfulQA | TriviaQA | Mean tokens (GSM8K) |
|---|---|---|---|---|---|
| Plain call | 82.0 | 97.5 | 76.7 | 66.7 | 261 |
| BOOTH's prompt, no gate | 50.0 | 35.0 | 56.7 | 63.3 | 36 |
booth.check() default |
42.0 | 35.0 | 30.0 | 63.3 | 36 |
booth.check() with a reasoning-first suffix |
65.0 | 92.5 | 36.7 | 56.7 | 267 |
_CONFIDENCE_SUFFIXends with "Output ONLY the JSON object, nothing else." The GSM8K base prompt says "Reason step by step". The two instructions contradict each other, and the model follows the last one.- Letting the model reason before the JSON recovers GSM8K to 92.5%. It does not help TruthfulQA or TriviaQA, and it makes BOOTH about 1.8x slower than the plain call on mean latency.
- The AMBIGUOUS gate rejected 15 of 100 questions (14 of them TruthfulQA). The plain call answered 13 of those 15 correctly.
- Self-reported confidence still didn't separate right from wrong: 28 of 43 accepted wrong answers had confidence >= 0.95.
The reasoning arm was built from outside the library: a call_fn wrapper swaps the suffix and returns only the final JSON object. It works on 0.5.8 as released, but it depends on the private _CONFIDENCE_SUFFIX, so it could break in any release.
Plan
Two tracks, split by whether the public API changes.
0.5.9 (no new API; ships when ready)
- Docs: a README/TROUBLESHOOTING note that the JSON-only suffix hurts reasoning-heavy tasks on small models, with the example link and a short
call_fnrecipe. - Any small fix that doesn't change the public API and that someone can show is strictly better. Example: resolving the contradiction between "reason step by step" in a caller's prompt and "JSON only" in the suffix, if a safe wording exists.
- Anything that would change default behaviour for existing users needs evidence on more than one model first.
0.6.0 (new public API; design agreed here before any PR)
- Caller-supplied suffix: a parameter that replaces
_CONFIDENCE_SUFFIX; BOOTH parses the last JSON object in the reply. This is the smallest change and keeps BOOTH's prompt surface small. - Opt-in reasoning mode: BOOTH supplies a reasoning-first suffix itself. Friendlier, but BOOTH then owns a second prompt, and the data above shows it isn't a general win.
- Both, or neither, depending on what the replication shows.
I lean towards option 1, but I'd like to hear other views.
Questions
- Does this reproduce on other models (larger, smaller, or models that reason by default)?
run_example.pytakes--modeland runs in about an hour on a laptop. - Does moving
ambiguous/interpretationsafteranswerin the JSON schema change the 15 AMBIGUOUS rejections? I haven't tested that. - Is there a safe prompt wording that is better for all tasks, or does this need a per-task switch?
I'd really appreciate it if you could review the design and share your thoughts, given your experience with parsing, robustness, and calibration. Your feedback would be very helpful before anything is added to core. Thanks!
Activity
rk042 commentedon Nov 4, 2026
Hey guys! Just wanted to share that we now have a [General Discussion thread](#34).
Let's use it for general conversations, longer discussions, meetings, product planning, and project vision.
This will help us keep issue-specific technical discussions separate from broader product and development conversations, making everything easier to track and organize.
cc:: @Vedantgitbot @Kaan081 @Emart29
Vedantgitbot commentedon Nov 7, 2026
Great, thanks @NitinGangwal
A few tips for the run:
mainfirst. The script was updated recently (4 arms, 100 questions, 1 repetition by default).--smokerun first (10 questions) to check everything works before the full run.ollama pull llama3:8b, thenpython examples/Vedantgitbot_latency_accuracy/run_example.py --model llama3:8b. On a laptop the full run can take an hour or more. If it's too slow,--n 50is fine, just say so in your post.output/summary.mdas a comment on Example: Benchmark BOOTH's Latency and Accuracy Overhead (100 Questions) #32. It already records the model digest, Ollama version and hardware, so please don't edit the numbers.Since you're doing #32, If anything fails or looks odd, post it anyway, because failures are useful too. @rk042 and I are around if you get stuck.
rk042 commentedon Nov 14, 2026
Hi team, I’ve opened draft PR #35 with the parser regression tests we discussed. It covers candidate ordering, nested objects, fallback behavior, retries, and sync/async checks.
The eight failing tests are expected against the current parser. I’ll rerun them once your implementation is available.