Skip to content

BOOTH's JSON-only prompt suppresses reasoning on a 7B model (GSM8K 97.5% → 35.0%): fixes for 0.5.9, API design for 0.6.0 #33

Open

Description

@Vedantgitbot

Background

This came out of #32 (latency/accuracy benchmark example, examples/Vedantgitbot_latency_accuracy/). The first run on qwen2.5:7b-instruct showed BOOTH at 42.0% accuracy against 82.0% for the plain call. I assumed the gate was to blame. Two diagnostic arms added to the example (BOOTH's exact prompt with no gate, and a reasoning-first suffix) show that most of the gap comes from the prompt instead.

This is one 7B model, one seed, 100 questions (30 TriviaQA, 30 TruthfulQA-MC, 40 GSM8K). It's enough to document a pitfall, but not enough to decide what the library should ship, so I'd like others to try other models.

Evidence (boothpy 0.5.8, full tables in the example's RESULTS.md)

Arm Overall GSM8K TruthfulQA TriviaQA Mean tokens (GSM8K)
Plain call 82.0 97.5 76.7 66.7 261
BOOTH's prompt, no gate 50.0 35.0 56.7 63.3 36
booth.check() default 42.0 35.0 30.0 63.3 36
booth.check() with a reasoning-first suffix 65.0 92.5 36.7 56.7 267
  • _CONFIDENCE_SUFFIX ends with "Output ONLY the JSON object, nothing else." The GSM8K base prompt says "Reason step by step". The two instructions contradict each other, and the model follows the last one.
  • Letting the model reason before the JSON recovers GSM8K to 92.5%. It does not help TruthfulQA or TriviaQA, and it makes BOOTH about 1.8x slower than the plain call on mean latency.
  • The AMBIGUOUS gate rejected 15 of 100 questions (14 of them TruthfulQA). The plain call answered 13 of those 15 correctly.
  • Self-reported confidence still didn't separate right from wrong: 28 of 43 accepted wrong answers had confidence >= 0.95.

The reasoning arm was built from outside the library: a call_fn wrapper swaps the suffix and returns only the final JSON object. It works on 0.5.8 as released, but it depends on the private _CONFIDENCE_SUFFIX, so it could break in any release.

Plan

Two tracks, split by whether the public API changes.

0.5.9 (no new API; ships when ready)

  • Docs: a README/TROUBLESHOOTING note that the JSON-only suffix hurts reasoning-heavy tasks on small models, with the example link and a short call_fn recipe.
  • Any small fix that doesn't change the public API and that someone can show is strictly better. Example: resolving the contradiction between "reason step by step" in a caller's prompt and "JSON only" in the suffix, if a safe wording exists.
  • Anything that would change default behaviour for existing users needs evidence on more than one model first.

0.6.0 (new public API; design agreed here before any PR)

  1. Caller-supplied suffix: a parameter that replaces _CONFIDENCE_SUFFIX; BOOTH parses the last JSON object in the reply. This is the smallest change and keeps BOOTH's prompt surface small.
  2. Opt-in reasoning mode: BOOTH supplies a reasoning-first suffix itself. Friendlier, but BOOTH then owns a second prompt, and the data above shows it isn't a general win.
  3. Both, or neither, depending on what the replication shows.

I lean towards option 1, but I'd like to hear other views.

Questions

  • Does this reproduce on other models (larger, smaller, or models that reason by default)? run_example.py takes --model and runs in about an hour on a laptop.
  • Does moving ambiguous/interpretations after answer in the JSON schema change the 15 AMBIGUOUS rejections? I haven't tested that.
  • Is there a safe prompt wording that is better for all tasks, or does this need a per-task switch?

CC: @rk042 @Kaan081 @Emart29

I'd really appreciate it if you could review the design and share your thoughts, given your experience with parsing, robustness, and calibration. Your feedback would be very helpful before anything is added to core. Thanks!

Activity

  1. rk042 commented on Nov 4, 2026

    @rk042
    Contributor

    Hey guys! Just wanted to share that we now have a [General Discussion thread](#34).

    Let's use it for general conversations, longer discussions, meetings, product planning, and project vision.

    This will help us keep issue-specific technical discussions separate from broader product and development conversations, making everything easier to track and organize.

    cc:: @Vedantgitbot @Kaan081 @Emart29

  2. Vedantgitbot commented on Nov 7, 2026

    @Vedantgitbot
    OwnerAuthor

    Great, thanks @NitinGangwal
    A few tips for the run:

    • Pull the latest main first. The script was updated recently (4 arms, 100 questions, 1 repetition by default).
    • Do a quick --smoke run first (10 questions) to check everything works before the full run.
    • Use a model that fits your RAM, with the exact tag, e.g. ollama pull llama3:8b, then python examples/Vedantgitbot_latency_accuracy/run_example.py --model llama3:8b. On a laptop the full run can take an hour or more. If it's too slow, --n 50 is fine, just say so in your post.
    • When it finishes, paste the contents of output/summary.md as a comment on Example: Benchmark BOOTH's Latency and Accuracy Overhead (100 Questions) #32. It already records the model digest, Ollama version and hardware, so please don't edit the numbers.

    Since you're doing #32, If anything fails or looks odd, post it anyway, because failures are useful too. @rk042 and I are around if you get stuck.

  3. rk042 commented on Nov 14, 2026

    @rk042
    Contributor

    Hi team, I’ve opened draft PR #35 with the parser regression tests we discussed. It covers candidate ordering, nested objects, fallback behavior, retries, and sync/async checks.

    The eight failing tests are expected against the current parser. I’ll rerun them once your implementation is available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions