Skip to content

Measure whether BOOTH's self-reported confidence predicts correct answers (calibration example) #24

Open

Description

@Vedantgitbot

Assigned to @Emart29. Follows #2 (provider examples) and informs #21 (booth.measure()).

Question: does the 0.7 self-reported confidence threshold separate right answers from wrong ones?

@Emart29 will paste the full scope as the first comment. Rules are the same as other examples: one folder under examples/, no changes to src/booth/, no new BOOTH dependencies, no keys committed, real output pasted into RESULTS.md, and negative results welcome. Claim window: 14 days.

Activity

  1. rk042 commented on Nov 14, 2026

    @rk042
    Contributor

    Hi @Vedantgitbot and @Emart29, is this calibration example still being worked on? In my small Issue #19 experiment, the local 7B model reported confidence 1.0 on both correct and incorrect answers, although the sample was too small to draw calibration conclusions.

    If help with another model, a reproducible run, or reviewing results would be useful, I’d be happy to contribute.

  2. Emart29 commented on Nov 14, 2026

    @Emart29

    Hi @rk042 , thanks for checking in. I’ve stepped back from this one, so please go ahead and take the calibration example if you’d like. Your Issue #19 result (confidence 1.0 on both right and wrong answers) is a good starting point. Good luck with it

  3. Vedantgitbot commented on Nov 14, 2026

    @Vedantgitbot
    OwnerAuthor

    Thanks @Emart29 for being upfront about it, and @rk042, yes please, this is yours. I’ve assigned it to you.

    Scope. Does the self-reported confidence (and the 0.7 threshold) separate right answers from wrong ones? Same rules as the other examples: one folder under examples/, no changes to src/booth/, no new dependencies, no keys committed, real output pasted into RESULTS.md. Negative results are welcome. New claim window: 14 days from today, and just say if you need longer.

    What we already know, so you don’t repeat it. In the benchmark from #32/#33 (qwen2.5:7b and gemma3:12b, default prompt), confidence and the AMBIGUOUS flag did not separate right from wrong answers. Your #19 run points the same way. Those samples were small and not set up as a calibration study, which is what this issue should be.

    Suggestions, not requirements:

    Use datasets with ground truth, and aim for at least 100 questions per model. Report the answered/wrong split per confidence bin, plus a threshold sweep (accuracy and coverage at 0.5, 0.7, 0.9, and so on).
    Add a simple calibration number (ECE or AUROC) and a few lines on how to read it.
    Run the default prompt and the reasoning-room variant from TROUBLESHOOTING entry 10, since the prompt changes how the model answers.
    Save raw replies so results can be re-scored. examples/Vedantgitbot_latency_accuracy/run_example.py already has the dataset loaders and a run structure you can copy.
    Two models are better than one. It’s fine to start with one and add more later.

    The results will feed into #21 (booth.measure()), which is still an early design, so no API is needed for this issue. Post early results in the thread whenever you have them.

  4. rk042 commented on Nov 15, 2026

    @rk042
    Contributor

    Hi @Vedantgitbot, I’ve opened PR #37 with the calibration example.

    In the 30-question arithmetic run, 27 answers were scoreable: 12 were correct and 15 were wrong. All 27 had confidence 1.0, so the 0.7 threshold did not separate correct answers from wrong ones in this sample. Three answers were excluded by the integer-only grading rule.

    The example includes the raw responses, confidence bins, ECE, threshold results and pilot findings. It uses one local model and a custom schema prompt, so the results are limited to this small sample and don’t compare BOOTH’s default and reasoning prompts.

    Thanks, and I’d appreciate your review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions