You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Assigned to @Emart29. Follows #2 (provider examples) and informs #21 (booth.measure()).
Question: does the 0.7 self-reported confidence threshold separate right answers from wrong ones?
@Emart29 will paste the full scope as the first comment. Rules are the same as other examples: one folder under examples/, no changes to src/booth/, no new BOOTH dependencies, no keys committed, real output pasted into RESULTS.md, and negative results welcome. Claim window: 14 days.
Hi @Vedantgitbot and @Emart29, is this calibration example still being worked on? In my small Issue #19 experiment, the local 7B model reported confidence 1.0 on both correct and incorrect answers, although the sample was too small to draw calibration conclusions.
If help with another model, a reproducible run, or reviewing results would be useful, I’d be happy to contribute.
Hi @rk042 , thanks for checking in. I’ve stepped back from this one, so please go ahead and take the calibration example if you’d like. Your Issue #19 result (confidence 1.0 on both right and wrong answers) is a good starting point. Good luck with it
Thanks @Emart29 for being upfront about it, and @rk042, yes please, this is yours. I’ve assigned it to you.
Scope. Does the self-reported confidence (and the 0.7 threshold) separate right answers from wrong ones? Same rules as the other examples: one folder under examples/, no changes to src/booth/, no new dependencies, no keys committed, real output pasted into RESULTS.md. Negative results are welcome. New claim window: 14 days from today, and just say if you need longer.
What we already know, so you don’t repeat it. In the benchmark from #32/#33 (qwen2.5:7b and gemma3:12b, default prompt), confidence and the AMBIGUOUS flag did not separate right from wrong answers. Your #19 run points the same way. Those samples were small and not set up as a calibration study, which is what this issue should be.
Suggestions, not requirements:
Use datasets with ground truth, and aim for at least 100 questions per model. Report the answered/wrong split per confidence bin, plus a threshold sweep (accuracy and coverage at 0.5, 0.7, 0.9, and so on).
Add a simple calibration number (ECE or AUROC) and a few lines on how to read it.
Run the default prompt and the reasoning-room variant from TROUBLESHOOTING entry 10, since the prompt changes how the model answers.
Save raw replies so results can be re-scored. examples/Vedantgitbot_latency_accuracy/run_example.py already has the dataset loaders and a run structure you can copy.
Two models are better than one. It’s fine to start with one and add more later.
The results will feed into #21 (booth.measure()), which is still an early design, so no API is needed for this issue. Post early results in the thread whenever you have them.
Hi @Vedantgitbot, I’ve opened PR #37 with the calibration example.
In the 30-question arithmetic run, 27 answers were scoreable: 12 were correct and 15 were wrong. All 27 had confidence 1.0, so the 0.7 threshold did not separate correct answers from wrong ones in this sample. Three answers were excluded by the integer-only grading rule.
The example includes the raw responses, confidence bins, ECE, threshold results and pilot findings. It uses one local model and a custom schema prompt, so the results are limited to this small sample and don’t compare BOOTH’s default and reasoning prompts.
Activity
rk042 commentedon Nov 14, 2026
Hi @Vedantgitbot and @Emart29, is this calibration example still being worked on? In my small Issue #19 experiment, the local 7B model reported confidence 1.0 on both correct and incorrect answers, although the sample was too small to draw calibration conclusions.
If help with another model, a reproducible run, or reviewing results would be useful, I’d be happy to contribute.
Emart29 commentedon Nov 14, 2026
Hi @rk042 , thanks for checking in. I’ve stepped back from this one, so please go ahead and take the calibration example if you’d like. Your Issue #19 result (confidence 1.0 on both right and wrong answers) is a good starting point. Good luck with it
Vedantgitbot commentedon Nov 14, 2026
Thanks @Emart29 for being upfront about it, and @rk042, yes please, this is yours. I’ve assigned it to you.
Scope. Does the self-reported confidence (and the 0.7 threshold) separate right answers from wrong ones? Same rules as the other examples: one folder under examples/, no changes to src/booth/, no new dependencies, no keys committed, real output pasted into RESULTS.md. Negative results are welcome. New claim window: 14 days from today, and just say if you need longer.
What we already know, so you don’t repeat it. In the benchmark from #32/#33 (qwen2.5:7b and gemma3:12b, default prompt), confidence and the AMBIGUOUS flag did not separate right from wrong answers. Your #19 run points the same way. Those samples were small and not set up as a calibration study, which is what this issue should be.
Suggestions, not requirements:
Use datasets with ground truth, and aim for at least 100 questions per model. Report the answered/wrong split per confidence bin, plus a threshold sweep (accuracy and coverage at 0.5, 0.7, 0.9, and so on).
Add a simple calibration number (ECE or AUROC) and a few lines on how to read it.
Run the default prompt and the reasoning-room variant from TROUBLESHOOTING entry 10, since the prompt changes how the model answers.
Save raw replies so results can be re-scored. examples/Vedantgitbot_latency_accuracy/run_example.py already has the dataset loaders and a run structure you can copy.
Two models are better than one. It’s fine to start with one and add more later.
The results will feed into #21 (booth.measure()), which is still an early design, so no API is needed for this issue. Post early results in the thread whenever you have them.
rk042 commentedon Nov 15, 2026
Hi @Vedantgitbot, I’ve opened PR #37 with the calibration example.
In the 30-question arithmetic run, 27 answers were scoreable: 12 were correct and 15 were wrong. All 27 had confidence 1.0, so the 0.7 threshold did not separate correct answers from wrong ones in this sample. Three answers were excluded by the integer-only grading rule.
The example includes the raw responses, confidence bins, ECE, threshold results and pilot findings. It uses one local model and a custom schema prompt, so the results are limited to this small sample and don’t compare BOOTH’s default and reasoning prompts.
Thanks, and I’d appreciate your review!