Skip to content

Add local confidence calibration example - #37

Merged
Vedantgitbot merged 2 commits into
Vedantgitbot:mainfrom
rk042:example/issue-24-confidence-calibration
Nov 15, 2026
Merged

Vedantgitbot merged 2 commits into
Vedantgitbot:mainfrom
rk042:example/issue-24-confidence-calibration

Conversation

@rk042

@rk042 rk042 commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

What does this change?

Adds a local Ollama example that compares self-reported confidence with
correctness on a fixed set of self-authored arithmetic questions.

Related to #24.

Why?

This example explores whether a confidence threshold separates correct
answers from wrong ones. In the 30-question run, 27 answers were scoreable:
12 were correct and 15 were wrong. All 27 reported confidence 1.0.

The other three answers were excluded by the integer-only grading rule.
RESULTS.md includes the raw responses, exclusions, confidence bins, ECE,
threshold coverage, pilot results, and limitations.

Checklist

  • I ran the full test suite locally (pytest -v) and it passes.
  • I added a new test that fails without this change and passes with it.
  • This does not modify any existing test's behavior/assertions, unless that's the explicit point of this PR.
  • If this changes public behavior, I added a CHANGELOG.md entry following the existing format.
  • If this is a bug fix, I can point to how I reproduced the original bug.

The unchecked items are not applicable: this PR adds an example, does not
change public behavior, and is not a bug fix.

Anything else worth knowing?

The study uses one local model and 30 arithmetic questions. Ollama receives
a short custom prompt with a JSON schema; BOOTH checks the returned response.
It does not compare BOOTH's default and reasoning prompts.

The settings were selected after exploratory pilots. The results are limited
to this sample and setup; they should not be generalized to other models,
questions, or prompts.

@Vedantgitbot

Copy link
Copy Markdown
Owner

Thanks @rk042, the 12/15/3 split and the 0.556 ECE check out. Two small requests before I merge:

run_example.py exits unless boothpy is exactly 0.5.8, and 0.5.9 is now the default install. Please record the version in the manifest and warn, or accept both 0.5.8 and 0.5.9. The script only parses a cached reply with max_retries=0.
Please say up front, in the first paragraph of RESULTS.md and in the PR description, that the model gets a custom short prompt plus a JSON schema and BOOTH only parses the saved reply. The result is the model’s stated confidence under constrained decoding, not the default BOOTH flow.

Optional: add the final run’s machine and Python version to RESULTS.md. A second model with varying confidence can be a follow-up, and #24 can stay open for it.

@rk042

rk042 commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @rk042, the 12/15/3 split and the 0.556 ECE check out. Two small requests before I merge:

run_example.py exits unless boothpy is exactly 0.5.8, and 0.5.9 is now the default install. Please record the version in the manifest and warn, or accept both 0.5.8 and 0.5.9. The script only parses a cached reply with max_retries=0. Please say up front, in the first paragraph of RESULTS.md and in the PR description, that the model gets a custom short prompt plus a JSON schema and BOOTH only parses the saved reply. The result is the model’s stated confidence under constrained decoding, not the default BOOTH flow.

Optional: add the final run’s machine and Python version to RESULTS.md. A second model with varying confidence can be a follow-up, and #24 can stay open for it.

Thanks for the review! I’ve updated the PR:

  • The script now records the installed boothpy version and warns rather than exiting when it differs from 0.5.8.
  • The first paragraph of RESULTS.md and the PR description now explain the custom prompt, JSON schema and cached-reply parsing.
  • I added the final run’s Python, machine, Ollama and model details.

I checked the 30 cached replies with the 0.5.9 source and ran the test suite: 436 passed, 6 skipped.

The PR is ready for another review. Thanks!

@Vedantgitbot

Copy link
Copy Markdown
Owner

@rk042 lgtm merging it

@Vedantgitbot
Vedantgitbot merged commit a34ebcc into Vedantgitbot:main Nov 15, 2026
8 checks passed
@rk042
rk042 deleted the example/issue-24-confidence-calibration branch November 15, 2026 12:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants