Repository navigation
Add local confidence calibration example - #37
Vedantgitbot merged 2 commits into
Conversation
|
Thanks @rk042, the 12/15/3 split and the 0.556 ECE check out. Two small requests before I merge: run_example.py exits unless boothpy is exactly 0.5.8, and 0.5.9 is now the default install. Please record the version in the manifest and warn, or accept both 0.5.8 and 0.5.9. The script only parses a cached reply with max_retries=0. Optional: add the final run’s machine and Python version to RESULTS.md. A second model with varying confidence can be a follow-up, and #24 can stay open for it. |
Thanks for the review! I’ve updated the PR:
I checked the 30 cached replies with the 0.5.9 source and ran the test suite: 436 passed, 6 skipped. The PR is ready for another review. Thanks! |
|
@rk042 lgtm merging it |
What does this change?
Adds a local Ollama example that compares self-reported confidence with
correctness on a fixed set of self-authored arithmetic questions.
Related to #24.
Why?
This example explores whether a confidence threshold separates correct
answers from wrong ones. In the 30-question run, 27 answers were scoreable:
12 were correct and 15 were wrong. All 27 reported confidence 1.0.
The other three answers were excluded by the integer-only grading rule.
RESULTS.md includes the raw responses, exclusions, confidence bins, ECE,
threshold coverage, pilot results, and limitations.
Checklist
pytest -v) and it passes.CHANGELOG.mdentry following the existing format.The unchecked items are not applicable: this PR adds an example, does not
change public behavior, and is not a bug fix.
Anything else worth knowing?
The study uses one local model and 30 arithmetic questions. Ollama receives
a short custom prompt with a JSON schema; BOOTH checks the returned response.
It does not compare BOOTH's default and reasoning prompts.
The settings were selected after exploratory pilots. The results are limited
to this sample and setup; they should not be generalized to other models,
questions, or prompts.