From ca4cb46a526318ed1097e8b6055edf2b8c4fd5c8 Mon Sep 17 00:00:00 2001 From: rk042 Date: Sun, 15 Nov 2026 10:46:01 +0530 Subject: [PATCH 1/2] Add confidence calibration example --- .../rk042_confidence_calibration/.gitignore | 9 + .../rk042_confidence_calibration/RESULTS.md | 146 ++++++++++++++ .../run_example.py | 189 ++++++++++++++++++ 3 files changed, 344 insertions(+) create mode 100644 examples/rk042_confidence_calibration/.gitignore create mode 100644 examples/rk042_confidence_calibration/RESULTS.md create mode 100644 examples/rk042_confidence_calibration/run_example.py diff --git a/examples/rk042_confidence_calibration/.gitignore b/examples/rk042_confidence_calibration/.gitignore new file mode 100644 index 0000000..c4c13e8 --- /dev/null +++ b/examples/rk042_confidence_calibration/.gitignore @@ -0,0 +1,9 @@ +.env +.env.* +.venv/ +venv/ +__pycache__/ +*.py[cod] +*.log +output/ +.cache/ diff --git a/examples/rk042_confidence_calibration/RESULTS.md b/examples/rk042_confidence_calibration/RESULTS.md new file mode 100644 index 0000000..f8d8b00 --- /dev/null +++ b/examples/rk042_confidence_calibration/RESULTS.md @@ -0,0 +1,146 @@ +# Confidence calibration on local arithmetic + +## 30-question study: observed results + +Run 2026-11-15, UTC; 30 planned and attempted calls, 514.1 seconds. All 30 +responses completed, and BOOTH reported `ACCEPTED` / `confidence` for all 30. +There were 27 scoreable answers: **12 correct and 15 wrong**. Three were +excluded under the pre-specified integer-only rule. Among scoreable responses, +all reported confidence 1.0; accuracy was 12/27 = 0.444 and the ten-bin ECE +was 15/27 = 0.556. These describe this sample, not general model calibration. + +```text +Planned: 30; attempted: 30; eligible: 27; excluded: 3 +Exclusions: {'answer is not an integer string': 3} +BOOTH statuses: {'ACCEPTED': 30} +Bins: range, count, mean confidence, observed accuracy + [0.9, 1.0]: 27, 1.000, 0.444 +ECE: 0.556 +Confidence thresholds: threshold, retained, coverage of attempted, retained accuracy + 0.5: 27, 0.900, 0.444 + 0.7: 27, 0.900, 0.444 + 0.9: 27, 0.900, 0.444 +``` + +All scoreable answers had the same confidence, so the threshold sweep did +not distinguish between them. The three excluded replies were `:97` +(reference 87), `{"result":96,"confidence":1}` inside an answer string +(reference 96), and `1.5` (reference 13). The nested result contains the +correct reference, but the fixed integer-only rule does not interpret it. +Thus the 27-answer accuracy is conditional on that rule, not accuracy over all +30 questions. BOOTH acceptance means its output checks passed; it does not +verify factual correctness. + +ECE is the weighted average of the absolute differences between each +confidence bin's mean confidence and observed accuracy; lower is better on +the evaluated sample. Here there is only one populated bin: |1.0 - 12/27|. +The reported 90% threshold coverage counts only scoreable answers that meet +the threshold, divided by all 30 attempts. All 30 responses reported 1.0; +the other three are excluded because their answers could not be scored, not +because their confidence was below the threshold. + +The raw model replies below are pasted from the actual run. Their `raw` values +are JSON-escaped strings and can be recovered with a JSON parser. Locally, the +full records and manifest are in ignored `output/20261115T045922348251Z/`. + + +## Setup + +- Python 3.9+, `boothpy==0.5.8`, Ollama, and the `qwen2.5-coder:7b` model. +- Standard-library HTTP; no new BOOTH dependencies. The questions are + self-authored integer arithmetic and their labels are computed locally. +- Fixed 30-question study, temperature 0.2, seed 101, no retries. The 30 + questions were chosen after the pilots, and are distinct from the recorded + pilot and Issue #19 questions. +- This is a small, hand-selected arithmetic set with one model and one seed; + it is smaller than the issue's suggested 100 questions per model. Results + cannot be generalized to other questions, models, prompts or settings. +- Ollama receives a short custom prompt and JSON schema; BOOTH checks the + returned response. **The model does not receive BOOTH's default prompt.** + This example does not compare default and reasoning prompts. +- The scorer accepts integer strings only. Non-integer answers, parser + failures and incomplete calls are reported and excluded from calibration. + Confidence-bin accuracy and ECE are calculated only on scoreable responses; + threshold coverage uses all attempted calls as its denominator. Incomplete + study runs report both planned and attempted counts. +- An API failure, timeout or incomplete response stops the run; partial raw + records are preserved. A new run makes a new output directory. + +From this folder, after installing Ollama and the model: + +```bash +python3 -m venv .venv +.venv/bin/python -m pip install boothpy==0.5.8 +.venv/bin/python run_example.py +``` + +Raw records and the manifest are stored under the ignored `output/` directory. +They include model replies and questions; review before sharing. The exact +model digest and Ollama version are printed and saved in the manifest. + +## Exploratory pilots + +The approach and study settings were chosen after local feasibility tests; +these results are not a held-out evaluation. + +- GSM8K pilot, qwen2.5-coder:7b, temperature 0: 10 calls using BOOTH's + default and reasoning prompts; 0 scoreable responses. The previous script + did not record completion status, so it cannot determine completion for all + responses. A later completion-check pilot stopped after an empty completed + reply and an incomplete reply. +- GSM8K short-schema pilot, 7B, temperature 0.2: the first call timed out after + 180 seconds. A 3B short-schema pilot attempted three questions: two completed + but were not scoreable as integers, and one was incomplete. +- Short-schema arithmetic feasibility pilot, 7B, temperature 0.2, seed 101: + three previously used questions yielded one correct and two wrong scoreable + answers, all confidence 1.0. +- Ten new self-authored arithmetic pilot questions, same 7B settings: all ten + responses completed; nine were scoreable, with five correct and four wrong, + all confidence 1.0. One answer, `:83`, was excluded by the integer-only + scorer even though the reference was 83. For the nine scoreable responses, + pilot ECE was 4/9 (approximately 0.444). The ten calls took 166.1 seconds. + +Pilot machine: boothpy 0.5.8, Ollama 0.24.0, qwen2.5-coder:7b digest +`dae161e27b0e90dd1856c8bb3209201fd6736d8eb66298e75ed87571486f4364`, +Python 3.12.3, Linux x86_64. Pilot prompts, datasets and selection changed; +there is no controlled comparison between them. The models and settings were +selected after inspecting the pilots. The final run must remain separate. + +## Raw study responses + +Each line is a JSON object containing the question number, question, reference +answer and exact model response text. All replies had `done: true`; detailed +BOOTH status and grading appear in the saved records. + +```jsonl +{"index": 1, "question": "What is 31 plus 27?", "reference": 58, "raw": "{\n\"answer\": \"58\",\n\"confidence\": 1.0\n}"} +{"index": 2, "question": "What is 43 plus 29?", "reference": 72, "raw": "{\n\"answer\": \"62\",\n\"confidence\": 1\n}"} +{"index": 3, "question": "What is 58 plus 16?", "reference": 74, "raw": "{\n\"answer\": \"24\",\n\"confidence\": 1\n\n}"} +{"index": 4, "question": "What is 72 plus 19?", "reference": 91, "raw": "{\n\"answer\": \"131\",\n\"confidence\": 1\n }"} +{"index": 5, "question": "What is 36 plus 48?", "reference": 84, "raw": "{\n\"answer\": \"120\",\n\"confidence\": 1.0\n}"} +{"index": 6, "question": "What is 54 plus 37?", "reference": 91, "raw": "{\n\"answer\": \"81\",\n \"confidence\": 1\n}"} +{"index": 7, "question": "What is 67 plus 25?", "reference": 92, "raw": "{\n\"answer\": \"92\",\n\"confidence\": 1\n\n}"} +{"index": 8, "question": "What is 81 plus 34?", "reference": 115, "raw": "{\n\"answer\": \"2\",\n\"confidence\": 1\n}"} +{"index": 9, "question": "What is 93 minus 47?", "reference": 46, "raw": "{\n\"answer\": \"56\",\n \"confidence\": 1\n}"} +{"index": 10, "question": "What is 86 minus 38?", "reference": 48, "raw": "{\n\"answer\": \"58\",\n \"confidence\": 1\n\n}"} +{"index": 11, "question": "What is 120 minus 54?", "reference": 66, "raw": "{\n \"answer\": \"66\",\n \"confidence\": 1\n }\n"} +{"index": 12, "question": "What is 175 minus 89?", "reference": 86, "raw": "{\n\"answer\": \"36\",\n\"confidence\": 1\n}"} +{"index": 13, "question": "What is 141 minus 58?", "reference": 83, "raw": "{\n \"answer\": \"83\",\n \"confidence\": 1\n }"} +{"index": 14, "question": "What is 99 minus 46?", "reference": 53, "raw": "{\n\"answer\": \"53\",\n\"confidence\": 1\n}"} +{"index": 15, "question": "What is 162 minus 75?", "reference": 87, "raw": "{\n\"answer\": \":97\",\n\"confidence\": 1\n}"} +{"index": 16, "question": "What is 134 minus 68?", "reference": 66, "raw": "{\n \"answer\": \"66\",\n \"confidence\":1.0\n}"} +{"index": 17, "question": "What is 13 multiplied by 8?", "reference": 104, "raw": "{\n\"answer\": \"56\",\n\"confidence\": 1\n}\n"} +{"index": 18, "question": "What is 16 multiplied by 6?", "reference": 96, "raw": "{\n\"answer\": \"{\\\"result\\\":96,\\\"confidence\\\":1}\"\n\n,\n\n\"confidence\":1\n}"} +{"index": 19, "question": "What is 21 multiplied by 4?", "reference": 84, "raw": "{\n\"answer\": \"84\",\n \"confidence\": 1\n }\n"} +{"index": 20, "question": "What is 18 multiplied by 5?", "reference": 90, "raw": "{\n\"answer\": \"90\",\n\"confidence\": 1\n\n}"} +{"index": 21, "question": "What is 24 multiplied by 3?", "reference": 72, "raw": "{\n\"answer\": \"8\",\n\"confidence\": 1\n}"} +{"index": 22, "question": "What is 12 multiplied by 7?", "reference": 84, "raw": "{\n\"answer\": \"42\",\n \"confidence\": 1\n}"} +{"index": 23, "question": "What is 17 multiplied by 4?", "reference": 68, "raw": "{\n\"answer\": \"68\",\n \"confidence\": 1\n }\n"} +{"index": 24, "question": "What is 96 divided by 8?", "reference": 12, "raw": "{\n\"answer\": \"24\",\n\"confidence\": 1\n\n}"} +{"index": 25, "question": "What is 154 divided by 11?", "reference": 14, "raw": "{\n\"answer\": \"14\",\n\"confidence\": 1\n}"} +{"index": 26, "question": "What is 156 divided by 12?", "reference": 13, "raw": "{\n\"answer\": \"1.5\",\n\"confidence\": 1\n }"} +{"index": 27, "question": "What is 81 divided by 9?", "reference": 9, "raw": "{\n\"answer\": \"3\",\n\"confidence\": 1\n}"} +{"index": 28, "question": "What is 84 divided by 7?", "reference": 12, "raw": "{\n\"answer\": \"6\",\n \"confidence\": 1\n\n}"} +{"index": 29, "question": "What is 120 divided by 10?", "reference": 12, "raw": "{\n\"answer\": \"12\",\n\"confidence\": 1\n}"} +{"index": 30, "question": "What is 110 divided by 11?", "reference": 10, "raw": "{\n\"answer\": \"10\",\n\"confidence\": 1\n}"} +``` diff --git a/examples/rk042_confidence_calibration/run_example.py b/examples/rk042_confidence_calibration/run_example.py new file mode 100644 index 0000000..b621ec4 --- /dev/null +++ b/examples/rk042_confidence_calibration/run_example.py @@ -0,0 +1,189 @@ +"""Measure confidence against a fixed, self-authored arithmetic sample. + +Uses a custom short prompt and Ollama's JSON schema. BOOTH evaluates the cached +model response; BOOTH's default prompt is not sent to the model. Stdlib plus +boothpy 0.5.8; Ollama and the model must already be installed. Results are +saved under output/; no questions or model calls require internet access. +""" +import json +import platform +import re +import sys +import time +from collections import Counter +from datetime import datetime, timezone +from importlib import metadata +from pathlib import Path +from urllib.error import HTTPError +from urllib.request import Request, urlopen + +import booth + +HOST = 'http://127.0.0.1:11434' +MODEL = 'qwen2.5-coder:7b' +TEMPERATURE = 0.2 +SEED = 101 +TIMEOUT = 90 +THRESHOLDS = (0.5, 0.7, 0.9) +SCHEMA = {'type': 'object', 'properties': { + 'answer': {'type': 'string'}, + 'confidence': {'type': 'number', 'minimum': 0, 'maximum': 1}, +}, 'required': ['answer', 'confidence']} +# Fixed before the study; operands are distinct from our recorded pilots. +# Division problems have integer results. References are computed, not model-generated. +ITEMS = ( + ('add', 31, 27), ('add', 43, 29), ('add', 58, 16), ('add', 72, 19), + ('add', 36, 48), ('add', 54, 37), ('add', 67, 25), ('add', 81, 34), + ('subtract', 93, 47), ('subtract', 86, 38), ('subtract', 120, 54), + ('subtract', 175, 89), ('subtract', 141, 58), ('subtract', 99, 46), + ('subtract', 162, 75), ('subtract', 134, 68), + ('multiply', 13, 8), ('multiply', 16, 6), ('multiply', 21, 4), + ('multiply', 18, 5), ('multiply', 24, 3), ('multiply', 12, 7), + ('multiply', 17, 4), + ('divide', 96, 8), ('divide', 154, 11), ('divide', 156, 12), + ('divide', 81, 9), ('divide', 84, 7), ('divide', 120, 10), ('divide', 110, 11), +) + + +def make_question(kind, left, right): + if kind == 'add': + return f'What is {left} plus {right}?', left + right + if kind == 'subtract': + return f'What is {left} minus {right}?', left - right + if kind == 'multiply': + return f'What is {left} multiplied by {right}?', left * right + if kind == 'divide' and right and left % right == 0: + return f'What is {left} divided by {right}?', left // right + raise ValueError('Invalid arithmetic question') + + +def request_json(path, payload=None, timeout=10): + data = None if payload is None else json.dumps(payload).encode('utf-8') + request = Request(HOST + path, data=data, headers={'Content-Type': 'application/json'}) + try: + with urlopen(request, timeout=timeout) as response: + return json.load(response) + except HTTPError as exc: + detail = exc.read(4096).decode('utf-8', errors='replace') + raise RuntimeError(f'HTTP {exc.code}: {detail}') from exc + + +def score(answer, reference): + """Grade integer strings only; do not guess what malformed text meant.""" + if not isinstance(answer, str) or not re.fullmatch(r'[+-]?[0-9]+', answer.strip()): + return None + return int(answer.strip()) == reference + + +def run_one(index, question, reference): + prompt = (f'{question}\nReturn a JSON object with answer as an integer written ' + 'as a string and confidence as your estimated probability of correctness ' + 'between 0 and 1.') + payload = {'model': MODEL, 'prompt': prompt, 'stream': False, 'format': SCHEMA, + 'options': {'temperature': TEMPERATURE, 'seed': SEED}} + record = {'index': index, 'question': question, 'reference': reference, + 'raw': None, 'done': None, 'done_reason': None, 'eval_count': None, + 'status': None, 'method': None, 'call_failed': None, 'answer': None, + 'confidence': None, 'correct': None, 'error': None, 'exclusion': None} + started = time.monotonic() + try: + response = request_json('/api/generate', payload, timeout=TIMEOUT) + record.update(raw=response.get('response'), done=response.get('done'), + done_reason=response.get('done_reason'), eval_count=response.get('eval_count')) + if record['done'] is not True: + record['exclusion'] = 'response incomplete or completion unknown' + elif not isinstance(record['raw'], str): + record['exclusion'] = 'response text missing' + else: + # The model call has already happened. BOOTH parses and gates the cached response. + result = booth.check(lambda _: record['raw'], question, max_retries=0) + record.update(status=result.status, method=result.method, + call_failed=result.call_failed, answer=result.answer, + confidence=result.confidence, error=result.attempts[0].error) + if not result.attempts[0].parse_ok: + record['exclusion'] = 'BOOTH parse failure' + else: + record['correct'] = score(result.answer, reference) + if record['correct'] is None: + record['exclusion'] = 'answer is not an integer string' + elif result.confidence is None: + record['exclusion'] = 'confidence missing' + except TimeoutError as exc: + record.update(error=f'{type(exc).__name__}: {exc}', exclusion='client timeout') + except Exception as exc: + record.update(error=f'{type(exc).__name__}: {exc}', exclusion='request or processing error') + record['seconds'] = round(time.monotonic() - started, 3) + return record + + +def summarize(records, planned): + eligible = [r for r in records if r['exclusion'] is None and r['correct'] is not None] + print(f'Planned: {planned}; attempted: {len(records)}; eligible: {len(eligible)}; ' + f'excluded: {len(records) - len(eligible)}') + print('Exclusions:', dict(Counter(r['exclusion'] for r in records if r['exclusion']))) + print('BOOTH statuses:', dict(Counter(str(r['status']) for r in records if r['status'] is not None))) + print('Bins: range, count, mean confidence, observed accuracy') + ece = 0.0 + for i in range(10): + group = [r for r in eligible if min(int(r['confidence'] * 10), 9) == i] + if group: + mean = sum(r['confidence'] for r in group) / len(group) + accuracy = sum(r['correct'] for r in group) / len(group) + ece += len(group) / len(eligible) * abs(mean - accuracy) + print(f" [{i/10:.1f}, {(i+1)/10:.1f}{']' if i == 9 else ')'}: " + f'{len(group)}, {mean:.3f}, {accuracy:.3f}') + print(f'ECE: {ece:.3f}' if eligible else 'ECE: unavailable (no eligible answers)') + print('Confidence thresholds: threshold, retained, coverage of attempted, retained accuracy') + for threshold in THRESHOLDS: + retained = [r for r in eligible if r['confidence'] >= threshold] + coverage = len(retained) / len(records) if records else 0.0 + accuracy = (f"{sum(r['correct'] for r in retained) / len(retained):.3f}" + if retained else 'unavailable') + print(f' {threshold:.1f}: {len(retained)}, {coverage:.3f}, {accuracy}') + print('ECE uses scoreable responses only; coverage uses all attempted calls.') + print('Thresholds use confidence alone and do not measure BOOTH ambiguity handling.') + + +def main(): + version = metadata.version('boothpy') + if version != '0.5.8': + sys.exit(f'This study uses boothpy 0.5.8; found {version}.') + try: + server = request_json('/api/version')['version'] + model = next(m for m in request_json('/api/tags')['models'] if m.get('name') == MODEL) + except Exception as exc: + sys.exit(f'Cannot reach Ollama or find {MODEL}: {type(exc).__name__}: {exc}') + questions = [make_question(*item) for item in ITEMS] + config = {'boothpy': version, 'ollama': server, 'model': MODEL, + 'model_digest': model.get('digest'), 'python': platform.python_version(), + 'os': platform.platform(), 'temperature': TEMPERATURE, 'seed': SEED, + 'timeout_seconds': TIMEOUT, 'max_retries': 0, 'questions': len(questions), + 'dataset': 'fixed self-authored integer arithmetic', + 'prompt': 'custom short JSON-schema prompt; BOOTH checks cached response'} + output = Path(__file__).resolve().parent / 'output' / datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%S%fZ') + output.mkdir(parents=True, exist_ok=False) + (output / 'manifest.json').write_text(json.dumps({'config': config, 'questions': questions}, indent=2), encoding='utf-8') + print(json.dumps(config, indent=2), flush=True) + print(f'Raw records: {output / "records.jsonl"}', flush=True) + records = [] + started = time.monotonic() + with (output / 'records.jsonl').open('x', encoding='utf-8') as handle: + for index, (question, reference) in enumerate(questions, 1): + print(f'{index}/{len(questions)}: {question}', flush=True) + record = run_one(index, question, reference) + records.append(record) + handle.write(json.dumps(record, ensure_ascii=False) + '\n') + handle.flush() + print(json.dumps(record, ensure_ascii=False), flush=True) + if record['exclusion'] in ('client timeout', 'request or processing error', + 'response incomplete or completion unknown', 'response text missing'): + print('Stopping early after a request or completion failure.', flush=True) + break + print(f'\nElapsed seconds: {time.monotonic() - started:.1f}; model calls: {len(records)}') + summarize(records, len(questions)) + if len(records) != len(questions): + sys.exit('Study incomplete; retain the partial records and investigate before rerunning.') + + +if __name__ == '__main__': + main() From 64f778cd5f4a15abc4181d2515ab3910cc54bc8b Mon Sep 17 00:00:00 2001 From: rk042 Date: Sun, 15 Nov 2026 11:57:23 +0530 Subject: [PATCH 2/2] Clarify calibration setup and allow newer boothpy --- examples/rk042_confidence_calibration/RESULTS.md | 15 +++++++++++++-- .../rk042_confidence_calibration/run_example.py | 5 +++-- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/examples/rk042_confidence_calibration/RESULTS.md b/examples/rk042_confidence_calibration/RESULTS.md index f8d8b00..7eec36c 100644 --- a/examples/rk042_confidence_calibration/RESULTS.md +++ b/examples/rk042_confidence_calibration/RESULTS.md @@ -1,5 +1,10 @@ # Confidence calibration on local arithmetic +The model receives a custom short prompt plus an Ollama JSON schema; BOOTH +only parses and evaluates the saved model reply. These results describe the +model's stated confidence under constrained decoding, not BOOTH's default +prompt flow. + ## 30-question study: observed results Run 2026-11-15, UTC; 30 planned and attempted calls, 514.1 seconds. All 30 @@ -9,6 +14,10 @@ excluded under the pre-specified integer-only rule. Among scoreable responses, all reported confidence 1.0; accuracy was 12/27 = 0.444 and the ten-bin ECE was 15/27 = 0.556. These describe this sample, not general model calibration. +Final run environment: boothpy 0.5.8, Ollama 0.24.0, Python 3.12.3, +Linux 7.0.0-43-generic x86_64 (glibc 2.39), and qwen2.5-coder:7b with digest +`dae161e27b0e90dd1856c8bb3209201fd6736d8eb66298e75ed87571486f4364`. + ```text Planned: 30; attempted: 30; eligible: 27; excluded: 3 Exclusions: {'answer is not an integer string': 3} @@ -46,7 +55,9 @@ full records and manifest are in ignored `output/20261115T045922348251Z/`. ## Setup -- Python 3.9+, `boothpy==0.5.8`, Ollama, and the `qwen2.5-coder:7b` model. +- Python 3.9+, `boothpy`, Ollama, and the `qwen2.5-coder:7b` model. The + recorded run used boothpy 0.5.8. Other installed versions are recorded in + the manifest, and the script warns that results may differ. - Standard-library HTTP; no new BOOTH dependencies. The questions are self-authored integer arithmetic and their labels are computed locally. - Fixed 30-question study, temperature 0.2, seed 101, no retries. The 30 @@ -70,7 +81,7 @@ From this folder, after installing Ollama and the model: ```bash python3 -m venv .venv -.venv/bin/python -m pip install boothpy==0.5.8 +.venv/bin/python -m pip install boothpy .venv/bin/python run_example.py ``` diff --git a/examples/rk042_confidence_calibration/run_example.py b/examples/rk042_confidence_calibration/run_example.py index b621ec4..e9410f1 100644 --- a/examples/rk042_confidence_calibration/run_example.py +++ b/examples/rk042_confidence_calibration/run_example.py @@ -2,7 +2,7 @@ Uses a custom short prompt and Ollama's JSON schema. BOOTH evaluates the cached model response; BOOTH's default prompt is not sent to the model. Stdlib plus -boothpy 0.5.8; Ollama and the model must already be installed. Results are +boothpy; Ollama and the model must already be installed. Results are saved under output/; no questions or model calls require internet access. """ import json @@ -147,7 +147,8 @@ def summarize(records, planned): def main(): version = metadata.version('boothpy') if version != '0.5.8': - sys.exit(f'This study uses boothpy 0.5.8; found {version}.') + print(f'Warning: the documented study used boothpy 0.5.8; ' + f'this run uses {version}. Results may differ.', file=sys.stderr, flush=True) try: server = request_json('/api/version')['version'] model = next(m for m in request_json('/api/tags')['models'] if m.get('name') == MODEL)