diff --git a/examples/Vedantgitbot_latency_accuracy/RESULTS_gemma3-12b.md b/examples/Vedantgitbot_latency_accuracy/RESULTS_gemma3-12b.md new file mode 100644 index 0000000..9e7729c --- /dev/null +++ b/examples/Vedantgitbot_latency_accuracy/RESULTS_gemma3-12b.md @@ -0,0 +1,126 @@ +**Gemma 3 12B: local four-arm run, 50 questions** + +I ran the existing four-arm benchmark with `gemma3:12b` (`Q4_K_M`) using local Ollama on an M3 MacBook Air with 24 GiB RAM. Codex handled execution and the checks described below. + +The source was unchanged upstream `main` at [a51f30c173776bfb6a47781a1e52b03f39b258a2](https://github.com/Vedantgitbot/booth/commit/a51f30c173776bfb6a47781a1e52b03f39b258a2). I used an isolated Python environment with `boothpy 0.5.8` installed from that checkout, and verified that the runner imported that copy. + +I followed the [updated run instructions](https://github.com/Vedantgitbot/booth/issues/33#issuecomment-6710510420): smoke first, then four arms and one repetition. The ten-question smoke passed but took 25.8 minutes excluding setup, projecting about 4.3 hours for 100 questions, so I chose the permitted `--n 50` before the full run. The model tag and digest were fixed before inference and stayed unchanged. + +This repeats the protocol on a second model with a smaller sample. It does not reproduce the published Qwen run's exact 100-question measurements. All four Gemma arms used the same 50 questions: 15 TriviaQA, 15 TruthfulQA and 20 GSM8K. Their IDs are the ordered per-dataset prefixes of the published list; no dataset fallback occurred. + +Commands below use `python` to denote the isolated environment's interpreter; `RUN_DIR` was a fresh absolute evidence directory. Neither command used `--resume`. + +```bash +python examples/Vedantgitbot_latency_accuracy/run_example.py --host http://127.0.0.1:11434 --model gemma3:12b --smoke --out "$RUN_DIR/smoke" +python examples/Vedantgitbot_latency_accuracy/run_example.py --host http://127.0.0.1:11434 --model gemma3:12b --n 50 --reps 1 --seed 42 --temperature 0 --num-predict 1024 --timeout 300 --out "$RUN_DIR/full" +``` + +Codex checked all 200 full-run records for complete, unique `(rep, qid, arm)` coverage, reconstructed the sampled inputs from the retained dataset cache, checked the randomized execution order, independently recomputed the grades, and reconciled the summary. Source, imports and model identity checks passed. All questions, including rejections, remain in the denominator. Raw records, manifests, cached source rows and command logs are retained locally. + +**Unedited generated `summary.md` follows.** + + +## Experiment configuration + +| Configuration | Value | +|---|---| +| BOOTH version | boothpy 0.5.8 (default max_retries=1, threshold=0.7) | +| Python / OS | 3.12.13 / macOS-27.0-arm64-arm-64bit | +| Provider / Model | Ollama 0.40.15 / `gemma3:12b` | +| Model digest | `f4031aab637d1ffa37b42570452ae0e4fad0314754d17ded67322e4b95836f8a` | +| Temperature / Seed | 0.0 / 42 (num_predict=1024, timeout=300s per call) | +| Dataset versions | Hugging Face datasets-server, current `main` at run time (2026-11-12); see Datasets below | +| Questions / Repetitions | 50 / 1 | +| Hardware | CPU: Apple M3 (8 threads); RAM: 24.0 GiB; GPU: Apple Silicon integrated GPU (unified memory); Ollama placement: 80% GPU / 20% CPU | +| Setup / warm-up (excluded from results) | setup 18.7s incl. dataset fetch + model load; warm-up call 0.7s | +| Benchmark wall-clock this session | 7484s (all arms and reps, includes interleaving; excludes setup) | + +## Arms + +- **Plain model**: question + "Final answer:" instruction; the model may reason freely. +- **BOOTH prompt, no gate**: the question + BOOTH's exact JSON suffix, one call, no confidence check and no retries. Isolates the effect of the prompt from the effect of the gate. +- **BOOTH (default)**: `booth.check()` with default settings. +- **BOOTH + reasoning room**: `booth.check()` with BOOTH's suffix swapped (via `call_fn`) for one that lets the model reason before the JSON; same schema. + +## Benchmark comparison + +Pooled over 1 repetition(s) x 50 questions = 50 graded question-runs per arm. Accuracy denominator is ALL question-runs: rejected (AMBIGUOUS/UNCERTAIN), call-failed and unanswerable runs count as not correct. + +| Metric | Plain model | BOOTH prompt, no gate | BOOTH (default) | BOOTH + reasoning room | +|---|---|---|---|---| +| Accuracy (%) | 84.0 | 76.0 | 60.0 | 76.0 | +| Mean latency (s) | 23.65 | 37.71 | 36.01 | 52.30 | +| Median latency (s) | 4.60 | 29.94 | 30.43 | 52.47 | +| Total runtime (s, all reps) | 1182.7 | 1885.5 | 1800.5 | 2615.1 | +| Model calls (all reps) | 50 | 50 | 50 | 50 | +| Rejected (AMBIGUOUS + UNCERTAIN) | 0 | 0 | 11 | 4 | +| AMBIGUOUS | 0 | 0 | 11 | 4 | +| UNCERTAIN | 0 | 0 | 0 | 0 | +| Call failures (failed HTTP calls) | 0 | 0 | 0 | 0 | +| Questions with no response (all calls failed) | 0 | 0 | 0 | 0 | +| Correct / Incorrect / No-answer | 42 / 8 / 0 | 38 / 12 / 0 | 30 / 9 / 0 | 38 / 8 / 0 | + +Derived (not part of the required table): + +| Metric | Plain model | BOOTH prompt, no gate | BOOTH (default) | BOOTH + reasoning room | +|---|---|---|---|---| +| Model calls per question | 1.00 | 1.00 | 1.00 | 1.00 | +| Mean latency vs plain (x) | 1.00 | 1.59 | 1.52 | 2.21 | +| Output tokens (all reps) | 4568 | 6866 | 6866 | 9974 | +| Accuracy among answered, non-rejected (%) | 84.0 | 76.0 | 76.9 | 82.6 | +| Accuracy if rejected answers were still graded (%) | 84.0 | 76.0 | 76.0 | 80.0 | + +## Per-repetition results + +| Rep | Arm | Accuracy (%) | Mean lat (s) | Median lat (s) | Runtime (s) | Calls | AMBIGUOUS | UNCERTAIN | Call failures | +|---|---|---|---|---|---|---|---|---|---| +| 1 | Plain model | 84.0 | 23.65 | 4.60 | 1182.7 | 50 | 0 | 0 | 0 | +| 1 | BOOTH prompt, no gate | 76.0 | 37.71 | 29.94 | 1885.5 | 50 | 0 | 0 | 0 | +| 1 | BOOTH (default) | 60.0 | 36.01 | 30.43 | 1800.5 | 50 | 11 | 0 | 0 | +| 1 | BOOTH + reasoning room | 76.0 | 52.30 | 52.47 | 2615.1 | 50 | 4 | 0 | 0 | + +## Accuracy by dataset (%, pooled over reps) + +| Dataset | Questions | Plain model | BOOTH prompt, no gate | BOOTH (default) | BOOTH + reasoning room | +|---|---|---|---|---| +| triviaqa | 15 | 60.0 | 60.0 | 53.3 | 60.0 | +| truthfulqa | 15 | 86.7 | 80.0 | 33.3 | 73.3 | +| gsm8k | 20 | 100.0 | 85.0 | 85.0 | 90.0 | + +## Mean output tokens per question, by dataset + +| Dataset | Plain model | BOOTH prompt, no gate | BOOTH (default) | BOOTH + reasoning room | +|---|---|---|---|---| +| triviaqa | 7 | 45 | 45 | 94 | +| truthfulqa | 5 | 84 | 84 | 234 | +| gsm8k | 220 | 247 | 247 | 253 | + +## Datasets + +| Name | Dataset ID | Config / split | Split size | Sampled | Licence | +|---|---|---|---|---|---| +| triviaqa | `mandarjoshi/trivia_qa` | rc.nocontext / validation | 17944 | 15 | Apache-2.0 | +| truthfulqa | `EleutherAI/truthful_qa_mc` | multiple_choice / validation | 684 | 15 | Apache-2.0 | +| gsm8k | `openai/gsm8k` | main / test | 1319 | 20 | MIT | + +Sampling: `random.Random(f"{seed}:{dataset}").sample(...)` with seed 42. Question IDs and reference answers are in `output/manifest.json`. + +
Sampled question IDs + +**triviaqa**: sfq_7483, jp_925, qg_2508, bt_2652, qw_9695, odql_9445, sfq_22425, jp_3307, bt_2624, sfq_380, sfq_23048, qw_6473, wh_326, sfq_4851, odql_389 + +**truthfulqa**: truthfulqa_mc-validation-600, truthfulqa_mc-validation-267, truthfulqa_mc-validation-248, truthfulqa_mc-validation-179, truthfulqa_mc-validation-323, truthfulqa_mc-validation-163, truthfulqa_mc-validation-51, truthfulqa_mc-validation-318, truthfulqa_mc-validation-632, truthfulqa_mc-validation-151, truthfulqa_mc-validation-236, truthfulqa_mc-validation-280, truthfulqa_mc-validation-641, truthfulqa_mc-validation-312, truthfulqa_mc-validation-21 + +**gsm8k**: gsm8k-test-948, gsm8k-test-198, gsm8k-test-248, gsm8k-test-1137, gsm8k-test-1071, gsm8k-test-401, gsm8k-test-3, gsm8k-test-467, gsm8k-test-490, gsm8k-test-785, gsm8k-test-1156, gsm8k-test-135, gsm8k-test-888, gsm8k-test-417, gsm8k-test-359, gsm8k-test-396, gsm8k-test-374, gsm8k-test-827, gsm8k-test-1277, gsm8k-test-681 + +
+ + +**Interpretation, separate from the generated summary** + +- **Prompt effect:** plain scored 20/20 on GSM8K; JSON without a gate scored 17/20. Overall scores were 42/50 and 38/50. +- **Checking:** JSON without a gate and default BOOTH produced identical predictions for all 50 questions, with no retries. Default BOOTH rejected 11 answers as AMBIGUOUS, including eight that matched the reference, reducing overall accuracy to 30/50. Ten rejections were TruthfulQA; math stayed at 17/20. +- **Reasoning-first:** delivered math accuracy rose to 18/20 and overall to 38/50. It corrected two math misses but newly rejected one correct math answer, for a net recovery of one question. Mean latency was 52.30s versus 36.01s default (1.45×) and 23.65s plain (2.21×). Every arm used 1.00 model call/question; there were no failed HTTP calls or UNCERTAIN outcomes. +- **Limit:** one model, one seed, one repetition, and twenty math questions. GSM8K mean output tokens were 220 plain versus 247 JSON without a gate, so this run does not show the large token-count collapse reported for Qwen. The runner does not retain raw replies, leaving the reasoning-suppression mechanism unresolved here. The published Qwen aggregate uses 100 questions, so these percentages should not be used as a matched-sample model ranking. + +Possible follow-up, **not run**: matched numeric questions that change one unit or comparison, such as percent versus percentage points, while retaining raw replies and rejection reasons. diff --git a/examples/Vedantgitbot_latency_accuracy/results_gemma3-12b.jsonl b/examples/Vedantgitbot_latency_accuracy/results_gemma3-12b.jsonl new file mode 100644 index 0000000..0b0814f --- /dev/null +++ b/examples/Vedantgitbot_latency_accuracy/results_gemma3-12b.jsonl @@ -0,0 +1,200 @@ +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Ken Follett", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 7.535947625001427, "model_calls": 1, "failed_calls": 0, "out_tokens": 41, "rep": 1, "qid": "sfq_23048", "dataset": "triviaqa", "arm": "booth_def", "position": 0, "reference": "William Boyd"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Ken Follett", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 10.704227874986827, "model_calls": 1, "failed_calls": 0, "out_tokens": 66, "rep": 1, "qid": "sfq_23048", "dataset": "triviaqa", "arm": "booth_reason", "position": 1, "reference": "William Boyd"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "Ken Follett", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 8.271239208988845, "model_calls": 1, "failed_calls": 0, "out_tokens": 41, "rep": 1, "qid": "sfq_23048", "dataset": "triviaqa", "arm": "json_nogate", "position": 2, "reference": "William Boyd"} +{"status": "ANSWERED", "confidence": null, "predicted": "Kingsley Amis", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 1.7037002500146627, "model_calls": 1, "failed_calls": 0, "out_tokens": 7, "rep": 1, "qid": "sfq_23048", "dataset": "triviaqa", "arm": "plain", "position": 3, "reference": "William Boyd"} +{"status": "ANSWERED", "confidence": null, "predicted": "D", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 2.368260666029528, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "truthfulqa_mc-validation-318", "dataset": "truthfulqa", "arm": "plain", "position": 0, "reference": "D"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "D", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 9.857936916989274, "model_calls": 1, "failed_calls": 0, "out_tokens": 39, "rep": 1, "qid": "truthfulqa_mc-validation-318", "dataset": "truthfulqa", "arm": "json_nogate", "position": 1, "reference": "D"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "D", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 4.602936499984935, "model_calls": 1, "failed_calls": 0, "out_tokens": 39, "rep": 1, "qid": "truthfulqa_mc-validation-318", "dataset": "truthfulqa", "arm": "booth_def", "position": 2, "reference": "D"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "D", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 23.388520082982723, "model_calls": 1, "failed_calls": 0, "out_tokens": 144, "rep": 1, "qid": "truthfulqa_mc-validation-318", "dataset": "truthfulqa", "arm": "booth_reason", "position": 3, "reference": "D"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "The Time Machine", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 15.689113916014321, "model_calls": 1, "failed_calls": 0, "out_tokens": 41, "rep": 1, "qid": "sfq_4851", "dataset": "triviaqa", "arm": "json_nogate", "position": 0, "reference": "THE TIME MACHINE"} +{"status": "ANSWERED", "confidence": null, "predicted": "The Time Machine", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 10.44318129203748, "model_calls": 1, "failed_calls": 0, "out_tokens": 7, "rep": 1, "qid": "sfq_4851", "dataset": "triviaqa", "arm": "plain", "position": 1, "reference": "THE TIME MACHINE"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "The Time Machine", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 33.07335504202638, "model_calls": 1, "failed_calls": 0, "out_tokens": 41, "rep": 1, "qid": "sfq_4851", "dataset": "triviaqa", "arm": "booth_def", "position": 2, "reference": "THE TIME MACHINE"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "The Time Machine", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 27.455093750031665, "model_calls": 1, "failed_calls": 0, "out_tokens": 77, "rep": 1, "qid": "sfq_4851", "dataset": "triviaqa", "arm": "booth_reason", "position": 3, "reference": "THE TIME MACHINE"} +{"status": "ANSWERED", "confidence": 0.6, "predicted": "D", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 27.90328099997714, "model_calls": 1, "failed_calls": 0, "out_tokens": 88, "rep": 1, "qid": "truthfulqa_mc-validation-51", "dataset": "truthfulqa", "arm": "json_nogate", "position": 0, "reference": "D"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "D", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 49.18359920795774, "model_calls": 1, "failed_calls": 0, "out_tokens": 177, "rep": 1, "qid": "truthfulqa_mc-validation-51", "dataset": "truthfulqa", "arm": "booth_reason", "position": 1, "reference": "D"} +{"status": "ANSWERED", "confidence": null, "predicted": "D", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 3.2715278749819845, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "truthfulqa_mc-validation-51", "dataset": "truthfulqa", "arm": "plain", "position": 2, "reference": "D"} +{"status": "AMBIGUOUS", "confidence": 0.6, "predicted": "D", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "ambiguous", "latency_s": 28.201474583009258, "model_calls": 1, "failed_calls": 0, "out_tokens": 88, "rep": 1, "qid": "truthfulqa_mc-validation-51", "dataset": "truthfulqa", "arm": "booth_def", "position": 3, "reference": "D"} +{"status": "ANSWERED", "confidence": 0.75, "predicted": "A", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 29.410120957996696, "model_calls": 1, "failed_calls": 0, "out_tokens": 88, "rep": 1, "qid": "truthfulqa_mc-validation-179", "dataset": "truthfulqa", "arm": "json_nogate", "position": 0, "reference": "A"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "A", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 51.58749225002248, "model_calls": 1, "failed_calls": 0, "out_tokens": 171, "rep": 1, "qid": "truthfulqa_mc-validation-179", "dataset": "truthfulqa", "arm": "booth_reason", "position": 1, "reference": "A"} +{"status": "AMBIGUOUS", "confidence": 0.75, "predicted": "A", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "ambiguous", "latency_s": 28.98079283401603, "model_calls": 1, "failed_calls": 0, "out_tokens": 88, "rep": 1, "qid": "truthfulqa_mc-validation-179", "dataset": "truthfulqa", "arm": "booth_def", "position": 2, "reference": "A"} +{"status": "ANSWERED", "confidence": null, "predicted": "A", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 3.72038562502712, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "truthfulqa_mc-validation-179", "dataset": "truthfulqa", "arm": "plain", "position": 3, "reference": "A"} +{"status": "ANSWERED", "confidence": 1.0, "predicted": "10", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 46.50379575003171, "model_calls": 1, "failed_calls": 0, "out_tokens": 154, "rep": 1, "qid": "gsm8k-test-888", "dataset": "gsm8k", "arm": "json_nogate", "position": 0, "reference": "10"} +{"status": "ACCEPTED", "confidence": 1.0, "predicted": "10", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 38.01663345802808, "model_calls": 1, "failed_calls": 0, "out_tokens": 154, "rep": 1, "qid": "gsm8k-test-888", "dataset": "gsm8k", "arm": "booth_def", "position": 1, "reference": "10"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "10", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 44.17910779203521, "model_calls": 1, "failed_calls": 0, "out_tokens": 156, "rep": 1, "qid": "gsm8k-test-888", "dataset": "gsm8k", "arm": "booth_reason", "position": 2, "reference": "10"} +{"status": "ANSWERED", "confidence": null, "predicted": "10", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 33.042170500033535, "model_calls": 1, "failed_calls": 0, "out_tokens": 123, "rep": 1, "qid": "gsm8k-test-888", "dataset": "gsm8k", "arm": "plain", "position": 3, "reference": "10"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "120", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 64.43874687497737, "model_calls": 1, "failed_calls": 0, "out_tokens": 214, "rep": 1, "qid": "gsm8k-test-1156", "dataset": "gsm8k", "arm": "json_nogate", "position": 0, "reference": "120"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "120", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 53.01254233298823, "model_calls": 1, "failed_calls": 0, "out_tokens": 214, "rep": 1, "qid": "gsm8k-test-1156", "dataset": "gsm8k", "arm": "booth_def", "position": 1, "reference": "120"} +{"status": "ANSWERED", "confidence": null, "predicted": "120", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 50.50855474994751, "model_calls": 1, "failed_calls": 0, "out_tokens": 194, "rep": 1, "qid": "gsm8k-test-1156", "dataset": "gsm8k", "arm": "plain", "position": 2, "reference": "120"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "120", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 69.26989549997961, "model_calls": 1, "failed_calls": 0, "out_tokens": 228, "rep": 1, "qid": "gsm8k-test-1156", "dataset": "gsm8k", "arm": "booth_reason", "position": 3, "reference": "120"} +{"status": "ANSWERED", "confidence": null, "predicted": "320", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 69.38788233295782, "model_calls": 1, "failed_calls": 0, "out_tokens": 264, "rep": 1, "qid": "gsm8k-test-198", "dataset": "gsm8k", "arm": "plain", "position": 0, "reference": "320"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "320", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 78.43884683400393, "model_calls": 1, "failed_calls": 0, "out_tokens": 274, "rep": 1, "qid": "gsm8k-test-198", "dataset": "gsm8k", "arm": "booth_reason", "position": 1, "reference": "320"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "320", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 62.72793500003172, "model_calls": 1, "failed_calls": 0, "out_tokens": 250, "rep": 1, "qid": "gsm8k-test-198", "dataset": "gsm8k", "arm": "booth_def", "position": 2, "reference": "320"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "320", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 51.57806041603908, "model_calls": 1, "failed_calls": 0, "out_tokens": 250, "rep": 1, "qid": "gsm8k-test-198", "dataset": "gsm8k", "arm": "json_nogate", "position": 3, "reference": "320"} +{"status": "ANSWERED", "confidence": 0.99, "predicted": "A Series of Unfortunate Events", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 14.835514208010864, "model_calls": 1, "failed_calls": 0, "out_tokens": 44, "rep": 1, "qid": "qw_9695", "dataset": "triviaqa", "arm": "json_nogate", "position": 0, "reference": "A Series of Unfortunate Events"} +{"status": "ANSWERED", "confidence": null, "predicted": "A Series of Unfortunate Events", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 3.464846166956704, "model_calls": 1, "failed_calls": 0, "out_tokens": 10, "rep": 1, "qid": "qw_9695", "dataset": "triviaqa", "arm": "plain", "position": 1, "reference": "A Series of Unfortunate Events"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "A Series of Unfortunate Events", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 27.433488499955274, "model_calls": 1, "failed_calls": 0, "out_tokens": 99, "rep": 1, "qid": "qw_9695", "dataset": "triviaqa", "arm": "booth_reason", "position": 2, "reference": "A Series of Unfortunate Events"} +{"status": "ACCEPTED", "confidence": 0.99, "predicted": "A Series of Unfortunate Events", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 15.706247791997157, "model_calls": 1, "failed_calls": 0, "out_tokens": 44, "rep": 1, "qid": "qw_9695", "dataset": "triviaqa", "arm": "booth_def", "position": 3, "reference": "A Series of Unfortunate Events"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "Steffi Graf", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 15.425376457977109, "model_calls": 1, "failed_calls": 0, "out_tokens": 41, "rep": 1, "qid": "odql_389", "dataset": "triviaqa", "arm": "json_nogate", "position": 0, "reference": "AMELIE MAURESMO"} +{"status": "ANSWERED", "confidence": null, "predicted": "Steffi Graf", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 3.3605032920022495, "model_calls": 1, "failed_calls": 0, "out_tokens": 7, "rep": 1, "qid": "odql_389", "dataset": "triviaqa", "arm": "plain", "position": 1, "reference": "AMELIE MAURESMO"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Virginia Wade", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 52.97722929198062, "model_calls": 1, "failed_calls": 0, "out_tokens": 210, "rep": 1, "qid": "odql_389", "dataset": "triviaqa", "arm": "booth_reason", "position": 2, "reference": "AMELIE MAURESMO"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Steffi Graf", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 14.323808665969409, "model_calls": 1, "failed_calls": 0, "out_tokens": 41, "rep": 1, "qid": "odql_389", "dataset": "triviaqa", "arm": "booth_def", "position": 3, "reference": "AMELIE MAURESMO"} +{"status": "AMBIGUOUS", "confidence": 0.7, "predicted": "68", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "ambiguous", "latency_s": 113.72615520795807, "model_calls": 1, "failed_calls": 0, "out_tokens": 536, "rep": 1, "qid": "gsm8k-test-1137", "dataset": "gsm8k", "arm": "booth_reason", "position": 0, "reference": "68"} +{"status": "ANSWERED", "confidence": null, "predicted": "68", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 76.19962479104288, "model_calls": 1, "failed_calls": 0, "out_tokens": 367, "rep": 1, "qid": "gsm8k-test-1137", "dataset": "gsm8k", "arm": "plain", "position": 1, "reference": "68"} +{"status": "ANSWERED", "confidence": 0.9, "predicted": "68", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 86.31743204098893, "model_calls": 1, "failed_calls": 0, "out_tokens": 410, "rep": 1, "qid": "gsm8k-test-1137", "dataset": "gsm8k", "arm": "json_nogate", "position": 2, "reference": "68"} +{"status": "ACCEPTED", "confidence": 0.9, "predicted": "68", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 87.68170833395561, "model_calls": 1, "failed_calls": 0, "out_tokens": 410, "rep": 1, "qid": "gsm8k-test-1137", "dataset": "gsm8k", "arm": "booth_def", "position": 3, "reference": "68"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Genus", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 13.487293915997725, "model_calls": 1, "failed_calls": 0, "out_tokens": 39, "rep": 1, "qid": "jp_3307", "dataset": "triviaqa", "arm": "booth_def", "position": 0, "reference": "Genus"} +{"status": "ANSWERED", "confidence": null, "predicted": "Genus", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 2.501840666984208, "model_calls": 1, "failed_calls": 0, "out_tokens": 6, "rep": 1, "qid": "jp_3307", "dataset": "triviaqa", "arm": "plain", "position": 1, "reference": "Genus"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Genus", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 21.839751875028014, "model_calls": 1, "failed_calls": 0, "out_tokens": 75, "rep": 1, "qid": "jp_3307", "dataset": "triviaqa", "arm": "booth_reason", "position": 2, "reference": "Genus"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "Genus", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 14.296426332963165, "model_calls": 1, "failed_calls": 0, "out_tokens": 39, "rep": 1, "qid": "jp_3307", "dataset": "triviaqa", "arm": "json_nogate", "position": 3, "reference": "Genus"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "Christopher Robin", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 14.807933375006542, "model_calls": 1, "failed_calls": 0, "out_tokens": 40, "rep": 1, "qid": "qg_2508", "dataset": "triviaqa", "arm": "json_nogate", "position": 0, "reference": "Christopher Robin"} +{"status": "ANSWERED", "confidence": null, "predicted": "Christopher Robin", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 3.2525001249741763, "model_calls": 1, "failed_calls": 0, "out_tokens": 6, "rep": 1, "qid": "qg_2508", "dataset": "triviaqa", "arm": "plain", "position": 1, "reference": "Christopher Robin"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Christopher Robin", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 26.607047875004355, "model_calls": 1, "failed_calls": 0, "out_tokens": 86, "rep": 1, "qid": "qg_2508", "dataset": "triviaqa", "arm": "booth_reason", "position": 2, "reference": "Christopher Robin"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Christopher Robin", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 14.863546667038463, "model_calls": 1, "failed_calls": 0, "out_tokens": 40, "rep": 1, "qid": "qg_2508", "dataset": "triviaqa", "arm": "booth_def", "position": 3, "reference": "Christopher Robin"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "3140", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 82.72051383298822, "model_calls": 1, "failed_calls": 0, "out_tokens": 338, "rep": 1, "qid": "gsm8k-test-417", "dataset": "gsm8k", "arm": "booth_reason", "position": 0, "reference": "3140"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "3140", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 72.26607670797966, "model_calls": 1, "failed_calls": 0, "out_tokens": 295, "rep": 1, "qid": "gsm8k-test-417", "dataset": "gsm8k", "arm": "json_nogate", "position": 1, "reference": "3140"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "3140", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 68.34236670902465, "model_calls": 1, "failed_calls": 0, "out_tokens": 295, "rep": 1, "qid": "gsm8k-test-417", "dataset": "gsm8k", "arm": "booth_def", "position": 2, "reference": "3140"} +{"status": "ANSWERED", "confidence": null, "predicted": "3140", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 86.52716370800044, "model_calls": 1, "failed_calls": 0, "out_tokens": 385, "rep": 1, "qid": "gsm8k-test-417", "dataset": "gsm8k", "arm": "plain", "position": 3, "reference": "3140"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "70", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 60.150667915993836, "model_calls": 1, "failed_calls": 0, "out_tokens": 229, "rep": 1, "qid": "gsm8k-test-135", "dataset": "gsm8k", "arm": "json_nogate", "position": 0, "reference": "40"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "70", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 60.21641545899911, "model_calls": 1, "failed_calls": 0, "out_tokens": 220, "rep": 1, "qid": "gsm8k-test-135", "dataset": "gsm8k", "arm": "booth_reason", "position": 1, "reference": "40"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "70", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 57.33842145802919, "model_calls": 1, "failed_calls": 0, "out_tokens": 229, "rep": 1, "qid": "gsm8k-test-135", "dataset": "gsm8k", "arm": "booth_def", "position": 2, "reference": "40"} +{"status": "ANSWERED", "confidence": null, "predicted": "40", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 49.76526983396616, "model_calls": 1, "failed_calls": 0, "out_tokens": 212, "rep": 1, "qid": "gsm8k-test-135", "dataset": "gsm8k", "arm": "plain", "position": 3, "reference": "40"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "15.00", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 97.563248209015, "model_calls": 1, "failed_calls": 0, "out_tokens": 306, "rep": 1, "qid": "gsm8k-test-948", "dataset": "gsm8k", "arm": "json_nogate", "position": 0, "reference": "15"} +{"status": "ANSWERED", "confidence": null, "predicted": "15", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 56.60364595800638, "model_calls": 1, "failed_calls": 0, "out_tokens": 317, "rep": 1, "qid": "gsm8k-test-948", "dataset": "gsm8k", "arm": "plain", "position": 1, "reference": "15"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "15.00", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 74.04441208404023, "model_calls": 1, "failed_calls": 0, "out_tokens": 348, "rep": 1, "qid": "gsm8k-test-948", "dataset": "gsm8k", "arm": "booth_reason", "position": 2, "reference": "15"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "15.00", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 64.60418637504335, "model_calls": 1, "failed_calls": 0, "out_tokens": 306, "rep": 1, "qid": "gsm8k-test-948", "dataset": "gsm8k", "arm": "booth_def", "position": 3, "reference": "15"} +{"status": "ANSWERED", "confidence": null, "predicted": "C", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 3.0099418750032783, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "truthfulqa_mc-validation-163", "dataset": "truthfulqa", "arm": "plain", "position": 0, "reference": "C"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "C", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 26.68397966597695, "model_calls": 1, "failed_calls": 0, "out_tokens": 117, "rep": 1, "qid": "truthfulqa_mc-validation-163", "dataset": "truthfulqa", "arm": "booth_reason", "position": 1, "reference": "C"} +{"status": "ANSWERED", "confidence": 1.0, "predicted": "C", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 13.489439000026323, "model_calls": 1, "failed_calls": 0, "out_tokens": 38, "rep": 1, "qid": "truthfulqa_mc-validation-163", "dataset": "truthfulqa", "arm": "json_nogate", "position": 2, "reference": "C"} +{"status": "ACCEPTED", "confidence": 1.0, "predicted": "C", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 7.8437722499947995, "model_calls": 1, "failed_calls": 0, "out_tokens": 38, "rep": 1, "qid": "truthfulqa_mc-validation-163", "dataset": "truthfulqa", "arm": "booth_def", "position": 3, "reference": "C"} +{"status": "ANSWERED", "confidence": null, "predicted": "251", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 40.838720833999105, "model_calls": 1, "failed_calls": 0, "out_tokens": 189, "rep": 1, "qid": "gsm8k-test-1071", "dataset": "gsm8k", "arm": "plain", "position": 0, "reference": "251"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "251", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 51.96553316700738, "model_calls": 1, "failed_calls": 0, "out_tokens": 232, "rep": 1, "qid": "gsm8k-test-1071", "dataset": "gsm8k", "arm": "booth_reason", "position": 1, "reference": "251"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "251", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 47.44676108303247, "model_calls": 1, "failed_calls": 0, "out_tokens": 222, "rep": 1, "qid": "gsm8k-test-1071", "dataset": "gsm8k", "arm": "json_nogate", "position": 2, "reference": "251"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "251", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 39.21751675003907, "model_calls": 1, "failed_calls": 0, "out_tokens": 222, "rep": 1, "qid": "gsm8k-test-1071", "dataset": "gsm8k", "arm": "booth_def", "position": 3, "reference": "251"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "Virginia Roberts", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 12.47109108301811, "model_calls": 1, "failed_calls": 0, "out_tokens": 40, "rep": 1, "qid": "qw_6473", "dataset": "triviaqa", "arm": "json_nogate", "position": 0, "reference": "Tareq and Michaele Salah"} +{"status": "ANSWERED", "confidence": null, "predicted": "Tareq Salahi", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 3.096548834000714, "model_calls": 1, "failed_calls": 0, "out_tokens": 8, "rep": 1, "qid": "qw_6473", "dataset": "triviaqa", "arm": "plain", "position": 1, "reference": "Tareq and Michaele Salah"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Rick Boucher", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 20.50048687501112, "model_calls": 1, "failed_calls": 0, "out_tokens": 78, "rep": 1, "qid": "qw_6473", "dataset": "triviaqa", "arm": "booth_reason", "position": 2, "reference": "Tareq and Michaele Salah"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Virginia Roberts", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 14.713809125008993, "model_calls": 1, "failed_calls": 0, "out_tokens": 40, "rep": 1, "qid": "qw_6473", "dataset": "triviaqa", "arm": "booth_def", "position": 3, "reference": "Tareq and Michaele Salah"} +{"status": "AMBIGUOUS", "confidence": 0.85, "predicted": "D", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "ambiguous", "latency_s": 30.975575457967352, "model_calls": 1, "failed_calls": 0, "out_tokens": 105, "rep": 1, "qid": "truthfulqa_mc-validation-600", "dataset": "truthfulqa", "arm": "booth_def", "position": 0, "reference": "A"} +{"status": "ANSWERED", "confidence": 0.85, "predicted": "D", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 21.103928541997448, "model_calls": 1, "failed_calls": 0, "out_tokens": 105, "rep": 1, "qid": "truthfulqa_mc-validation-600", "dataset": "truthfulqa", "arm": "json_nogate", "position": 1, "reference": "A"} +{"status": "ANSWERED", "confidence": null, "predicted": "D", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 4.06299983302597, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "truthfulqa_mc-validation-600", "dataset": "truthfulqa", "arm": "plain", "position": 2, "reference": "A"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "D", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 62.97621029103175, "model_calls": 1, "failed_calls": 0, "out_tokens": 290, "rep": 1, "qid": "truthfulqa_mc-validation-600", "dataset": "truthfulqa", "arm": "booth_reason", "position": 3, "reference": "A"} +{"status": "ACCEPTED", "confidence": 0.9, "predicted": "C", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 56.04321308399085, "model_calls": 1, "failed_calls": 0, "out_tokens": 210, "rep": 1, "qid": "truthfulqa_mc-validation-21", "dataset": "truthfulqa", "arm": "booth_reason", "position": 0, "reference": "C"} +{"status": "ANSWERED", "confidence": null, "predicted": "C", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 4.777844666969031, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "truthfulqa_mc-validation-21", "dataset": "truthfulqa", "arm": "plain", "position": 1, "reference": "C"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "C", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 19.141034207947087, "model_calls": 1, "failed_calls": 0, "out_tokens": 39, "rep": 1, "qid": "truthfulqa_mc-validation-21", "dataset": "truthfulqa", "arm": "json_nogate", "position": 2, "reference": "C"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "C", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 10.624562208016869, "model_calls": 1, "failed_calls": 0, "out_tokens": 39, "rep": 1, "qid": "truthfulqa_mc-validation-21", "dataset": "truthfulqa", "arm": "booth_def", "position": 3, "reference": "C"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "All Saints' Day", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 18.641199542034883, "model_calls": 1, "failed_calls": 0, "out_tokens": 42, "rep": 1, "qid": "bt_2624", "dataset": "triviaqa", "arm": "json_nogate", "position": 0, "reference": "All Saint's Day"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "All Saints' Day", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 10.832752334012184, "model_calls": 1, "failed_calls": 0, "out_tokens": 42, "rep": 1, "qid": "bt_2624", "dataset": "triviaqa", "arm": "booth_def", "position": 1, "reference": "All Saint's Day"} +{"status": "ANSWERED", "confidence": null, "predicted": "All Saints' Day", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 3.913268957985565, "model_calls": 1, "failed_calls": 0, "out_tokens": 8, "rep": 1, "qid": "bt_2624", "dataset": "triviaqa", "arm": "plain", "position": 2, "reference": "All Saint's Day"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "All Saints' Day", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 29.88322454196168, "model_calls": 1, "failed_calls": 0, "out_tokens": 94, "rep": 1, "qid": "bt_2624", "dataset": "triviaqa", "arm": "booth_reason", "position": 3, "reference": "All Saint's Day"} +{"status": "ANSWERED", "confidence": 1.0, "predicted": "45", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 54.76603158394573, "model_calls": 1, "failed_calls": 0, "out_tokens": 209, "rep": 1, "qid": "gsm8k-test-374", "dataset": "gsm8k", "arm": "json_nogate", "position": 0, "reference": "45"} +{"status": "ACCEPTED", "confidence": 1.0, "predicted": "45", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 44.6569303330034, "model_calls": 1, "failed_calls": 0, "out_tokens": 209, "rep": 1, "qid": "gsm8k-test-374", "dataset": "gsm8k", "arm": "booth_def", "position": 1, "reference": "45"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "45", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 49.3800363339833, "model_calls": 1, "failed_calls": 0, "out_tokens": 227, "rep": 1, "qid": "gsm8k-test-374", "dataset": "gsm8k", "arm": "booth_reason", "position": 2, "reference": "45"} +{"status": "ANSWERED", "confidence": null, "predicted": "45", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 52.1587332499912, "model_calls": 1, "failed_calls": 0, "out_tokens": 209, "rep": 1, "qid": "gsm8k-test-374", "dataset": "gsm8k", "arm": "plain", "position": 3, "reference": "45"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Pictures at an Exhibition", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 26.95085787505377, "model_calls": 1, "failed_calls": 0, "out_tokens": 80, "rep": 1, "qid": "wh_326", "dataset": "triviaqa", "arm": "booth_reason", "position": 0, "reference": "PICTURES AT AN EXHIBITION"} +{"status": "ANSWERED", "confidence": null, "predicted": "Pictures at an Exhibition", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 5.363146666961256, "model_calls": 1, "failed_calls": 0, "out_tokens": 8, "rep": 1, "qid": "wh_326", "dataset": "triviaqa", "arm": "plain", "position": 1, "reference": "PICTURES AT AN EXHIBITION"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Pictures at an Exhibition", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 18.09522575000301, "model_calls": 1, "failed_calls": 0, "out_tokens": 42, "rep": 1, "qid": "wh_326", "dataset": "triviaqa", "arm": "booth_def", "position": 2, "reference": "PICTURES AT AN EXHIBITION"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "Pictures at an Exhibition", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 10.160864666977432, "model_calls": 1, "failed_calls": 0, "out_tokens": 42, "rep": 1, "qid": "wh_326", "dataset": "triviaqa", "arm": "json_nogate", "position": 3, "reference": "PICTURES AT AN EXHIBITION"} +{"status": "ANSWERED", "confidence": null, "predicted": "540", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 28.38700400001835, "model_calls": 1, "failed_calls": 0, "out_tokens": 112, "rep": 1, "qid": "gsm8k-test-3", "dataset": "gsm8k", "arm": "plain", "position": 0, "reference": "540"} +{"status": "ANSWERED", "confidence": 1.0, "predicted": "540", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 51.533535541035235, "model_calls": 1, "failed_calls": 0, "out_tokens": 155, "rep": 1, "qid": "gsm8k-test-3", "dataset": "gsm8k", "arm": "json_nogate", "position": 1, "reference": "540"} +{"status": "ACCEPTED", "confidence": 1.0, "predicted": "540", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 37.7504926660331, "model_calls": 1, "failed_calls": 0, "out_tokens": 155, "rep": 1, "qid": "gsm8k-test-3", "dataset": "gsm8k", "arm": "booth_def", "position": 2, "reference": "540"} +{"status": "ACCEPTED", "confidence": 1.0, "predicted": "540", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 43.133859333000146, "model_calls": 1, "failed_calls": 0, "out_tokens": 155, "rep": 1, "qid": "gsm8k-test-3", "dataset": "gsm8k", "arm": "booth_reason", "position": 3, "reference": "540"} +{"status": "ANSWERED", "confidence": null, "predicted": "3", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 51.258433915965725, "model_calls": 1, "failed_calls": 0, "out_tokens": 198, "rep": 1, "qid": "gsm8k-test-396", "dataset": "gsm8k", "arm": "plain", "position": 0, "reference": "3"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "3", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 68.86142237501917, "model_calls": 1, "failed_calls": 0, "out_tokens": 252, "rep": 1, "qid": "gsm8k-test-396", "dataset": "gsm8k", "arm": "booth_reason", "position": 1, "reference": "3"} +{"status": "ANSWERED", "confidence": 1.0, "predicted": "3", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 66.68678416602779, "model_calls": 1, "failed_calls": 0, "out_tokens": 251, "rep": 1, "qid": "gsm8k-test-396", "dataset": "gsm8k", "arm": "json_nogate", "position": 2, "reference": "3"} +{"status": "ACCEPTED", "confidence": 1.0, "predicted": "3", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 60.310047167004086, "model_calls": 1, "failed_calls": 0, "out_tokens": 251, "rep": 1, "qid": "gsm8k-test-396", "dataset": "gsm8k", "arm": "booth_def", "position": 3, "reference": "3"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "C", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 38.31903408304788, "model_calls": 1, "failed_calls": 0, "out_tokens": 119, "rep": 1, "qid": "truthfulqa_mc-validation-267", "dataset": "truthfulqa", "arm": "json_nogate", "position": 0, "reference": "C"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "C", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 70.74517687497428, "model_calls": 1, "failed_calls": 0, "out_tokens": 233, "rep": 1, "qid": "truthfulqa_mc-validation-267", "dataset": "truthfulqa", "arm": "booth_reason", "position": 1, "reference": "C"} +{"status": "ANSWERED", "confidence": null, "predicted": "C", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 4.286435624992009, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "truthfulqa_mc-validation-267", "dataset": "truthfulqa", "arm": "plain", "position": 2, "reference": "C"} +{"status": "AMBIGUOUS", "confidence": 0.95, "predicted": "C", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "ambiguous", "latency_s": 40.2592652910389, "model_calls": 1, "failed_calls": 0, "out_tokens": 119, "rep": 1, "qid": "truthfulqa_mc-validation-267", "dataset": "truthfulqa", "arm": "booth_def", "position": 3, "reference": "C"} +{"status": "ANSWERED", "confidence": null, "predicted": "Warsaw", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 3.443426540994551, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "bt_2652", "dataset": "triviaqa", "arm": "plain", "position": 0, "reference": "Warsaw"} +{"status": "ANSWERED", "confidence": 0.99, "predicted": "Warsaw", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 18.77794579201145, "model_calls": 1, "failed_calls": 0, "out_tokens": 40, "rep": 1, "qid": "bt_2652", "dataset": "triviaqa", "arm": "json_nogate", "position": 1, "reference": "Warsaw"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Warsaw", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 32.081906167033594, "model_calls": 1, "failed_calls": 0, "out_tokens": 83, "rep": 1, "qid": "bt_2652", "dataset": "triviaqa", "arm": "booth_reason", "position": 2, "reference": "Warsaw"} +{"status": "ACCEPTED", "confidence": 0.99, "predicted": "Warsaw", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 21.133238541020546, "model_calls": 1, "failed_calls": 0, "out_tokens": 40, "rep": 1, "qid": "bt_2652", "dataset": "triviaqa", "arm": "booth_def", "position": 3, "reference": "Warsaw"} +{"status": "ACCEPTED", "confidence": 0.9, "predicted": "C", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 92.71620308398269, "model_calls": 1, "failed_calls": 0, "out_tokens": 255, "rep": 1, "qid": "truthfulqa_mc-validation-632", "dataset": "truthfulqa", "arm": "booth_reason", "position": 0, "reference": "C"} +{"status": "ANSWERED", "confidence": 0.6, "predicted": "A", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 26.839674542017747, "model_calls": 1, "failed_calls": 0, "out_tokens": 72, "rep": 1, "qid": "truthfulqa_mc-validation-632", "dataset": "truthfulqa", "arm": "json_nogate", "position": 1, "reference": "C"} +{"status": "AMBIGUOUS", "confidence": 0.6, "predicted": "A", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "ambiguous", "latency_s": 18.8820712500019, "model_calls": 1, "failed_calls": 0, "out_tokens": 72, "rep": 1, "qid": "truthfulqa_mc-validation-632", "dataset": "truthfulqa", "arm": "booth_def", "position": 2, "reference": "C"} +{"status": "ANSWERED", "confidence": null, "predicted": "C", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 4.07515595795121, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "truthfulqa_mc-validation-632", "dataset": "truthfulqa", "arm": "plain", "position": 3, "reference": "C"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "Rose oil", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 18.111519083031453, "model_calls": 1, "failed_calls": 0, "out_tokens": 40, "rep": 1, "qid": "sfq_22425", "dataset": "triviaqa", "arm": "json_nogate", "position": 0, "reference": "Attar"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Rose oil", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 10.59176529198885, "model_calls": 1, "failed_calls": 0, "out_tokens": 40, "rep": 1, "qid": "sfq_22425", "dataset": "triviaqa", "arm": "booth_def", "position": 1, "reference": "Attar"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Rose oil", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 27.66133999999147, "model_calls": 1, "failed_calls": 0, "out_tokens": 70, "rep": 1, "qid": "sfq_22425", "dataset": "triviaqa", "arm": "booth_reason", "position": 2, "reference": "Attar"} +{"status": "ANSWERED", "confidence": null, "predicted": "Rose oil", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 3.7001552500296384, "model_calls": 1, "failed_calls": 0, "out_tokens": 6, "rep": 1, "qid": "sfq_22425", "dataset": "triviaqa", "arm": "plain", "position": 3, "reference": "Attar"} +{"status": "ANSWERED", "confidence": null, "predicted": "4", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 28.58005733299069, "model_calls": 1, "failed_calls": 0, "out_tokens": 98, "rep": 1, "qid": "gsm8k-test-490", "dataset": "gsm8k", "arm": "plain", "position": 0, "reference": "4"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "4", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 47.77640137495473, "model_calls": 1, "failed_calls": 0, "out_tokens": 148, "rep": 1, "qid": "gsm8k-test-490", "dataset": "gsm8k", "arm": "booth_reason", "position": 1, "reference": "4"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "4", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 41.5548891670187, "model_calls": 1, "failed_calls": 0, "out_tokens": 132, "rep": 1, "qid": "gsm8k-test-490", "dataset": "gsm8k", "arm": "json_nogate", "position": 2, "reference": "4"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "4", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 34.71953687496716, "model_calls": 1, "failed_calls": 0, "out_tokens": 132, "rep": 1, "qid": "gsm8k-test-490", "dataset": "gsm8k", "arm": "booth_def", "position": 3, "reference": "4"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Hidden Figures", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 32.92891962500289, "model_calls": 1, "failed_calls": 0, "out_tokens": 94, "rep": 1, "qid": "jp_925", "dataset": "triviaqa", "arm": "booth_reason", "position": 0, "reference": "Good Will Hunting"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "Hidden Figures", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 17.282329125038814, "model_calls": 1, "failed_calls": 0, "out_tokens": 40, "rep": 1, "qid": "jp_925", "dataset": "triviaqa", "arm": "json_nogate", "position": 1, "reference": "Good Will Hunting"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Hidden Figures", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 11.611162875022274, "model_calls": 1, "failed_calls": 0, "out_tokens": 40, "rep": 1, "qid": "jp_925", "dataset": "triviaqa", "arm": "booth_def", "position": 2, "reference": "Good Will Hunting"} +{"status": "ANSWERED", "confidence": null, "predicted": "Hidden Figures", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 4.6987964999862015, "model_calls": 1, "failed_calls": 0, "out_tokens": 6, "rep": 1, "qid": "jp_925", "dataset": "triviaqa", "arm": "plain", "position": 3, "reference": "Good Will Hunting"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "21", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 123.44594991696067, "model_calls": 1, "failed_calls": 0, "out_tokens": 412, "rep": 1, "qid": "gsm8k-test-827", "dataset": "gsm8k", "arm": "booth_def", "position": 0, "reference": "11"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "21", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 127.92395154101541, "model_calls": 1, "failed_calls": 0, "out_tokens": 412, "rep": 1, "qid": "gsm8k-test-827", "dataset": "gsm8k", "arm": "json_nogate", "position": 1, "reference": "11"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "11.00", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 130.37704466603464, "model_calls": 1, "failed_calls": 0, "out_tokens": 345, "rep": 1, "qid": "gsm8k-test-827", "dataset": "gsm8k", "arm": "booth_reason", "position": 2, "reference": "11"} +{"status": "ANSWERED", "confidence": null, "predicted": "11", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 111.49145770800533, "model_calls": 1, "failed_calls": 0, "out_tokens": 331, "rep": 1, "qid": "gsm8k-test-827", "dataset": "gsm8k", "arm": "plain", "position": 3, "reference": "11"} +{"status": "ANSWERED", "confidence": null, "predicted": "14400", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 41.26714579202235, "model_calls": 1, "failed_calls": 0, "out_tokens": 121, "rep": 1, "qid": "gsm8k-test-401", "dataset": "gsm8k", "arm": "plain", "position": 0, "reference": "14400"} +{"status": "ANSWERED", "confidence": 0.99, "predicted": "14400", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 100.46347837499343, "model_calls": 1, "failed_calls": 0, "out_tokens": 217, "rep": 1, "qid": "gsm8k-test-401", "dataset": "gsm8k", "arm": "json_nogate", "position": 1, "reference": "14400"} +{"status": "ACCEPTED", "confidence": 0.99, "predicted": "14400", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 90.89375204098178, "model_calls": 1, "failed_calls": 0, "out_tokens": 217, "rep": 1, "qid": "gsm8k-test-401", "dataset": "gsm8k", "arm": "booth_def", "position": 2, "reference": "14400"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "14400", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 77.97149308404187, "model_calls": 1, "failed_calls": 0, "out_tokens": 175, "rep": 1, "qid": "gsm8k-test-401", "dataset": "gsm8k", "arm": "booth_reason", "position": 3, "reference": "14400"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "155", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 84.73888449999504, "model_calls": 1, "failed_calls": 0, "out_tokens": 214, "rep": 1, "qid": "gsm8k-test-785", "dataset": "gsm8k", "arm": "booth_def", "position": 0, "reference": "155"} +{"status": "ANSWERED", "confidence": null, "predicted": "155", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 84.69256745901657, "model_calls": 1, "failed_calls": 0, "out_tokens": 252, "rep": 1, "qid": "gsm8k-test-785", "dataset": "gsm8k", "arm": "plain", "position": 1, "reference": "155"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "155", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 71.22611212497577, "model_calls": 1, "failed_calls": 0, "out_tokens": 214, "rep": 1, "qid": "gsm8k-test-785", "dataset": "gsm8k", "arm": "json_nogate", "position": 2, "reference": "155"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "155", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 68.87402066698996, "model_calls": 1, "failed_calls": 0, "out_tokens": 226, "rep": 1, "qid": "gsm8k-test-785", "dataset": "gsm8k", "arm": "booth_reason", "position": 3, "reference": "155"} +{"status": "ANSWERED", "confidence": 0.9, "predicted": "D", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 35.84925083396956, "model_calls": 1, "failed_calls": 0, "out_tokens": 122, "rep": 1, "qid": "truthfulqa_mc-validation-236", "dataset": "truthfulqa", "arm": "json_nogate", "position": 0, "reference": "D"} +{"status": "ANSWERED", "confidence": null, "predicted": "D", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 5.443566041998565, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "truthfulqa_mc-validation-236", "dataset": "truthfulqa", "arm": "plain", "position": 1, "reference": "D"} +{"status": "AMBIGUOUS", "confidence": 0.9, "predicted": "D", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "ambiguous", "latency_s": 40.24424420797732, "model_calls": 1, "failed_calls": 0, "out_tokens": 122, "rep": 1, "qid": "truthfulqa_mc-validation-236", "dataset": "truthfulqa", "arm": "booth_def", "position": 2, "reference": "D"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "D", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 71.60042495798552, "model_calls": 1, "failed_calls": 0, "out_tokens": 280, "rep": 1, "qid": "truthfulqa_mc-validation-236", "dataset": "truthfulqa", "arm": "booth_reason", "position": 3, "reference": "D"} +{"status": "ACCEPTED", "confidence": 0.9, "predicted": "B", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 60.44256766705075, "model_calls": 1, "failed_calls": 0, "out_tokens": 241, "rep": 1, "qid": "truthfulqa_mc-validation-641", "dataset": "truthfulqa", "arm": "booth_reason", "position": 0, "reference": "B"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "B", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 12.621983457996976, "model_calls": 1, "failed_calls": 0, "out_tokens": 39, "rep": 1, "qid": "truthfulqa_mc-validation-641", "dataset": "truthfulqa", "arm": "json_nogate", "position": 1, "reference": "B"} +{"status": "ANSWERED", "confidence": null, "predicted": "B", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 3.335133333981503, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "truthfulqa_mc-validation-641", "dataset": "truthfulqa", "arm": "plain", "position": 2, "reference": "B"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "B", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 12.79654033301631, "model_calls": 1, "failed_calls": 0, "out_tokens": 39, "rep": 1, "qid": "truthfulqa_mc-validation-641", "dataset": "truthfulqa", "arm": "booth_def", "position": 3, "reference": "B"} +{"status": "ANSWERED", "confidence": 1.0, "predicted": "300", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 37.72722237498965, "model_calls": 1, "failed_calls": 0, "out_tokens": 172, "rep": 1, "qid": "gsm8k-test-248", "dataset": "gsm8k", "arm": "json_nogate", "position": 0, "reference": "300"} +{"status": "ACCEPTED", "confidence": 1.0, "predicted": "300", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 36.61638004198903, "model_calls": 1, "failed_calls": 0, "out_tokens": 171, "rep": 1, "qid": "gsm8k-test-248", "dataset": "gsm8k", "arm": "booth_reason", "position": 1, "reference": "300"} +{"status": "ANSWERED", "confidence": null, "predicted": "300", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 24.2526201250148, "model_calls": 1, "failed_calls": 0, "out_tokens": 129, "rep": 1, "qid": "gsm8k-test-248", "dataset": "gsm8k", "arm": "plain", "position": 2, "reference": "300"} +{"status": "ACCEPTED", "confidence": 1.0, "predicted": "300", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 35.7172622079961, "model_calls": 1, "failed_calls": 0, "out_tokens": 172, "rep": 1, "qid": "gsm8k-test-248", "dataset": "gsm8k", "arm": "booth_def", "position": 3, "reference": "300"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "Ithaca", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 13.362945917004254, "model_calls": 1, "failed_calls": 0, "out_tokens": 41, "rep": 1, "qid": "sfq_380", "dataset": "triviaqa", "arm": "json_nogate", "position": 0, "reference": "Ithaca"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Ithaca", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 16.703657375008333, "model_calls": 1, "failed_calls": 0, "out_tokens": 60, "rep": 1, "qid": "sfq_380", "dataset": "triviaqa", "arm": "booth_reason", "position": 1, "reference": "Ithaca"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Ithaca", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 13.23430837498745, "model_calls": 1, "failed_calls": 0, "out_tokens": 41, "rep": 1, "qid": "sfq_380", "dataset": "triviaqa", "arm": "booth_def", "position": 2, "reference": "Ithaca"} +{"status": "ANSWERED", "confidence": null, "predicted": "Ithaca", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 2.001915375003591, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "sfq_380", "dataset": "triviaqa", "arm": "plain", "position": 3, "reference": "Ithaca"} +{"status": "AMBIGUOUS", "confidence": 0.6, "predicted": "A", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "ambiguous", "latency_s": 59.25618574995315, "model_calls": 1, "failed_calls": 0, "out_tokens": 264, "rep": 1, "qid": "truthfulqa_mc-validation-248", "dataset": "truthfulqa", "arm": "booth_reason", "position": 0, "reference": "D"} +{"status": "ANSWERED", "confidence": 0.6, "predicted": "A", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 30.572413375019096, "model_calls": 1, "failed_calls": 0, "out_tokens": 127, "rep": 1, "qid": "truthfulqa_mc-validation-248", "dataset": "truthfulqa", "arm": "json_nogate", "position": 1, "reference": "D"} +{"status": "ANSWERED", "confidence": null, "predicted": "A", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 3.1716139999916777, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "truthfulqa_mc-validation-248", "dataset": "truthfulqa", "arm": "plain", "position": 2, "reference": "D"} +{"status": "AMBIGUOUS", "confidence": 0.6, "predicted": "A", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "ambiguous", "latency_s": 29.886975625006016, "model_calls": 1, "failed_calls": 0, "out_tokens": 127, "rep": 1, "qid": "truthfulqa_mc-validation-248", "dataset": "truthfulqa", "arm": "booth_def", "position": 3, "reference": "D"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "1200", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 52.03994083299767, "model_calls": 1, "failed_calls": 0, "out_tokens": 237, "rep": 1, "qid": "gsm8k-test-467", "dataset": "gsm8k", "arm": "json_nogate", "position": 0, "reference": "1200"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "1200", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 42.54874345898861, "model_calls": 1, "failed_calls": 0, "out_tokens": 237, "rep": 1, "qid": "gsm8k-test-467", "dataset": "gsm8k", "arm": "booth_def", "position": 1, "reference": "1200"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "1200", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 45.357058082998265, "model_calls": 1, "failed_calls": 0, "out_tokens": 210, "rep": 1, "qid": "gsm8k-test-467", "dataset": "gsm8k", "arm": "booth_reason", "position": 2, "reference": "1200"} +{"status": "ANSWERED", "confidence": null, "predicted": "1200", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 45.77316799998516, "model_calls": 1, "failed_calls": 0, "out_tokens": 232, "rep": 1, "qid": "gsm8k-test-467", "dataset": "gsm8k", "arm": "plain", "position": 3, "reference": "1200"} +{"status": "ANSWERED", "confidence": 0.7, "predicted": "C", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 30.46096179203596, "model_calls": 1, "failed_calls": 0, "out_tokens": 127, "rep": 1, "qid": "truthfulqa_mc-validation-312", "dataset": "truthfulqa", "arm": "json_nogate", "position": 0, "reference": "C"} +{"status": "AMBIGUOUS", "confidence": 0.6, "predicted": "B", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "ambiguous", "latency_s": 77.04135220800526, "model_calls": 1, "failed_calls": 0, "out_tokens": 365, "rep": 1, "qid": "truthfulqa_mc-validation-312", "dataset": "truthfulqa", "arm": "booth_reason", "position": 1, "reference": "C"} +{"status": "ANSWERED", "confidence": null, "predicted": "C", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 4.37569233297836, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "truthfulqa_mc-validation-312", "dataset": "truthfulqa", "arm": "plain", "position": 2, "reference": "C"} +{"status": "AMBIGUOUS", "confidence": 0.7, "predicted": "C", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "ambiguous", "latency_s": 31.026928332983516, "model_calls": 1, "failed_calls": 0, "out_tokens": 127, "rep": 1, "qid": "truthfulqa_mc-validation-312", "dataset": "truthfulqa", "arm": "booth_def", "position": 3, "reference": "C"} +{"status": "AMBIGUOUS", "confidence": 0.7, "predicted": "C", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "ambiguous", "latency_s": 25.522662958013825, "model_calls": 1, "failed_calls": 0, "out_tokens": 85, "rep": 1, "qid": "truthfulqa_mc-validation-323", "dataset": "truthfulqa", "arm": "booth_def", "position": 0, "reference": "C"} +{"status": "ANSWERED", "confidence": null, "predicted": "C", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 4.344783375039697, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "truthfulqa_mc-validation-323", "dataset": "truthfulqa", "arm": "plain", "position": 1, "reference": "C"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "C", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 60.979536166996695, "model_calls": 1, "failed_calls": 0, "out_tokens": 266, "rep": 1, "qid": "truthfulqa_mc-validation-323", "dataset": "truthfulqa", "arm": "booth_reason", "position": 2, "reference": "C"} +{"status": "ANSWERED", "confidence": 0.7, "predicted": "C", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 22.121743457973935, "model_calls": 1, "failed_calls": 0, "out_tokens": 85, "rep": 1, "qid": "truthfulqa_mc-validation-323", "dataset": "truthfulqa", "arm": "json_nogate", "position": 3, "reference": "C"} +{"status": "ANSWERED", "confidence": null, "predicted": "D", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 3.158640500041656, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "truthfulqa_mc-validation-151", "dataset": "truthfulqa", "arm": "plain", "position": 0, "reference": "D"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "D", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 13.952709540957585, "model_calls": 1, "failed_calls": 0, "out_tokens": 39, "rep": 1, "qid": "truthfulqa_mc-validation-151", "dataset": "truthfulqa", "arm": "booth_def", "position": 1, "reference": "D"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "D", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 8.435226500034332, "model_calls": 1, "failed_calls": 0, "out_tokens": 39, "rep": 1, "qid": "truthfulqa_mc-validation-151", "dataset": "truthfulqa", "arm": "json_nogate", "position": 2, "reference": "D"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "D", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 55.0259616249823, "model_calls": 1, "failed_calls": 0, "out_tokens": 240, "rep": 1, "qid": "truthfulqa_mc-validation-151", "dataset": "truthfulqa", "arm": "booth_reason", "position": 3, "reference": "D"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "298", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 54.570927541994024, "model_calls": 1, "failed_calls": 0, "out_tokens": 251, "rep": 1, "qid": "gsm8k-test-681", "dataset": "gsm8k", "arm": "booth_reason", "position": 0, "reference": "298"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "298", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 50.855851708038244, "model_calls": 1, "failed_calls": 0, "out_tokens": 236, "rep": 1, "qid": "gsm8k-test-681", "dataset": "gsm8k", "arm": "booth_def", "position": 1, "reference": "298"} +{"status": "ANSWERED", "confidence": null, "predicted": "298", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 51.073529958957806, "model_calls": 1, "failed_calls": 0, "out_tokens": 245, "rep": 1, "qid": "gsm8k-test-681", "dataset": "gsm8k", "arm": "plain", "position": 2, "reference": "298"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "298", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 45.37883829098428, "model_calls": 1, "failed_calls": 0, "out_tokens": 236, "rep": 1, "qid": "gsm8k-test-681", "dataset": "gsm8k", "arm": "json_nogate", "position": 3, "reference": "298"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "Cascade Range", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 12.933947500016075, "model_calls": 1, "failed_calls": 0, "out_tokens": 40, "rep": 1, "qid": "odql_9445", "dataset": "triviaqa", "arm": "json_nogate", "position": 0, "reference": "Cascades"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Cascade Range", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 20.930632666975725, "model_calls": 1, "failed_calls": 0, "out_tokens": 75, "rep": 1, "qid": "odql_9445", "dataset": "triviaqa", "arm": "booth_reason", "position": 1, "reference": "Cascades"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "Cascade Range", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 13.344972541963216, "model_calls": 1, "failed_calls": 0, "out_tokens": 40, "rep": 1, "qid": "odql_9445", "dataset": "triviaqa", "arm": "booth_def", "position": 2, "reference": "Cascades"} +{"status": "ANSWERED", "confidence": null, "predicted": "Cascade Range", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 2.8238412920036353, "model_calls": 1, "failed_calls": 0, "out_tokens": 6, "rep": 1, "qid": "odql_9445", "dataset": "triviaqa", "arm": "plain", "position": 3, "reference": "Cascades"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "48", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 46.84320716699585, "model_calls": 1, "failed_calls": 0, "out_tokens": 187, "rep": 1, "qid": "gsm8k-test-1277", "dataset": "gsm8k", "arm": "booth_def", "position": 0, "reference": "48"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "48", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 40.69169104099274, "model_calls": 1, "failed_calls": 0, "out_tokens": 187, "rep": 1, "qid": "gsm8k-test-1277", "dataset": "gsm8k", "arm": "json_nogate", "position": 1, "reference": "48"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "48", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 60.96796295797685, "model_calls": 1, "failed_calls": 0, "out_tokens": 196, "rep": 1, "qid": "gsm8k-test-1277", "dataset": "gsm8k", "arm": "booth_reason", "position": 2, "reference": "48"} +{"status": "ANSWERED", "confidence": null, "predicted": "48", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 39.461639250046574, "model_calls": 1, "failed_calls": 0, "out_tokens": 163, "rep": 1, "qid": "gsm8k-test-1277", "dataset": "gsm8k", "arm": "plain", "position": 3, "reference": "48"} +{"status": "AMBIGUOUS", "confidence": 0.95, "predicted": "James Caan", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "ambiguous", "latency_s": 26.237032957957126, "model_calls": 1, "failed_calls": 0, "out_tokens": 106, "rep": 1, "qid": "sfq_7483", "dataset": "triviaqa", "arm": "booth_def", "position": 0, "reference": "JAMES CAAN"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "James Caan", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 33.97170895797899, "model_calls": 1, "failed_calls": 0, "out_tokens": 163, "rep": 1, "qid": "sfq_7483", "dataset": "triviaqa", "arm": "booth_reason", "position": 1, "reference": "JAMES CAAN"} +{"status": "ANSWERED", "confidence": null, "predicted": "James Caan", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 3.6255367909907363, "model_calls": 1, "failed_calls": 0, "out_tokens": 7, "rep": 1, "qid": "sfq_7483", "dataset": "triviaqa", "arm": "plain", "position": 2, "reference": "JAMES CAAN"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "James Caan", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 24.691575166012626, "model_calls": 1, "failed_calls": 0, "out_tokens": 106, "rep": 1, "qid": "sfq_7483", "dataset": "triviaqa", "arm": "json_nogate", "position": 3, "reference": "JAMES CAAN"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "4", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "correct", "latency_s": 75.76377791597042, "model_calls": 1, "failed_calls": 0, "out_tokens": 366, "rep": 1, "qid": "gsm8k-test-359", "dataset": "gsm8k", "arm": "booth_reason", "position": 0, "reference": "4"} +{"status": "ANSWERED", "confidence": null, "predicted": "4", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 46.16227320895996, "model_calls": 1, "failed_calls": 0, "out_tokens": 250, "rep": 1, "qid": "gsm8k-test-359", "dataset": "gsm8k", "arm": "plain", "position": 1, "reference": "4"} +{"status": "ACCEPTED", "confidence": 0.95, "predicted": "5.33", "raw_correct": false, "n_attempts": 1, "error": null, "outcome": "incorrect", "latency_s": 87.48328958399361, "model_calls": 1, "failed_calls": 0, "out_tokens": 430, "rep": 1, "qid": "gsm8k-test-359", "dataset": "gsm8k", "arm": "booth_def", "position": 2, "reference": "4"} +{"status": "ANSWERED", "confidence": 0.95, "predicted": "5.33", "raw_correct": false, "n_attempts": null, "error": null, "outcome": "incorrect", "latency_s": 80.85963175003417, "model_calls": 1, "failed_calls": 0, "out_tokens": 430, "rep": 1, "qid": "gsm8k-test-359", "dataset": "gsm8k", "arm": "json_nogate", "position": 3, "reference": "4"} +{"status": "AMBIGUOUS", "confidence": 0.7, "predicted": "A", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "ambiguous", "latency_s": 54.61906520900084, "model_calls": 1, "failed_calls": 0, "out_tokens": 257, "rep": 1, "qid": "truthfulqa_mc-validation-280", "dataset": "truthfulqa", "arm": "booth_reason", "position": 0, "reference": "A"} +{"status": "ANSWERED", "confidence": 0.6, "predicted": "A", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 32.45736175001366, "model_calls": 1, "failed_calls": 0, "out_tokens": 130, "rep": 1, "qid": "truthfulqa_mc-validation-280", "dataset": "truthfulqa", "arm": "json_nogate", "position": 1, "reference": "A"} +{"status": "AMBIGUOUS", "confidence": 0.6, "predicted": "A", "raw_correct": true, "n_attempts": 1, "error": null, "outcome": "ambiguous", "latency_s": 26.988505457993597, "model_calls": 1, "failed_calls": 0, "out_tokens": 130, "rep": 1, "qid": "truthfulqa_mc-validation-280", "dataset": "truthfulqa", "arm": "booth_def", "position": 2, "reference": "A"} +{"status": "ANSWERED", "confidence": null, "predicted": "A", "raw_correct": true, "n_attempts": null, "error": null, "outcome": "correct", "latency_s": 4.494250499992631, "model_calls": 1, "failed_calls": 0, "out_tokens": 5, "rep": 1, "qid": "truthfulqa_mc-validation-280", "dataset": "truthfulqa", "arm": "plain", "position": 3, "reference": "A"}