Run full LME-S 500q benchmark on GPU workstation

[Liorrr]

Context

KS76/77 changes (universal prompt, temporal boost, importance scoring, abstention fix) are merged to master. A 52-question sample shows very promising results:

• Abstention: 19% → 0% (10 recovered answers in sample)
• Zero regressions on previously correct answers
• 29 previously-wrong answers changed with visibly improved specificity
• Projected ~2.5x improvement over 24.2% baseline

Task

Run the full 500-question LME-S benchmark with GPT-4o judge on a machine with enough resources. Dev machine (CPU-only, 16GB RAM) chokes when running alongside other workloads.

Steps

1. Set up GPU workstation with Ollama + qwen2.5:1.5b (and optionally 3b)
2. Build latest master: cargo build --release -p shrimpk-daemon
3. Start daemon: shrimpk-daemon --port 11435
4. Run generation: python benchmarks/run_longmemeval.py (500q, ~11h on CPU)
5. Run GPT-4o evaluation on results
6. Compare vs baseline (24.2% overall, 25.3% task-avg)

Partial results (52/500 sample)

Old baseline: 13/52 correct (25.0%)
Abstentions:  10/52 → 0/52
Regressions:  0
Recovered:    10 answers
Changed:      29 answers (quality TBD)

Success criteria

• Full 500q scored with GPT-4o judge
• Per-category breakdown (IE, KU, TR, NR, AB)
• Comparison table vs 24.2% baseline

Labels

benchmark, infrastructure

[Liorrr]

Review (2026-06-19): keep open (infra/GPU) — corrections before the run

Run-readiness verified: dataset present (500q, 30 abstention IDs), oracle present, GPT-4o judge at evaluate_qa.py (reads OPENAI_API_KEY), and baseline provenance confirmed (results/lme_500q_v1_overnight...eval-results-gpt-4o scores exactly 0.242).

Corrections before anyone executes:

1. Runner: step 4 names run_longmemeval.py (v1), but the promising abstention numbers (19%→0%) came from v2 — only v2 instruments refusals (v1 has zero). Confirm v2 is the runner of record.
2. Missing dep: requests is absent from requirements-lite.txt (only in -full).
3. Taxonomy mismatch: success criteria ask for IE/KU/TR/NR/AB (the seeded micro-benchmark taxonomy); evaluate_qa.py emits the 6 LongMemEval qtypes — needs a mapping or separate scorer.
4. Pin the exact reader model (issue says qwen2.5:1.5b; artifact is named _qwen25_1b).

Unattended-run risk: clear_memories() degrades silently on daemon-restart failure (v2:110-111) → could feed empty context to the rest of a 500q run with no abort.

[Liorrr]

Validation readiness (2026-06-20, master @8e179c3)

Engine validated on merged master before the GPU run: micro-benchmark suite is green where expected — embedding-only 17/20, seeded 18/20, abstention 5/5, negative-recall 3/3, consolidation 16/20 (deterministic across 2 runs). KS76/77/78 retrieval changes are active and exercised via /api/echo.

This run should ALSO capture #12's reader-model effect (qwen2.5:1.5b default, now merged). Pre-run reminders from the earlier review still apply: confirm v2 vs v1 runner of record; pip install requests; map the IE/KU/TR/NR/AB success-criteria taxonomy to the 6 LME qtypes evaluate_qa.py emits; and isolate SHRIMPK_DATA_DIR so the run doesn't delete the dev store. Still requires the GPU workstation (CPU is ~11h).