Default benchmark reader model to qwen2.5:1.5b, tighten v2 depth

[Liorrr]

Summary

Both LongMemEval benchmark runners hardcoded gemma3:1b as the reader LLM (the model that turns retrieved memories + question into the final answer). #12 identifies the reader model as the accuracy bottleneck, and these runners were never updated when the project moved to qwen2.5 defaults — so benchmark scores were being capped by a weak 1B reader regardless of retrieval quality.

This sets the default reader to qwen2.5:1.5b and tightens v2's retrieval depth for higher precision.

Why qwen2.5:1.5b as the default (not 3b)

• The official 24.2% baseline was measured with gemma3:1b, and the promising 52-question sample used qwen2.5:1.5b. Defaulting to 1.5b keeps the Run full LME-S 500q benchmark on GPU workstation #19 500-question comparison clean (same reader across the comparison rather than introducing a confound).
• The dev box is CPU-only/16GB and chokes on 3B+ models, so 1.5b is the practical default for local runs.
• qwen2.5:3b / qwen2.5:4b remain available via the existing --model flag for the explicit bigger-reader hypothesis test on a GPU box.

Changes

• Both runners (run_longmemeval.py, run_longmemeval_v2.py): default reader model → qwen2.5:1.5b, applied at all default sites (ask_ollama, run_benchmark, and the --model argparse default). Still fully overridable via --model. Docstring usage examples left as-is.
• run_longmemeval_v2.py only: max_results default 15 → 5 (higher retrieval precision). v1's max_results default (10) is unchanged.

Pure Python under benchmarks/. No kernel/crate changes.

Verification

python -m py_compile benchmarks/run_longmemeval.py benchmarks/run_longmemeval_v2.py → passes. Benchmark/daemon not run (host is CPU-only/16GB).

Part of #12 (issue stays open for v3 hybrid chunking).

🤖 Generated with Claude Code

[Liorrr]

Independent review (clean-context agent) — logic + correctness review of the diff for PR #33, cross-checked against #12. I did not write this code and reviewed it fresh.

Verdict

The core change is correct and complete for its stated goal. In both runners, all three intended default sites flip from gemma3:1b → qwen2.5:1.5b:

• ask_ollama(model=...)
• run_benchmark(model=...)
• argparse --model default

And in run_longmemeval_v2.py only, max_results 15 → 5 is applied consistently at both the run_benchmark default and the argparse --max-results default. v1's max_results=10 is correctly left untouched. The 1.5b default (vs 3b) is well-justified in the PR body for keeping the #19 500q comparison clean on a CPU-only host, and --model still overrides for the bigger-reader hypothesis. No P1 issues.

Findings

P2 — run_longmemeval_v2.py:60: sibling echo_query default left stale.
def echo_query(query, max_results=15): still carries the old 15, while the PR moved the v2 retrieval depth to 5 everywhere else. WHY it matters: this is precisely the sibling/fallback default the rest of the PR was tightening, and leaving 15 here misrepresents the file's intended retrieval depth (a future caller that drops the explicit arg silently gets the old, noisier 15). WHY it's P2 not P1: the only call site (run_longmemeval_v2.py:303) always passes max_results=max_results explicitly, so the stale default is currently unreachable — no behavior bug in this run. Fix: def echo_query(query, max_results=5):. (v1 is fine — its echo_query default of 10 matches v1's unchanged run_benchmark/argparse default of 10.)

P3 — PR title vs. change mismatch.
Title: "Bump benchmark reader model to qwen2.5:3b", but every changed default is qwen2.5:1.5b (correctly, per the body's rationale). The title misdescribes the change. Fix: retitle to qwen2.5:1.5b so the merge-commit/history is accurate.

P3 — docstring usage examples drift (run_longmemeval.py:17, run_longmemeval_v2.py:12).
Both still show --model gemma3:1b, now a non-default that the issue identifies as the weak-reader bottleneck. The PR body intentionally leaves these "as-is," which is a defensible scope call, but the example now points at the exact model this PR moves away from. Optional: update the first example to qwen2.5:1.5b (and the second example already shows a valid override).

Confidence Score: 4/5

Core defaults are all changed correctly and consistently for the actual run path; the one real loose end is the stale echo_query(max_results=15) fallback in v2 (P2, currently unreachable). Title/docstring drift are cosmetic. Mergeable as-is; tightening the v2 echo_query default would make it 5/5.