feat(memoria): improve RAG retrieval and evaluation

[juemimgcd]

Summary

• expand dense and lexical retrieval candidate pools to 50, apply Dense 1.0 / Lexical 0.3 weighted RRF, and tune filtered pgvector HNSW searches
• add memory-revision embeddings with a nullable migration, best-effort writes, lexical fallback during partial backfill, and a separate resumable embedding backfill CLI
• add a reproducible BEIR SciFact benchmark with model/text-aware vector caching, repository provenance, failure samples, and production-consistent weighted reranker candidates

Why

The 50-query, 1000-document SciFact evaluation improved Recall@10 from 0.80 to 0.84, MRR@10 from 0.6629 to 0.6682, and NDCG@10 from 0.6976 to 0.7081. Failure analysis showed that the previous 10-item candidate pool was too small and equal-weight lexical fusion introduced noise.

Validation

• ruff check app main.py tests
• python -m compileall -q app main.py
• python -m pytest -q -p no:cacheprovider -m "not integration" — 345 passed, 2 deselected, 8 subtests passed
• fresh alembic -c app/mneme/memoria/server/alembic.ini upgrade head against PostgreSQL 17 + pgvector 0.8.5
• disposable PostgreSQL reproduction verified lexical ordering while scoped embeddings are incomplete and semantic ordering after completion
• cache diagnostic verified that changing one document re-encodes only that document while unchanged vectors remain cached

Rollout

The migration keeps historical memory embeddings nullable. Retrieval remains lexical for an affected scope until its historical revisions are fully backfilled:

python -m app.mneme.memoria.server.cli.embedding_backfill --memory --dry-run
python -m app.mneme.memoria.server.cli.embedding_backfill --memory --batch-size 100