local-rag-template

A template for building a Retrieval-Augmented Generation pipeline that runs entirely locally — and, more to the point, for measuring it. No RAG framework: every stage (ingest, chunking, indexing, retrieval, generation) is written explicitly in this repository, readable and profileable.

This is a template of the METHOD, not of the constants. The defaults here — final_k=8, dense retrieval, 1200-character chunks — were measured on two documents and 21 spans. They hold for this corpus and are not recommendations. What you reuse is the evaluation harness and the discipline of measuring before promoting. Start at Adapting it to your corpus.

Status: complete end-to-end prototype (ingest → query → cited answer, evaluation harness, web UI). See "Known limits" and "Negative results".

Language note: the pipeline is language-agnostic by design, but it was developed and measured on an Italian legal corpus (Italian Constitution

• GDPR in Italian). All harness numbers below come from that corpus; the sample data and the golden set are in Italian on purpose.

What it does

• Indexes local documents (PDF, DOCX, Markdown, TXT) into a single SQLite file (text + metadata, FTS5 for BM25, dense vectors as float32 blobs).
• Answers questions in the corpus language using local Ollama: bge-m3 for embeddings, qwen3:30b-a3b for generation.
• Every answer cites its sources (file + location). If the retrieved context doesn't contain the answer, the system says so instead of inventing one.

Negative results (the asset of this repo)

The pipeline is deliberately simple; the value here is the measured list of things that did not work, each with numbers and a mechanism in docs/evidence/. None of these is discoverable without an evaluation harness built before the pipeline:

• IDF gate on the BM25 branch: falsified. BM25's wins came from common domain words ("dati", "altro"), not from high-IDF anchors — the filter removed exactly the terms that produced the recoveries.
• Hybrid BM25+RRF not promoted: over 21 spans, dense and hybrid within the noise band (±1 span = 4.8%); the RRF constant is flat (sweep 10/20/60).
• Dedup/MMR on candidates: kills the target. sim(art. 33, art. 34) = 0.86: dedup eats the right twin, not the distractors.
• Neighbor expansion (seq): dominated. At equal recall, raising k uses half the context (18/21 with 8 chunks vs 14.5 avg with ±1 neighbors).
• Per-considerando breadcrumbs: made the case it was meant to fix worse (Q13: 1→11). Rejected. Splitting the formal preamble opening was promoted instead (Q13: 7→1, recall@5 0.714→0.857).
• LLM groundedness judge: 8 wrong negations out of 10 (manual audit in rag/eval/judge_audit.md); the second judge never denies support. Metric archived as non-conclusive.
• LLM correctness judge: 3 out of 5 "regressions" were better answers penalized for adding correct detail; and at temperature 0 a run-to-run flip on identical input — unquantified variance here too.
• Reasoning on grounded extraction: adds no measurable accuracy (/no_think arm), but can run away (316k characters of thinking, empty answer): that's why the fallback exists, not for elegance.

Requirements

• Windows/Linux/macOS with Python ≥ 3.11 and uv.

• Ollama running on http://localhost:11434 with:

  ollama pull bge-m3
  ollama pull qwen3:30b-a3b

• VRAM: qwen3:30b-a3b is a MoE (~19 GB in q4); on smaller GPUs Ollama offloads to CPU/RAM and generation slows accordingly. Developed and tested on an RTX 5090.

Quick start in three commands

uv sync
uv run python -m rag ingest data/sample
uv run python -m rag query "Chi nomina i giudici della Corte costituzionale?"

Web UI (single page, no build step): uv run python -m rag serve then http://127.0.0.1:8000 — streaming answers, expandable context chunks with score and source, hot-editable parameters.

Warning: the server binds to 127.0.0.1 and must stay that way. The parameters panel is an unauthenticated endpoint that changes the model and num_ctx at runtime: do not expose it behind a reverse proxy without adding authentication.

Adapting it to your corpus

The pipeline runs on any corpus. The numbers do not transfer — they were measured on two clean Markdown documents. Follow the steps in order: each one guards against a failure the next one would hide.

1. Index your documents. Drop them in data/corpus/ (git-ignored) and:

uv run python -m rag ingest data/corpus

2. Check text extraction before anything else. There is no automated quality check yet (see "Known limits"). Read a few chunks by hand:

python -c "import sqlite3; c=sqlite3.connect('index/rag.db'); [print(r[0][:300],'\n---') for r in c.execute('SELECT text FROM chunks ORDER BY RANDOM() LIMIT 5')]"

If you see letter-spaced words (par ticolare le ragionev oli), the PDF extraction is broken and everything downstream is measuring noise. Convert from a cleaner source instead — that is what tools/eurlex_to_md.py does for the sample corpus. Scanned PDFs need OCR, which is not implemented.

3. Check that your documents have structure. The chunker splits on Markdown headings and carries a breadcrumb into the embedding. The single largest measured win in this project came from structure (recall@5 0.714→0.857). With no headings, every chunk gets an empty breadcrumb and you land in the degenerate case:

python -c "import sqlite3; c=sqlite3.connect('index/rag.db'); print(c.execute(\"SELECT COUNT(*) FILTER (WHERE breadcrumb=''), COUNT(*) FROM chunks\").fetchone())"

A large fraction of empty breadcrumbs means writing a converter for your formats is the highest-value work available — before any retrieval tuning.

4. Harvest questions from real use, not from reading the documents.

uv run python -m rag serve

Ask real questions, watch which chunks come back, and keep the ones that matter. Questions written by paraphrasing the source text make every retrieval configuration look good and the harness unable to tell them apart.

5. Write your golden set. Start from the commented scaffold:

cp rag/eval/golden.example.yaml rag/eval/golden.yaml

The rules and the reasoning behind them are in the file. Aim for ~20 spans plus 4-6 negatives; below that, one span is ~5% and every comparison lands inside the noise.

6. Re-measure the defaults. The harness validates every span at startup, then reports all retrieval arms at once:

uv run python -m rag eval

Read final_k straight off the recall@k columns: pick the smallest window that captures your spans, plus margin — the last span sitting exactly at the cutoff will fall out on the first re-ingest. Compare the dense row against the hybrid arms on your corpus; on ours the difference was inside the noise band, which is why dense is the default here and may not be for you. Change what you decide in rag/config.py.

7. Only now trust the numbers — and only the ones you measured. The optional LLM judges (--groundedness) are archived as non-conclusive on this setup: read "Negative results" and rag/eval/judge_audit.md before letting them decide anything.

Layout

rag/
├── ingest/    per-format loaders behind load(path) -> list[Document]
├── chunk/     structural chunking (headings/paragraphs) with breadcrumbs
├── index/     SQLite storage: chunks + FTS5 + vectors (numpy brute force)
├── retrieve/  dense (default) and hybrid BM25+RRF (measured, in config)
├── generate/  prompt + Ollama call (explicit num_ctx, thinking handled)
├── eval/      golden set + harness: per-span recall@k/MRR + LLM judges
└── web/       minimal UI: FastAPI + SSE, no build step

Metrics on the sample corpus

Harness: 21 expected spans from 18 positive questions (+5 negatives) in rag/eval/golden.yaml; the unit of measurement is the span (file + expected text), never chunk ids. Pure dense retrieval, final_k=8 (operating window):

metric	value
recall@5	0.857
recall@8 (operating window)	0.905
recall@20	0.905
MRR (per span)	0.686

These are OUR defaults, measured on 2 documents and 21 spans — not tuned recommendations. final_k, dense retrieval, chunking parameters should be re-measured on your corpus: the harness is in the repo (uv run python -m rag eval) and is the intended way to do it.

Process rule: every number carries the arm and the index revision it was measured on; when an arm is promoted, previous numbers become history, not evidence. The runs cited as evidence are in docs/evidence/ (full JSON: answers, contexts, raw verdicts); new runs go to rag/eval/results/ (git-ignored).

Step (d): two LLM judges, and why they don't get to decide

Two independent LLM judges were built and measured for answer quality. Neither could answer the question it was built for:

• Groundedness (claims supported by the chunks?): qwen3 gets 8 negations out of 10 wrong (manual audit, rag/eval/judge_audit.md); gemma3 decomposes well but never denies support (1.0 everywhere). Two opposite biases, zero discriminating power. Metric archived as non-conclusive.
• Correctness (vs golden-set reference answers): of the 5 "regressions" it flagged between k=6 and k=10, 3 were better answers penalized for adding correct detail. Same warning sign as groundedness: at temperature 0, a run-to-run flip was observed on identical input (same answer, same reference, different verdict across runs) — correctness is less noisy, but its variance is not zero and not quantified. Its aggregates are indicative, not solid.

The choice of final_k=8 does NOT depend on these instruments — worth stating explicitly, because that's the difference between a fragile conclusion and one that survives the demolition of half the measurement tooling.

At 6/8/10, window recall is identical (19/21). That number is deterministic and reproducible with python -m rag eval. Correctness and abstentions come out equivalent too, but after manual re-adjudication, not as raw judge output: the gemma3 verdicts read 13/4/1 at k=6 against 11/6/1 at k=10, and inspecting the answers showed 3 of the 5 flagged regressions were better answers penalized for adding correct detail. Overruling an instrument that this same README documents as unreliable is the correct treatment — but the resulting figures are adjudicated, not measured. Raw verdicts are in docs/evidence/rejudge-*.json, the adjudication in rag/eval/judge_audit.md.

So the decision was made by retrieval recall (deterministic) and token cost, deliberately not by the judges. It's a margin-under-corpus-drift choice: the 19th span sits at rank 6, and k=6 would lose it on the first re-ingest; k=10 costs ~2000 tokens with no additional recall.

Generator thinking: /no_think by default, with fallback

think: false doesn't work on Ollama 0.32.5 (reasoning leaks inline into content); think: true separates it into the thinking field but the model still reasons on every answer. The /no_think soft switch appended to the query was measured as an arm: generation latency ~-25% (NOT halved: on this version it sometimes fails to suppress reasoning), and correctness and abstentions equivalent to the standard arm — again after adjudication. Raw gemma3 verdicts for this arm are 12/3/3 at both k against 13/4/1 and 11/6/1 for standard, and one negative (Q23 at k=10) was scored partial on what manual inspection found to be a clean, well-motivated abstention. Two of the three raw "wrong" verdicts were judge errors; the third pair was real — the empty answers described next. Observed failure mode: the model can loop inside the thinking block (316k characters) until the context is exhausted and return an empty answer (~4%). That's why the default uses /no_think with a fallback: empty answer → retry without the suffix, and the actual mode (no_think / fallback_think) is recorded in every answer of eval runs.

Hybrid BM25+RRF (w05 arm, weights 1.0/0.5): recall@8 0.905, MRR 0.703 — available via retrieval_mode: "hybrid" in rag/config.py. Decisions made via before/after on the harness: final_k 6→8; formal-opening/preamble split in the EUR-Lex converter (arm B); rrf_k=10 sealed; IDF gate, dedup/MMR and per-considerando headings (arm C) measured and rejected.

Known limits

• Lay/legal vocabulary mismatch: the unresolved residual is Q17b — "fornitore" in the query vs "responsabile del trattamento" in the text (dense rank 53, BM25 29: both branches blind). Same class as Q12, where rephrasing with the legal term moves the rank from 7 to 3. The fix is query expansion (rephrasings generated by the local model): planned, not yet implemented.
• Golden set coverage gap: no "citational" questions (a user pasting a sentence from the text or citing an article by number). Deliberate (see the golden.yaml header), but on legal corpora it's a real use case, and it's where the BM25 branch — measured and NOT promoted to default — would shine. To be addressed when the corpus grows, not to decide the default retrieval.
• FTS5 uses the unicode61 tokenizer (with remove_diacritics 2): no Italian stemming. If one is added, it will be because the harness proves it earns its place.
• Dense retrieval is brute force (matrix in RAM): declared sufficient under ~100k chunks, not beyond.
• bge-m3 via Ollama exposes dense vectors only (no sparse, no ColBERT): that's why the BM25 lexical branch exists.
• No OCR in this version (the loader interface allows it).
• PDFs with letter-spaced text (e.g. EUR-Lex PDFs: "par ticolare le ragionev oli") silently degrade embeddings and BM25: an ingest-time quality check is planned, not yet implemented.
• The LLM judges (groundedness and correctness) are archived as non-conclusive on this setup: see "Negative results" and rag/eval/judge_audit.md. They remain in the code for future re-experiments, not as evidence.

License

Apache-2.0 — see LICENSE and NOTICE. Models are not in the repository: they are downloaded at runtime via Ollama under their respective licenses (bge-m3: MIT; qwen3: Apache-2.0). The sample corpus in data/sample/ contains only redistributable documents: origin and license of each file in data/sample/SOURCES.md.