[Badri-Rajendran]

Problem

_bm25_index() read data/corpus/indices/bm25.pkl from local disk and raised FileNotFoundError when it was absent. Nothing catches that exception — grep -rn "FileNotFoundError" src/ finds only the two raise sites — so the chat route's catch-all turns it into a 500.

An app running anywhere but the machine that built the corpus therefore answers no questions at all. Not degraded, not dense-only: every question fails.

That never showed up because the app had only ever run where the file was. It is also not fixable where it stands:

• data/ is gitignored and stays that way, so no image can be built containing it;
• the file is produced by make ingest, a long local job;
• the constraint is provenance, not size (it was 3.4 MB).

Object storage would mean a storage account, a credential, an SDK and a new startup failure mode. Building the image locally is not CD.

Implementation

The index is derived data — every byte comes from the chunks table, which already holds each chunk's text and id. So it moves next to them.

• search_indexes: name (PK), payload, chunks, built_at. One row; a second index would be a second row, not a second table. New migration.
• make build-index rebuilds it from the chunks already stored — no fetching, chunking or embedding. Seeding a fresh database becomes a restore plus one command, and the index can't silently disagree with a corpus loaded some other way. make ingest still builds it as its last step.
• A missing index raises MissingSearchIndexError naming the command to run, instead of a FileNotFoundError naming a path that won't exist wherever the error was raised.
• bm25_index_path and INDEX_DIR are gone. There is no path to configure.
• The payload drops its texts key — a second copy of the whole corpus, read by nothing outside one test. 3.4 MB → 1.8 MB for the same 1,568 chunks.
• Also added make test / make lint / make check, so "it passed locally" means what CI means.

On unpickling: the payload is a pickle because BM25Okapi is a Python object. It is written only by this project's ingestion, into a table only this app writes to, and it was already being unpickled from a file. Moving the bytes from a file the app owns to a row the app owns adds no source of untrusted input. ADR 0021 records what would change that assessment and what the answer would be.

Validation

• uv run pytest -q --cov=src --cov-fail-under=85 — 413 passing, 92.27% (up from 406 / 91.8%).
• New tests/test_search_index.py (7 tests): the row and its chunk count, the payload carrying ids but not texts, a rebuild replacing the one row, retrieval reading it, the error naming make build-index, and build_index rebuilding from stored chunks. Two of these caught real problems while being written — a pre-existing committed row leaking into assertions, and a 2-document BM25 corpus where idf is exactly 0 so nothing scores above the filter.
• The orchestrator's tests now assert on what reaches the indexer at the call rather than by reading back a pickle, which is what they were really checking.
• Built against live data: make build-index → bm25 | 1568 chunks | 1852 kB.
• ruff check, alembic check (no new operations after the migration), bandit -r src -ll clean.

Security / DB / API impact

• DB: one new table, search_indexes. Additive; the downgrade drops it.
• Security: unpickling moves from a local file to a row in the app's own database — same trust boundary, documented in ADR 0021. The # nosec B301 justification is updated to match.
• API: none. A previously-uncatchable 500 becomes a named exception, but the deployment this fixes does not exist yet.

The decisive test is a container with no data/ directory answering a question correctly — that comes with the Dockerfile PR, which this unblocks.

Note

Stacked on #32 → #31 → #30 → #29 → #28.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.