PolicyPal

Cited answers for US health coverage. A RAG assistant over a curated insurance corpus and live CMS Marketplace data. Every claim cites its source; ungrounded questions are refused, not guessed.

PolicyPal lets a logged-in user ask natural-language questions about any kind of insurance — health, auto, life, home, and more — and returns grounded answers generated by an LLM using context retrieved from a vector store. Every answer cites the source passages it was built from.

Built: comparison of real purchasable health plans drawn from the CMS Marketplace API, and for California from CMS's state-based exchange data (ADR 0024). That half is deliberately narrower than the Q&A — ACA individual and family plans, in the 30 states served by HealthCare.gov plus California's Covered California (190 plans from 11 issuers for 2026, priced from CMS's filed rates). The other 20 state-based exchanges, employer coverage, Medicare and Medicaid are out of scope. All five phases of docs/plans are complete; the catalog is loaded for eighteen states (3,276 plans, 137 issuers), and 2,274 of those plans (69.4%) have their Summary of Benefits read here — 2,136 read whole and 138 in part (ADR 0017). Keeping the documents current is a routine now, not a phase: docs/runbooks/sbc.md.

PolicyPal will compare plans; it will not recommend one. Advice-shaped questions are refused by design, and that refusal is a measured property with its own regression floor (ADR 0004).

The corpus is drawn from two sources chosen to cover different registers (ADR 0003): Wikipedia for what insurance is across every line of business, and HealthCare.gov — the CMS Uniform Glossary plus its consumer articles, public domain under 17 U.S.C. § 105 — for what a policyholder actually does. Content is cleaned, chunked, embedded, and stored in pgvector for semantic search.

Because HealthCare.gov is the authoritative consumer source, answers about health coverage rules are US-specific.

Features

• Conversational Q&A, organized into threads you can rename, over a curated insurance knowledge base
• Hybrid RAG pipeline — BM25 + pgvector semantic search, cross-encoder reranked and relevance-filtered before reaching the LLM, so weakly-relevant matches never become context
• Cited answers — answers render as Markdown, and every claim carries a numbered seal. A seal opens the Sources panel on the passage it cites: a brief quote with its document, section, match score and a link to the carrier's PDF or the Wikipedia article, with Wikipedia's licence credit, and checked against a hash of what was cited, so a passage rebuilt since the answer is never shown as if it were the one cited (ADR 0027). The plans an answer compared show in a numbered table, and a plan named in the text jumps to its row; "Ask about this plan" starts a follow-up. Both survive reloading the conversation. Each plan in the table says whether its Summary of Benefits was read, and why not (ADR 0017)
• Streaming answers — an answer arrives as it is written, after progress steps ("Searching plans near you", "Reading Summaries of Benefits", …). Stop (or Esc) cancels it and keeps the question, with "Ask again" (ADR 0027)
• A chat that stays out of the way — threads grouped by date, searched as you type, and renamed or deleted inline; a sidebar that hides (a drawer on a phone); System, Light or Dark themes; and shortcuts: Ctrl/⌘K for a new question, / to type, Esc to stop or close. Only the theme and the sidebar's hidden state are kept in the browser
• Real plan comparison — ask about ACA Marketplace plans; the model searches the ingested catalog and CMS prices the plans live for your age (ADR 0010). In California, premiums are CMS's filed rates for your age and ZIP's rating area, and answers name Covered California (ADR 0024). It compares plans and never recommends one
• A profile, kept off the model — signup takes a ZIP code, date of birth and county, and plan questions use them without their ever being sent to the LLM. Nobody under 13 can sign up (ADR 0012)
• JWT-authenticated API — only a signed-in user can query, and only ever sees their own threads
• Reproducible ingestion — a single command runs fetch → normalize → chunk → embed → store, across every registered source

Tech Stack

Layer	Technology
Frontend	React 19 (Vite, plain JSX); react-markdown + remark-gfm (answers, no raw HTML), motion (reduced-motion aware), lucide-react (icons) — all pinned
Backend	Flask
Database	PostgreSQL + pgvector
RAG	rank-bm25, sentence-transformers, HF transformers
ORM / Migrations	SQLAlchemy + Alembic
Tooling	uv, npm, Docker Compose, Make

Architecture

Wikipedia ─────┐
               ├──► Ingestion pipeline ──► pgvector
HealthCare.gov ┘                              │
                                              │
User ──► React UI ──► Flask API ──► Retrieval ──► LLM ──► Cited answer
                         ▲                         │   ▲       │
                         │            search_plans ▼   │       │
                         │           plan_coverage     │       │
                         │  plan tables, CMS premium, SBC text │
                         └──────────── persisted reply ───────┘

• Offline ingestion (make ingest) populates the vector store and BM25 index. Run it once, or again whenever the source corpus changes.
• Online serving (make api) retrieves relevant chunks per query, reranks and filters them, and generates a grounded answer, persisting the conversation.
• Plan questions reach the search_plans tool, offered whenever the plan catalog is loaded. It resolves the ZIP to its county, filters the catalog, and fetches premiums for the user's age from CMS. A reply that drew on no chunk and no search is refused, not sent (ADR 0010).
• Coverage questions about a specific plan ("does the second one cover MRIs?") reach the plan_coverage tool. It reranks that plan's own SBC sections and returns the best four, cited by plan and section. The plans last shown in the thread are handed to the model so "the second one" resolves. A situational question ("will my MRI be covered?") gets a fixed boundary sentence and the plan's terms, never a yes or no (ADR 0014).

Both halves share the same embedding model (src/core/embedding.py) so ingestion-time and query-time vectors stay comparable.

Chat API

Every chat endpoint needs a JWT, and a thread or citation that isn't yours is a 404, never a 403. Each is rate-limited per user and answers 429 with Retry-After past its limit.

Endpoint	Does	Limit
GET /api/chat/threads	Your threads, most recently active first	60/min
POST /api/chat/threads	Starts a thread	30/min
PATCH /api/chat/threads/<id>	Renames it (1–200 characters); its place in the list stays	30/min
DELETE /api/chat/threads/<id>	Deletes it with its messages	30/min
GET /api/chat/threads/<id>/messages	The transcript, with citations and plans	60/min
POST /api/chat/threads/<id>/messages	Asks, and returns the answer as JSON	15/min, shared with the stream
POST /api/chat/threads/<id>/messages/stream	Asks, and streams the answer	15/min, shared with the JSON route
GET /api/chat/sources/<id>	One of your citations, as a brief quote	60/min

Streaming (ADR 0027). The stream route sends Server-Sent Events: user_message (the saved question), then stage for progress (understanding, searching, plans, coverage, writing), notice, delta for answer text and reset to discard it, and finally done with the saved answer and thread, or error. Errors that can be known up front (401, 404, 422, 429) come back as ordinary responses before the stream starts. Text is streamed only once the answer is grounded, the answer is saved before done is sent, and a client that goes away saves no answer but is still charged for what the model wrote. The frontend reads the stream with fetch, since EventSource can't send a POST with a Bearer header.

Cited passages. GET /api/chat/sources/<id> takes a citation's own id (each source in a message carries one) and returns its kind, title, document, section, link, licence and a quote of at most 300 characters, taken from the part of the passage most like the answer. Its status says whether the quote can be trusted: ok, unverified (an answer from before hashes were stored), changed or missing; the last two carry no quote.

Project Structure

PolicyPal/
├── src/
│   ├── core/           # shared: db, embedding, reranker, device, logging
│   ├── models/         # SQLAlchemy ORM (users, threads, messages, chunks, plans, SBCs)
│   ├── schemas/        # Pydantic request/response models
│   ├── api/            # Flask app factory, routes, rate limiting, auth deps
│   ├── services/       # retrieval, generation, auth logic
│   ├── ingestion/      # fetch → normalize → chunk → embed pipeline
│   │   ├── sources/    # one module per corpus source (see ADR 0003)
│   │   └── sbc/        # plan documents: fetch, parse, store (ADR 0013)
│   └── policypal/      # settings (config.py)
├── migrations/         # Alembic revisions
├── docs/decisions/     # ADRs for material architectural choices
├── docs/plans/         # phased roadmap for the plan-comparison expansion
├── docs/findings/      # what each live run actually measured
├── docs/runbooks/      # keeping the plan documents current
├── frontend/           # React + Vite chat UI (design: docs/design/chat-redesign/)
├── scripts/            # ask.py (manual RAG smoke test), eval_retrieval.py,
│                     # eval_generation.py, eval_sbc_ranking.py
├── tests/
├── docker-compose.yml  # PostgreSQL + pgvector
└── pyproject.toml

Getting Started

Prerequisites

• Python 3.12+ and uv
• Node.js 20+ and npm (for the frontend)
• Docker (for PostgreSQL + pgvector)

Setup

# 1. Install backend dependencies
uv sync

# 2. Start PostgreSQL + pgvector
docker compose up -d

# 3. Apply database migrations
make migrate

# 4. Populate the vector store (one-time, or whenever the corpus changes)
make ingest

# 5. Start the API (http://localhost:5000)
make api

# 6. In a separate terminal, install and start the frontend (http://localhost:5173)
cd frontend && npm install && cd ..
make ui-dev

The rest of frontend/package.json is wrapped the same way, so the UI can be driven from the repo root:

Target	Runs
make ui-dev	Vite dev server
make ui-build	production build into frontend/dist
make ui-lint	ESLint
make ui-test	Vitest
make ui-preview	serves frontend/dist — run make ui-build first

And what CI runs, in one command each:

Target	Runs
make test	pytest with the 85% coverage floor
make lint	ruff
make check	lint, tests, alembic check, bandit, pip-audit

Copy .env.example to .env at the repo root and fill in the three required values before running the API.

Running the API in a container

docker build -t policypal-api .
docker run --rm -p 8000:8000 --env-file .env -e ENVIRONMENT=production policypal-api

The image carries the API only — ingestion, refresh and the reports stay on the machine that holds data/ (ADR 0016, 0019), and .dockerignore keeps the issuer PDFs out of it. It needs no data/ directory at all: the BM25 index comes from the database (ADR 0021) and both models are baked in at their pinned revisions, so a cold start makes no HuggingFace call. torch comes from PyTorch's CPU index on Linux, so no CUDA libraries ship (ADR 0022).

It runs one gunicorn worker with eight threads. That is a deliberate ceiling, not a default: Flask-Limiter has no shared storage yet, so a second worker would keep its own counters and double the configured rate limits.

macOS note: AirPlay Receiver (Control Center) listens on *:5000. Flask still binds 127.0.0.1:5000 alongside it, but localhost can resolve to ::1 and reach AirPlay instead — which answers 403 and looks like the API is broken. The frontend therefore calls 127.0.0.1:5000, not localhost:5000. Only if make api genuinely fails to bind do you need another port (uv run flask --app src.api.main run --port 5050), with frontend/.env's VITE_API_BASE_URL set to match.

Environment Variables

Variable	Description
DATABASE_URL	SQLAlchemy Postgres connection string
JWT_SECRET_KEY	Signing key for access tokens
OPENAI_API_KEY	Required. Answer generation runs on a hosted model (ADR 0008)
HF_API_KEY	Optional Hugging Face token (for gated models)
CMS_MARKETPLACE_API_KEY	Optional. Needed for make ingest-plans, and for live age-rated premiums in chat (request one); without it, chat shows the stored age-27 premium
CMS_LIVE_TIMEOUT_SECONDS	Timeout for the live premium call in chat (default 8); on failure the answer falls back to the age-27 premium
PLAN_ANSWER_MAX_OUTPUT_TOKENS	Output cap for the reply after a plan search (default 2048); a ten-plan comparison does not fit max_output_tokens
DEVICE	auto | cpu | mps | cuda — device for the local embedding and reranker models
ENVIRONMENT	development | production
LOG_LEVEL	DEBUG | INFO | WARNING | ERROR | CRITICAL

See src/policypal/config.py for the full list, including retrieval, reranking, and LLM tuning defaults (chunk size, top-k, relevance threshold, model name, output cap, reasoning effort).

Only answer generation is hosted. Embeddings (bge-small-en-v1.5) and the reranker (ms-marco-MiniLM-L-6-v2) still run locally, so ingestion and retrieval cost nothing and work offline.

Note on the output cap: max_output_tokens bounds reasoning tokens and the reply together. Set it too low and the model spends the whole budget reasoning and returns an empty string — finish_reason is length and no error is raised.

Every user gets a daily generation budget (llm_daily_token_budget, default 200,000 tokens — roughly 65 messages a day at current usage). Once spent, POST /messages answers 429 with a Retry-After header and an error string distinct from the rate limiter's, so the UI can tell "slow down" apart from "come back tomorrow" (ADR 0008). A hosted call is bounded by llm_request_timeout per attempt and llm_max_retries retries on top of that, so the real worst case is the two multiplied together, not the timeout alone.

The frontend reads VITE_API_BASE_URL (defaults to http://127.0.0.1:5000) from frontend/.env.

Ingestion Pipeline

make ingest

Runs in order: fetch (download each source) → normalize (convert to markdown) → chunk (per-source splitting + a shared BM25 index) → embed (vectorize and upsert to pgvector). Every phase is idempotent, so a re-run only picks up what's missing.

The BM25 index is stored in the database beside the chunks it indexes, not on local disk (ADR 0021), so the app can run somewhere with no data/ directory. It is derived data: make build-index rebuilds it from the chunks already stored, with no fetching, chunking or embedding — which is what makes seeding a fresh database a restore plus one command.

Adding a corpus means writing one Source subclass in src/ingestion/sources/ — fetch, normalize, chunk_documents — and registering it in that package's __init__.py. No pipeline, chunking, or embedding code changes. Chunking strategy lives with the source because document shape differs: a glossary term is one atomic chunk, while an article is split recursively.

Plan catalog

make ingest-plans STATES=TX,FL      # or STATES=ALL for all 30 HealthCare.gov states

Loads real purchasable plans from the CMS Marketplace API into relational tables — issuers, plans, plan_counties, plan_cost_shares — kept apart from the RAG corpus because a deductible is a WHERE clause, not a similarity search (ADR 0009). Each run also rewrites zip_counties, the ZIP-to-county crosswalk for the plan year across every state, which chat uses to resolve a user's ZIP (ADR 0010) and signup checks a ZIP code against (ADR 0012) — so run it at least once before anyone signs up. On a database ingested before that table existed, re-run once. Needs CMS_MARKETPLACE_API_KEY; make ingest does not.

• Scope is chosen per run. STATES is required, with no default, so a bare make ingest-plans cannot start a multi-hour run.
• Plans are sold per county, and each county takes about 14 paged requests. One state is thousands of requests and ALL is tens of thousands, paced deliberately against a free public API. For a quick check: uv run python -m src.ingestion.plans --states TX --max-counties 2.
• Re-runs are safe. Each county is one transaction and every write is an upsert, so an interrupted run keeps what it finished and a re-run changes no row count. A county that fails is reported and skipped; the run continues.
• premium_reference is indicative, not a quote: the unsubsidized premium for a single 27-year-old, CMS's own convention for comparing plans.
• Plan year defaults to the current year. During open enrollment, pass the next one explicitly: uv run python -m src.ingestion.plans --states TX --year 2027.

The county list is cached per plan year in data/plans/raw/; delete it to refresh.

California (Covered California)

make ingest-ca-plans YEAR=2026      # ZIP=path for a file downloaded by hand, REFRESH=1 to re-download

The Marketplace API has no California plans, so they come from CMS's state-based exchange PUF, one zip per plan year, published May to August of that year (ADR 0024). The loader:

• writes the same tables, plus rating_areas, plan_rates and catalog_loads;
• validates the whole file before writing, in one transaction, so a bad file changes nothing;
• keeps a plan only in counties where it has a filed rate.

Premiums are read from plan_rates in SQL; CMS is never called for California. Rating areas and issuer names are transcribed by hand and re-checked yearly: docs/runbooks/california.md. What the file holds is in docs/findings/ca-sbe-puf.md.

Source registry

Every source PolicyPal reads or links is recorded in src/ingestion/sources/registry.toml, with its licence, permission status, robots.txt status and removal steps (ADR 0025). Only approved sources run. Covered California is recorded as link-only: its Terms of Use forbid automated access. Each California carrier whose SBCs are used is an sbc_host entry with status mandated_disclosure, which is not a licence, so its commercial use stays under review (ADR 0026). Verified API behaviour is recorded in docs/findings/cms-marketplace-api.md.

Plan documents (SBC)

make ingest-sbc STATES=NH,DE        # after make ingest-plans for the same states
make ingest-sbc STATES=FL,TX TOP_ISSUERS=1   # only the ten largest parent companies
make ingest-sbc STATES=NH YEAR=2027          # a run during open enrollment

Reads the Summary of Benefits and Coverage behind each catalog plan's benefits_url: the federally standardized document that says what a plan covers, what it excludes and what each common medical event costs. The text is split along the template's own sections into sbc_chunks, kept apart from the corpus that make ingest rebuilds (ADR 0013).

• Polite and never pushy. HTTPS to public hosts only, robots.txt obeyed, two seconds between requests to one host, and a 15 MB cap. A host that refuses automated requests is recorded as blocked and left alone. Its plans get no SBC answers.
• Parsed once. A document stored by the current parser is skipped on the next run, with no request. Bumping PARSER_VERSION in src/ingestion/sbc/extract.py reads every document again from its kept PDF (ADR 0015, 0016).
• One line per service. Chart rows are rebuilt from the table's ruled grid, so a wrapped service name stays beside its price; a chart ruled across but not down falls back to reading the band between two rules (ADR 0018). A scanned PDF with no text layer would be recorded as unparseable, and there is no OCR — but measuring eighteen states found zero of them, so no issuer publishes a scanned SBC.
• Every PDF is kept. Downloaded SBCs stay in data/sbc/raw/ (gitignored, never served or committed), about 0.75 MB each. A stored document whose PDF has gone missing is downloaded again, and one rejected as another year's is moved to data/sbc/rejected/ (ADR 0016).
• The largest issuers. TOP_ISSUERS=1 limits the run to the ten largest parent companies by 2025 enrollment, listed in src/ingestion/sbc/top_issuers.py; ISSUERS=40788,66252 limits it to the HIOS issuer IDs given.
• Plan year is checked. A document whose coverage period is for another year is refused.
• Every attempt is recorded in sbc_documents, and the run ends by naming the plans left without an SBC. A recorded failure is left to make refresh-sbc (ADR 0019), so ingesting another state does not re-request every blocked host.
• For a quick check: uv run python -m src.ingestion.sbc --states NH --limit 5.

California's SBCs

make apply-sbc-manifest YEAR=2026            # links from src/ingestion/sbc/manifests/ca-2026.csv
make ingest-sbc STATES=CA YEAR=2026          # reads only the carriers that may be crawled
make import-sbc YEAR=2026 DIR=~/Downloads/ca-sbc   # PDFs a person downloaded in a browser
make check-sbc-manifest YEAR=2026            # what still needs a person's eye

CMS's California file has no SBC links. A committed, hand-built manifest gives each plan its link, and each carrier has a registry entry (ADR 0026).

• Crawled: carriers whose robots.txt and website Terms both allow automated access.
• Manual only: carriers whose Terms ban robots or scrapers, and hosts that block bots. A person downloads their SBCs in a browser, and make import-sbc reads the file with no request to the carrier. The document is recorded as manual and never re-requested.

California answers say the SBC is the plan's standard version, since cost-sharing reductions lower what it shows. The yearly steps are in docs/runbooks/california.md, and the carrier research in docs/findings/ca-sbc.md.

Keeping documents current

make refresh-sbc STATES=NH,DE YEAR=2026

Asks every stored document whether the issuer has changed it, sending the ETag and Last-Modified it served last time, and retries the failures ingest-sbc skips. A changed file is downloaded, the one it replaces is moved to data/sbc/archive/, and the new text replaces the old. A temporary failure (a 5xx or a timeout) leaves the stored text alone; a 404, a refusal or a page instead of a PDF drops it, so answers link the issuer's document instead.

The cadence, the plan-year rollover and what to check are in docs/runbooks/sbc.md.

Coverage report

make sbc-report YEAR=2026                       # every state with plans
make sbc-report YEAR=2026 STATES=FL,TX VERIFY=1 # and hash every kept PDF

Counts, per state and issuer, how many catalog plans have a Summary of Benefits with text behind them and what stopped the rest: no link at all, never read, or a failure with its reason. It also reports documents no plan points at any more, documents an older parser stored, plans the latest catalog run did not return, and how much disk the kept PDFs use. It reads and writes nothing.

Per-issuer results of the live run are in docs/findings/sbc-documents.md.

Once loaded, chat answers coverage questions about those plans from their SBC, citing each passage as "plan name - Summary of Benefits - section". For a plan whose SBC could not be read, the answer says so by name, gives the reason, links the issuer's PDF, and stops: it never describes that plan from general material. A comparison names every plan it has no document for, and each saved plan card records whether its SBC could be read (ADR 0017). Links in answers are clickable (https only).

A document can also be read only in part — the costs chart drawn in a way that yields no table, or a questions column that interleaves its answers. Those are stored as partial: the text that was read is kept and searched, the plan card says the chart is not all there, and an answer that finds nothing says the part that would answer wasn't among what was read, rather than implying the plan doesn't cover it. 138 of the 1,498 documents stored are partial.

Testing

# backend
uv run pytest

# frontend
cd frontend
npm run lint
npm test
npm run build

CI (.github/workflows/ci.yml) also runs ruff, alembic check (schema drift), bandit (SAST), pip-audit/npm audit (dependency CVEs), a gitleaks secret scan, a backend coverage floor (--cov-fail-under=85), and a Docker build scanned by Trivy — which also runs the image and fails if any issuer PDF reached it. Every pull request, and every push to main.

Deploying

.github/workflows/cd.yml deploys to Azure after a green CI run on main: the API to Container Apps, the browser app to Static Web Apps, against a PostgreSQL Flexible Server. Migrations run as a Container Apps job on the exact image digest being deployed, so the database needs no public opening. It waits on a required reviewer, and rollback is reactivating the previous revision. Sign-in is OIDC, so no Azure credential is stored in GitHub.

Setup, the deploy path, and what to do when one goes wrong are in docs/runbooks/deploy.md; the reasoning is in ADR 0023. Ingestion stays on the machine that holds data/ — the issuer PDFs are never uploaded (ADR 0016).

Retrieval quality eval

tests/ proves the retrieval/generation code is wired correctly (with mocked models and data). It can't catch a change that keeps everything wired correctly but quietly makes real answers worse. For that, scripts/eval_retrieval.py runs the real search() against the real ingested corpus and measures two different things:

• Routing — for a question about a known topic, does the document covering it come back? Catches embedding and reranking regressions.
• Coverage — for a question a real user would ask, does the corpus hold an answer at all, and does the retrieved text carry the evidence needed to answer it?

Coverage exists because routing alone is self-fulfilling: every routing question is derived from a document already in the corpus, so it can only ever pass. The Wikipedia-only corpus scored 25/25 on routing while failing to answer 12 of 20 realistic consumer questions — the gap that motivated ADR 0003. Both measurements have regression floors and the script exits non-zero below them. Run it after any change that could affect retrieval quality or the corpus:

uv run python -m scripts.eval_retrieval

Not a CI gate — it needs the corpus already ingested and loads real models, so it's slow and environment-dependent by nature.

Generation quality eval

Retrieval metrics grade context, never answers, and the gap between them is not theoretical: coverage once read 19/20 while one of the questions inside that 19 was returning a fabricated comparison, because the context held only one of the two concepts being compared. No retrieval metric could have seen it.

scripts/eval_generation.py runs the full answer_query() path and grades the generated text: whether an answer states the facts it must, and whether an ungrounded question is refused rather than invented. The refusal half is the hallucination guard.

uv run python -m scripts.eval_generation

A fourth set checks that plan questions reach search_plans. Its ZIP, 75801, lies only in Anderson County, TX, so load that county first: uv run python -m src.ingestion.plans --states TX --max-counties 1.

Two more check plan coverage (ADR 0014). COVERAGE asks about shown plans and requires each plan's SBC term, cited from that plan's SBC alone. BOUNDARY asks "will it be paid?" and requires the boundary sentence and the plan's terms, with no yes or no. Both need the NH, DE, TX and FL SBCs: make ingest-plans STATES=NH,DE,TX,FL, then make ingest-sbc STATES=NH,DE, make ingest-sbc STATES=TX,FL TOP_ISSUERS=1 and make ingest-sbc STATES=TX ISSUERS=40788,66252 (Baylor Scott & White and CHRISTUS).

MISSING DOCUMENTS (ADR 0017) asks about plans whose SBC couldn't be read: a bot-challenged BCBS of North Carolina plan, and blocked UnitedHealthcare and Oscar plans, alone and in a comparison. The answer must name each such plan and must not cite general material or state a figure for it. Its floor is full marks, and every COVERAGE answer is held to the same rule. It also needs the NC catalog: make ingest-plans STATES=NC and make ingest-sbc STATES=NC TOP_ISSUERS=1.

PLAN SEARCH requires every plan question to reach search_plans. That includes comparisons by metal level ("Compare the gold plans."), which used to skip it, and two California questions that must name Covered California, never HealthCare.gov. They need the California catalog: make ingest-ca-plans YEAR=2026. CONCEPTS requires the opposite for a question about what a level means ("What is a silver plan?"), which is answered from the corpus.

uv run python -m scripts.eval_sbc_ranking checks, without calling a model, that the section answering each of ten questions is among the four plan_coverage ranks highest, for two plans per issuer with stored SBCs. Run it before and after a parser change.

Slower still than the retrieval eval — it generates once per question — and likewise a manual tool rather than a CI gate. Unlike the retrieval eval it also costs money: generation is hosted. The first four sets are roughly 17k tokens; the coverage set adds about 26k and the boundary set three more answers, a few cents in all at current gpt-5-mini rates. The per-user daily token budget does not apply here — it is enforced at the API layer, and this calls answer_query() directly.

License

This project is for educational and portfolio purposes. Ingested content is sourced from Wikipedia under its respective license.