Repository navigation
California plan comparison, a source registry, and plan-search fixes - #38
Merged
Merged
Conversation
Badri-Rajendran
changed the title
Badri-Rajendran
marked this pull request as ready for review
September 25, 2026 04:39
… data Checking the design against CMS's 2026 California file found three things the spec had wrong. Western Health Advantage files one service area across two rating areas, with a separate plan for each, so a plan is sold only where it has a filed rate. Four real Los Angeles ZIPs have a prefix CMS lists in neither rating area, so they show unpriced instead of aborting the load. And only in-network individual cost shares are loaded, the only values whose API wording is verified.
California runs Covered California, which the Marketplace API does not serve. exchange_for() gives the exchange a state's plans are sold on, and CATALOG_STATES the states PolicyPal has plans for: the 30 API states plus the filed-rate ones. MARKETPLACE_STATES is untouched. plan_year_on_sale() says which plan year is being sold, next year's from 1 November, so answers can say when California data is for a year no longer on sale.
The pilot may go commercial, and its sources differ: Wikipedia is CC BY-SA, CMS data is public domain, and Covered California reserves all rights and forbids automated access. registry.toml records each one, with its permission status, robots.txt check and removal steps. registry.py refuses an invalid registry at import and allows only an approved permission to be enabled, and SOURCES is filtered to the enabled corpus sources. Refused sources are recorded disabled, so the decision is not re-litigated.
California's premiums are filed per plan, rating area and age rather than fetched live, and some of its plans are sold in only part of a county. The migration adds plans.catalog_source (checked to cms_api or ca_sbe_puf), plan_counties.zipcodes (NULL for a whole county, as every API row is), and rating_areas, plan_rates and catalog_loads. zip3 is '' rather than NULL for a whole county, so its unique key holds. Rates cascade with their plan, as cost shares do. Upgrade, downgrade and re-upgrade verified; alembic check clean.
The Marketplace API has no California plans; CMS publishes them instead as a yearly zip of CSVs. read.py parses it as pure functions. load.py validates the whole file before one transaction writes it: - no unknown issuer; - every county mapped; - every plan sold somewhere; - no plan rated in an area it is not sold in, which catches a mistyped county in the hand-transcribed rating-area map. A plan is sold only where it has a filed rate: Western Health Advantage files one service area over areas 2 and 3, with a plan per area. Los Angeles ZIPs whose prefix CMS lists in neither of its areas are reported and left unpriced, never guessed. An empty file is refused before any delete, since NOT IN () would match every plan. Loaded from the real 2026 file: 190 plans, 11 issuers, 1,199 plan-county rows (40 dropped), 28,101 rates; reloading changes nothing; prices match the file to the cent.
…e's exchange Plan search serves California once its plans are loaded: - Pricing: the filed rate for the plan, the ZIP's rating area and the person's age (clamped to CMS's 14 to 64 bands), read in SQL. CMS is never called, and age_rated_premiums refuses a state the API does not serve. - Service areas: a plan sold in only part of a county is listed only for its ZIPs. - Unpriced: a ZIP with no rating area shows its plans unpriced, never free. Results carry the exchange, premium_source, and whether the plan year is no longer on sale. In that case the answer opens with a notice written by the server: asked to write it, the model also said it where it was untrue. The profile says plan comparison is available in California only once its plans are loaded, which is exactly when search serves it. Fixes for every state: a Bronze search now includes Expanded Bronze, and prompts, the no-link reason and the plan table no longer send every state to HealthCare.gov. The plan table shows the plan year and describes each exchange's premiums when one answer mixes states. Checked against the real model: California answers name Covered California and never HealthCare.gov; Texas answers are unchanged.
"Compare the gold plans." and "Compare silver plans for me." often skipped search_plans in every state. It predates the California work: with main's prompts, 2 of 4 runs searched. Two symptoms had one cause. Either the first round ran out of its 512 tokens deliberating, or the model answered from the corpus as if asked what a metal level is. The prompt never said which reading was meant. It now says a request to compare, list or show plans by metal level or type is about real plans the user can buy, and keeps the exception: what a level means in general is still answered from the corpus. Measured against the real model at the unchanged 512-token cap: 35 of 35 search requests reached search_plans across seven phrasings, and 0 of 10 concept questions did. The first-round cap was not the cause: raising it alone still missed. The generation eval gains the hard phrasings (9 plan-search floor) and a concept set that must not search. The full eval meets every floor.
README and CLAUDE.md said plan comparison covered only the 30 HealthCare.gov states. The README now documents: - make ingest-ca-plans, and where its findings and yearly runbook live; - the source registry; - the eval's PLAN SEARCH and CONCEPTS sections, and the California catalog they need.
Badri-Rajendran
force-pushed
the
feature_ca_plan_catalog
branch
from
September 25, 2026 18:48
f28274f to
c1da544
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A user with a California ZIP got "not a marketplace state", and the model sent them to HealthCare.gov. That is the wrong exchange: California runs Covered California, and the CMS Marketplace API has none of its plans. The expansion is a pilot that may go commercial, so every data source also needs a recorded licence and permission status.
Testing also exposed a bug that predates this work. "Compare the gold plans." skipped the plan search in about half of runs, in every state.
Implementation
Sub-project 1 of 5 in the California expansion. Spec:
docs/superpowers/specs/2026-09-24-ca-plan-catalog-design.md; plan:docs/superpowers/plans/2026-09-24-ca-plan-catalog.md.src/ingestion/sources/registry.tomlrecords every source's licence, permission,robots.txtstatus and removal steps. It is validated at import, and only approved sources run. Covered California is recorded as link-only, because its Terms of Use forbid automated access.make ingest-ca-plans YEAR=reads CMS's state-based exchange PUF.plans.catalog_sourceandplan_counties.zipcodes, and new tablesrating_areas,plan_ratesandcatalog_loads.Validation
Tests
make checkis green: 559 passed, 93.45% coverage (floor 85%), ruff clean,alembic checkclean.The real 2026 California file
Full generation eval against the real model: every floor met.
Plan-search bug, measured
main's prompts: 2 of 4 runs searched.Prior-year notice
End to end in the browser (Playwright)
Independent review: no critical findings. Its three findings are fixed with tests:
NOT IN ().Security / DB / API impact
15bb18a57facis additive; API-sourced rows are unchanged. Removal steps are in the registry anddocs/runbooks/california.md.marketplace_stateis true for California once its plans are loaded. Authentication and 429 limits on/api/profileand/api/countiesare unchanged and re-tested.data/stays gitignored. Covered California is linked, never fetched.docs/runbooks/california.md, which I will confirm before doing.